A communication-efficient privacy-preserving recommendation system and method

Abstract

The application belongs to the technical field of data recommendation, and discloses a communication-efficient privacy protection recommendation system and method, which uses a recommendation model based on a graph neural network to capture user interest and realize prediction of user preference. First, user embedding and item embedding obtained in federated training are acquired. Then, the user embedding and embedding of items that have not been interacted with by the user are input into the trained recommendation model to obtain the interest score of the user for the items that have not been interacted with. Finally, the items are sorted and recommended according to the size of the interest score of the user for the items output by the recommendation model. The application realizes protection of user privacy, and captures user interest and predicts user preference.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data recommendation, and particularly relates to a communication-efficient privacy protection recommendation system and method. BACKGROUND

[0002] A recommendation system estimates user preferences based on collected personal data and generates candidates for future user selections. Recommendation systems are widely used in movie, music, news, and book recommendations. Effective use of user behavior information to provide appropriate product recommendations for users plays an important role in the success of various enterprises. As a typical personalized service, in order to better capture and model user behavior habits and historical preferences, a recommendation system needs to collect attribute information of users and items and interaction records between them, such as ratings, click logs, etc., and often needs a large amount of user data to provide high-quality services. However, a large amount of user behavior records and user private attribute information, while enabling the model to grasp the user's behavior patterns, inevitably causes users' concerns about privacy issues such as disclosure of sensitive information about themselves. Analysis of user data can improve the performance of the recommendation model, while highly sensitive user privacy information can also be inferred from user data used for model training.

[0003] In the past, privacy protection scenarios have considered collecting personal data of users centrally and uploading to a trusted central server for storage and model training. Obviously, such a mode will cause privacy leakage and security problems in the process of transmitting personal data and in the process of storing data in the server. Therefore, the existing use of federated learning technology combined with differential privacy and other privacy protection technologies can achieve user data not uploaded to the server, complete the update of the recommendation model and the recommendation service under the premise of protecting user privacy. The recommendation scheme based on federated learning is that the user receives the initial state model sent by the server, uses local personal data to train the recommendation model, and uploads it to the server. Finally, the server aggregates the updated models of each client to update the global model. After multiple rounds of model aggregation and update, a usable recommendation model can be obtained.

[0004] The existing use of federated learning for recommendation model training requires that the data of these clients have some homogeneous attributes. When aggregating the recommendation model, the gradient or model parameters need to be aggregated, and the gradient or parameter is also an implicit expression of user data, which should also be protected by privacy protection technology. For example, when users independently add noise to realize the disturbance of uploaded data, it is more difficult to balance the usability of the model and the privacy of the data. Model aggregation requires a large amount of communication between the server and the client. When the amount of local data is too small, it may cause overfitting of the local model, thereby affecting the performance of the aggregated model. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a communication-efficient privacy protection recommendation system and method, which ensures the training of a recommendation model using federated learning, protects user privacy, captures user interest, and predicts user preferences, while improving model training speed and accuracy, fully utilizing user device performance, and reducing model training time.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is:

[0007] A communication-efficient privacy protection recommendation method, comprising the following steps:

[0008] Step 1, a recommendation model based on a graph neural network is constructed to capture user interest and predict user preferences, and the recommendation model is federated trained; the recommendation model includes an embedding representation layer of users and items, an interest extraction module of users based on a graph neural network, and a score calculation layer of user preferences, and in addition, a high-order information extraction module is also provided on the server side; in the federated training, the model held by the user client is recorded as a local model, and the model held by the center server is a global model; wherein, the recommendation model training task is completed jointly by the user client and the center server, the user client uses local data to train and update the recommendation model, and the center server aggregates and updates the local model uploaded by each user client to obtain the latest global model and the updated distribution;

[0009] Step 2, for a cold start user, the user embedding and item embedding obtained in the federated training are acquired;

[0010] Step 3, the user embedding and the embedding of the items that the user has not interacted with are input into the trained recommendation model to obtain the interest score of the user for the items that have not been interacted with;

[0011] Step 4, the items are sorted and recommended according to the size of the interest score of the user for the items output by the recommendation model.

[0012] Further, the federated training process comprises:

[0013] The first stage involves user data protection: Each user client perturbs and adds protection to its local interaction data before transmission. This perturbed data is then uploaded to the central server for calculating user behavior similarity and user grouping. Members within each group jointly train a recommendation model, rather than all users training a global model. Simultaneously, the central server's high-order information extraction module extracts high-order embedding information within each group for each user. After completing user grouping and extracting high-order information, the central server sends the high-order information, item embeddings, and model parameters to each group member. User embeddings, item embeddings, and the recommendation model are then trained through federated learning.

[0014] In the second stage, each client uses local data and received higher-order information to train the model locally. After several rounds of model training, the model is uploaded to the central server. The central server aggregates the local models uploaded by each client and updates the global model. This second stage is repeated multiple times until the global model is available.

[0015] Furthermore, the specific steps for user data protection processing are as follows: Define U = {u1, u2, ..., u...} N} and T={t1,t2,…,t M} represents the user set and the item set, respectively, where N is the number of users, M is the number of items, and u is the number of users. n For project t m The rating is R nm Then the rating matrix formed by all users' ratings of the project is R∈R N×M Each user stores their rating data locally, therefore the rating matrix is ​​unavailable to the server; client c is defined as a local device that stores user rating data and actual user interaction data, and each client c n Associate a user u n Its actual rating data and interaction data are R n_ and h n Each user u n There is a local rating record vector Use these rating records to generate interaction data h n h nm Represents interactive data h n The method for converting the m-th bit is as follows:

[0016]

[0017] Subsequently, user interaction records after the disturbance Generated through a random flipping mechanism.

[0018] Further, the user behavior similarity calculation and user grouping are performed by the server, and the behavior similarity of users is estimated by Jaccard similarity, and the similarity function of two users 1 and 2 is defined as follows:

[0019]

[0020] where h1 represents the interaction data of user 1, and h2 represents the interaction data of user 2,

[0021] The key to calculating the user similarity is to accurately estimate h1∩h2 and h1∪h2 given the perturbation vectors and The perturbation vectors and are the interaction data of h1 and h2 after adding perturbation, and for i,j∈{0,1},n ij represents the number of ordered pairs of h1[s]=i and h2[s]=j at the s-th bit position of the real vector, and all n ij constitute the vector Similarly, for i,j∈{0,1},m ij represents the number of ordered pairs of and at the s-th bit position of the perturbation vector, and all m ij constitute the vector Let pr(1→1)=pr(0→0)=q1, which represents the probability that the value at a certain position remains the original value, and pr(1→0)=pr(0→1)=q2, which represents the probability that the value at a certain position is flipped; then the cardinalities of the union and intersection of vectors h1 and h2 can be estimated by the mean field model as follows:

[0022]

[0023] where the probability that the value at a certain position of the original vector remains the original value is q1, and the probability of flipping is q2; by inverting the matrix, four unbiased estimators of n ij can be obtained:

[0024]

[0025] The number of 00 ordered pairs in the original vectors h1 and h2 at the corresponding positions is determined by the vector length m and |h1∪h2|, that is, n 00 =m-|h1∪h2|, so |h1∪h2| can be estimated as,

[0026]

[0027] an estimate of |h1∪h2|;

[0028] Meanwhile, can be directly used as an estimate of h1∩h2,

[0029]

[0030] an estimate of h1∩h2;

[0031] Based on this calculation method, the behavior similarity of any two users can be estimated using the uploaded perturbation data, and a user similarity matrix S ∈ R N×N , the nth row of the similarity matrix S is taken as the user u n The behavior characteristics in the grouping stage, i.e., the user u n Similarity with all other users;

[0032] Then, the clustering algorithm is used to divide the users into different clusters G = {G l | l = 1, …, C}, where G l is the lth user group, and C is the number of user groups after clustering.

[0033] Further, the high-order information extraction process is as follows:

[0034] In the federated learning setting, the user's data is saved on the user's device. For user u n , only the items that the user u n has interacted with, i.e., the first-order nodes, are locally available; the user embedding retains the user's interest and contains certain privacy information, and under strict privacy restrictions, the embedding of other users who have the same interaction items as the user u n , i.e., the second-order nodes, is not available; the interaction items of other users who have the same interaction items as the user u n , i.e., the third-order nodes, are added to the local training of the user u n Once the server receives the perturbation records uploaded by each user, it can group users with similar preferences, extract the available high-order information of each user u n , and the high-order information consists of two parts: a high-order item list and the frequency of occurrence of each high-order item. Here, the interaction items of other users who have the same interaction items as the user u n are used to enhance the interest representation of the user u n ; After the server sends the extracted high-order information to each target user, federated training of the recommendation model can be performed.

[0035] Further, each client has three different types of embeddings, which are user embedding, rating item embedding and high-order item embedding, the user embedding is used to preserve the privacy features of the user itself and is always saved in the local device, the item embedding is the vector representation of the item features in the hidden space, the rating item embedding can provide information about the user's preference for a specific item, and the high-order item embedding provides information about the indirectly interacted items of the user to enhance the representation of user interest;

[0036] The item embedding is represented as E t ∈R d×M maintained by the server, each user u n has its own embedding representation maintained by the user itself;

[0037] For a user u n , its rating items are high-order items are its rating item embedding is high-order item embedding is respectively represent the total number of user rating items and high-order items, high-order item embedding and rating item embedding and user embedding as the input of the local model to train the recommendation model, and learn and predict user preferences, the method is as follows: use different GAT layers to learn the corresponding embedding weights, and aggregate the three types of embeddings:

[0038] Among them, for users and rating items (u n ,t k ), the attention score s nk is

[0039]

[0040] where W1∈R d×d is a linear mapping matrix, is the embedding representation of the user itself, is the embedding representation of the user rating items, the Attention function represents the attention layer, more specifically, the attention layer is defined as

[0041]

[0042] where a∈R 2d is a weight vector, and || represents the concatenation operation of two vectors;

[0043] For the rating items of a user u n , the attention weight a nk is calculated using the softmax function as follows:

[0044]

[0045] where s nk is the attention score of the user and the rating item;

[0046] The weight β np of the high-order term is defined as:

[0047]

[0048] where represents the number of repetitions of the high-order term p n for the user u , and β is the attention score of the user, i.e., the high-order item, which uses the same calculation mode as the attention score of the user and the rating item;

[0049] The information aggregation of the two types of items is as follows:

[0050]

[0051]

[0052] are the hidden embeddings of the aggregated user rating item and high-order item, respectively, is the embedding representation of the high-order item;

[0053] The two obtained hidden embeddings are calculated using a self-attention mechanism to obtain the current interest representation e agg ;

[0054] The final user representation is calculated as:

[0055]

[0056]

[0057] H = {u n , agg}:

[0058] H is the subscript set, which is used to identify the current e h , Att represents the attention mechanism calculation, and e h represents and e agg two parts;

[0059] When predicting user preferences, the final embedding representation of the learned user u n is given as and the embedding representation of the item t is e twherein the items t include user-interacted items and non-interacted items, then the user u n score the item t is calculated as follows:

[0060]

[0061] wherein T represents transposition.

[0062] Further, in the federated training process, the real user-item interaction data is saved in the local device for local model training; the user embedding learned by the local model is only saved by the local device and is not uploaded to the server, only the model parameters and item embedding are uploaded to the server for aggregation.

[0063] Further, in the federated training process, the protection of the user interaction record is realized by using a pseudo-interaction item sampling strategy: in the local client, Q items are sampled as pseudo-items, denoted as that is, Q items that the user has never interacted with and have not appeared in the high-order item list, the prediction score of the pseudo-item is obtained by rounding the current local model calculation; according to the difference between the prediction score and the rounded score, the pseudo-item embedding can be updated in the local device, then in the user local, three types of item embeddings are obtained, respectively, the embedding of the item that the user has interacted with, the high-order item embedding, and the pseudo-item embedding; in the federated aggregation stage, the three types of item embeddings, i.e., the user rating item embedding, the pseudo-item embedding and the high-order item embedding, are uploaded together, so as to hide the list of items that the user has really interacted with.

[0064] The application also provides a communication-efficient privacy protection recommendation system for implementing the recommendation method as described above, the system comprising a plurality of user clients and a central server, and the data transmission mode of each client and the server is as follows:

[0065] The local device of each user client uploads the server after perturbing the local interaction data thereof by using differential privacy technology;

[0066] The server end processes the obtained user perturbed data, calculates the similarity of the noise data in the presence of noise, and obtains the similarity between users; the server uses the calculated user similarity matrix to group users with the same interest, extracts the high-order information required for model training on the server after user individual grouping; the server sends the extracted high-order information to each user; repeat the following steps to complete the training of the recommendation model: the server distributes the global model to each client, each client trains the local model by using the local data and the received high-order information, and each client uploads the updated parameters of the local model to the server, and the server aggregates the received model parameters of each group to update the global model on the server.

[0067] Compared with the prior art, the present application has the advantages of:

[0068] (1) The user real data is not uploaded to the server, only the data protected by adding disturbance is uploaded, realizing the protection of user real preference and the protection of interaction record

[0069] (2) The user similarity calculation and user grouping based on user disturbance data are used for personalized model training to improve the model training accuracy.

[0070] (3) The introduction of high-order information introduces more data information for each user device, increases the local data quantity, more fully uses the computing power of the user device, and improves the model training speed.

[0071] (4) The designed recommendation model based on graph neural network can better utilize high-order information to model user interest and more accurately predict user preference.

[0072] (5) Further protect user privacy, the user embedding learned by the local model is only saved by the local device and is not uploaded to the server, only the model parameters and item embedding are uploaded to the server for aggregation. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0074] Figure 1 The method flowchart of the present application;

[0075] Figure 2 The federal training schematic diagram of the present application;

[0076] Figure 3 The recommendation model training schematic diagram of the present application;

[0077] Figure 4 The data disturbance schematic diagram of the present application;

[0078] Figure 5 The user grouping schematic diagram of the present application;

[0079] Figure 6 The high-order information acquisition schematic diagram of the present application;

[0080] Figure 7 The pseudo-interaction item sampling strategy schematic diagram of the present application;

[0081] Figure 8 A schematic diagram of data transmission mode of each client and server of the recommendation system of the present application. DETAILED DESCRIPTION

[0082] The present application will be further described below in conjunction with the drawings and specific embodiments.

[0083] The design idea of the present application is: a federated learning framework is designed, real user-item interaction data is saved in the terminal device, and is used for local model training. Before federated learning, participants are grouped individually, members in each group jointly train a recommendation model, rather than training a unified global model, and a high-order embedding information within each group is extracted for each user. The individualized grouping method is based on public user information (such as age, gender or nationality) and perturbed user interaction data in the server. In addition, the present application designs a recommendation model based on a graph neural network to capture user interest and realize prediction of user preference. In order to further protect user privacy, the user embedding learned by the local model is only saved by the local device and is not uploaded to the server, only the model parameters and item embedding are uploaded to the server for aggregation.

[0084] In combination Figure 1 As shown in the present embodiment, a communication-efficient privacy protection recommendation method is provided, comprising the following steps:

[0085] Step 1, a recommendation model based on a graph neural network is constructed to capture user interest and realize prediction of user preference, and the recommendation model is federated trained; the recommendation model includes an embedding representation layer of users and items, a user interest extraction module based on a graph neural network, a user preference score calculation layer, and in addition, a high-order information extraction module is also provided on the server side; the recommendation model has two types of local models on each user client and global models on the central server, in order to facilitate the distinction in federated learning, according to the different devices where the recommendation model is located, the model held by the user client can be recorded as a local model, and the model held by the central server is a global model. The embedding representation layer of users and items is used to extract the embedding representation of users and items, the user interest extraction module based on the graph neural network is used to extract the interest of the user, the user preference score calculation layer is used to calculate the score of the user preference, predict the user preference, and the high-order information extraction module is used to extract the high-order embedding information. The network structure of each module is not the design point of the present application, and will not be described in detail here. The recommendation model training task is completed by the user client and the central server jointly, the user client uses the local data to train and update the recommendation model, the central server aggregates and updates the local model uploaded by each user client to obtain the latest global model and the updated distribution, such as Figure 3 as shown;

[0086] Step 2: For a cold start user, obtain the user embedding and item embedding obtained from federated training;

[0087] Step 3: Feed the user embeddings and the embeddings of items that the user has not interacted with into the trained recommendation model to obtain the user's interest score for items that have not been interacted with.

[0088] Step 4: Sort and recommend items according to the user's interest score for each item output by the recommendation model.

[0089] The following section provides a detailed introduction to each part.

[0090] Combination Figure 2 As shown, the federated training process includes:

[0091] The first stage involves user data protection: Each user client perturbs and adds protection to its local interaction data before transmission. This perturbed data is then uploaded to the central server for calculating user behavior similarity and user grouping. Members within each group jointly train a recommendation model, rather than all users training a global model. Simultaneously, the central server extracts high-order embedding information for each user within their group. After completing user grouping and extracting high-order information, the central server sends the high-order information, item embedding table, and model parameters to each group member. User embeddings, item embeddings, and the recommendation model are then trained through federated learning.

[0092] In the second stage, each client uses local data and received higher-order information to train the model locally. After several rounds of model training, the model is uploaded to the central server. The central server aggregates the local models uploaded by each client and updates the global model. This second stage is repeated multiple times until the global model is available.

[0093] The specific steps for user data protection processing are as follows: Define U = {u1, u2, ..., u...} N} and T={t1,t2,…,t M} represents the user set and the item set, respectively, where N is the number of users, M is the number of items, and u is the number of users. n For project t m The rating is R nm Then the rating matrix formed by all users' ratings of the project is R∈R N×M Each user stores their rating data locally, therefore the rating matrix is ​​unavailable to the server; client c is defined as a local device that stores user rating data and actual user interaction data, and each client c n Associate a user u n Its actual rating data and interaction data are R n_ and h n Each user un There is a local rating record vector Interaction data h is generated using these rating records n , h nm represents the vector h n The mth bit of the converted result is as follows:

[0094]

[0095] Subsequently, the user interaction record of the perturbed version can be generated by a random flipping mechanism.

[0096] This embodiment uses differential privacy technology, given a privacy budget ∈, then each bit in h n is flipped with a probability of q2, and remains the original value with a probability of q1 = 1-q2; this symmetric flipping mechanism satisfies the ε-differential privacy. Figure 4 An example of user perturbed data upload. These perturbed data are uploaded to the server for user behavior similarity calculation and user grouping.

[0097] User behavior similarity calculation and user grouping are performed by the server, and the user behavior similarity is estimated by Jaccard similarity. The similarity function of two users 1 and 2 is defined as follows:

[0098]

[0099] where h1 represents the interaction data of user 1, h2 represents the interaction data of user 2,

[0100] The key to calculating user similarity is to accurately estimate h1∩h2 and h1∪h2 given the perturbed vectors and The perturbed vectors and are the interaction data of h1 and h2 after adding perturbation. For i,j∈{0,1},n ij represents the number of ordered pairs of h1[s] = i and h2[s] = j at the s-th bit position of the real vector, and all n ij constitute the vector Similarly, for i,j∈{0,1},m ij represents the number of ordered pairs of and at the s-th bit position of the perturbed vector, and all m ij constitute the vector Let pr(l→l) = pr(0→0) = q1, which represents the probability that the value at a certain position remains original value, pr(l→0) = pr(0→l) = q2, which represents the probability that the value at a certain position is flipped, then the cardinalities of the union and intersection of the two vectors h1 and h2 can be estimated by the mean field model as follows:

[0101]

[0102] where q1 is the probability that the value at a certain position of the original vector remains original value, and q2 is the probability that the value is flipped.

[0103] By inverting the matrix, the four unbiased estimators of n ij can be obtained as follows:

[0104]

[0105] The number of 00 pairs at corresponding positions in the original vectors h1 and h2 is determined by the vector length m and |h1∪h2|, i.e., n 00 =m-|h1∪h2|, so |h1∪h2| can be estimated as follows:

[0106]

[0107] is the estimated value of |h1∪h2|.

[0108] At the same time, can be directly used as an estimator of h1∩h2,

[0109]

[0110] is the estimated value of h1∩h2.

[0111] Based on this calculation method, the behavior similarity of any two users can be estimated using the uploaded perturbation data. As shown in the figure, a user similarity matrix S ∈ R N×N can be generated. R ij represents the similarity of users u i and u j . The nth row of the similarity matrix S is taken as the behavior feature of user u n in the grouping stage, i.e., the similarity of user u n to all other users.

[0112] Then, the users are divided into different clusters G = {G l | l = 1, …, C} using the k-means clustering algorithm (or other clustering algorithms), where G l is the lth user group, and C is the number of user groups after clustering.Figure 5 An example diagram illustrating user grouping.

[0113] Higher-order information extraction provides the necessary data for rapid model training and improved recommendation accuracy. The higher-order information extraction process is as follows: In a federated learning setup, user data is stored locally on the user's device. n Only user u n Interacted items (first-order nodes) are locally available. User embeddings retain user interests and contain certain privacy information; under strict privacy restrictions, they interact with the user's... n Embedded (second-order nodes) of other users with the same interaction items are not available.

[0114] Therefore, this design will be related to user u n Interaction items (third-order nodes) from other users with the same interaction items are added to user u. n In local training, local training data is enriched to enhance user representations. Once the server receives the perturbation records uploaded by each user, users with similar preferences can be grouped, and available high-level information for each target user can be extracted.

[0115] for Figure 6 Target user u n The higher-order information at l=3 consists of two parts: a list of higher-order items {i4,i5,i6} and the frequency of each higher-order item {1,2,1}. Here, interaction items from other users who have the same interactions with the target user are used to enhance the target user's interest representation. After the server sends the extracted higher-order information to each target user, federated training of the recommendation model can be performed.

[0116] The model architecture is designed to model user interests. After user grouping and extraction of higher-order information, the server sends the higher-order information, item embedding table, and model parameters to each group member. User / item embeddings and recommendation models are trained through federated learning. During federated training, real user-item interaction data is stored on the local device for local model training; user embeddings learned by the local model are only stored on the local device and not uploaded to the server. Only model parameters and item embeddings are uploaded to the server for aggregation.

[0117] Each client has three different types of embeddings: user embeddings, rated item embeddings, and higher-order item embeddings. User embeddings preserve the user's privacy and are always stored locally on the device. Item embeddings are vector representations of item features in the latent space. Rated item embeddings can provide information about a user's preferences for specific items. Higher-order item embeddings provide information about items that the user has indirectly interacted with to enhance the representation of user interests.

[0118] Item embedding is denoted as E t ∈R d×M , which is maintained by the server. Each user u n has its own embedding representation which is maintained by the user itself.

[0119] For a user u n , its rating items are The high-order item (e.g. l=3) term is Its rating item embedding is The high-order item embedding is K, P represent the total number of user rating items and high-order items respectively. denotes the embedding of user u n . The high-order item embedding rating item embedding and user embedding These embeddings are used as inputs of local GNN model to train the recommendation model and learn user / item embeddings and predict user preferences.

[0120] To obtain a comprehensive representation of user preference interest, it is necessary to aggregate the three types of embeddings. The present scheme uses different GAT layers to learn the corresponding embedding weights. For users and rating items (u n , t k ), the attention score s nk is

[0121]

[0122] where W1∈R d×d is a linear mapping matrix, is the embedding representation of the user itself, is the embedding representation of the user rating item, and the Attention function represents the attention layer. More specifically, the attention layer is defined as

[0123]

[0124] where a∈R 2d is a weight vector, and || represents the concatenation operation of two vectors.

[0125] For the rating items of user u n , the attention weight a nk is calculated using the softmax function as follows:

[0126]

[0127] where a nk is the final attention weight, and snk The attention score of the user and the rating item.

[0128] Since a user can have multiple same high-order items, the occurrence times of the high-order items need to be considered when extracting the attention weight. Therefore, the weight β of the high-order item is defined as np The calculation is as follows:

[0129]

[0130] Wherein represents the repetition times of the high-order item p of the user u n . The attention score of the user, i.e. the high-order item, is the same as the attention score of the user and the rating item, and the same calculation mode is adopted.

[0131] The information aggregation of the two types of items is as follows:

[0132]

[0133]

[0134] The hidden embedding of the aggregated user rating item and high-order item, respectively, is the embedding representation of the high-order item.

[0135] Since there is a hierarchical relationship between items of different types in the high-order connectivity graph, the contributions of the rating item embedding and the high-order item embedding to the learning process are not the same, and this heterogeneity should be handled in the aggregation step. Here, a self-attention mechanism is used to calculate the two obtained hidden embeddings to obtain the representation of the current GAT layer, i.e. the current interest representation e agg .

[0136] The final user representation is calculated as follows:

[0137]

[0138]

[0139] H is the index set, which is used to identify the current e h . h Att represents the attention mechanism calculation, e agg represents and e n two parts.

[0140] When predicting the user's preference, given the final embedding representation of the learned user u t and the embedding representation e n of the item t (user interaction item or non-interaction item), the user un score for item t

[0141]

[0142] where T denotes transpose. It is noted that item t is a large category, including user-interacted item t m and non-interacted item.

[0143] As shown in Figure 7 , the protection of user interaction records in model training is implemented by using a pseudo-interaction item sampling strategy. In the local client, Q items are sampled as pseudo items, denoted as , i.e. Q items that the user has never interacted with and have not appeared in the high-order item list. The predicted score of the pseudo item is obtained by rounding the score calculated by the current local model. According to the difference between the predicted score and the rounded score, the pseudo item embedding can be updated in the local device, and then three types of item embeddings are obtained in the user's local device, which are the embedding of the item that the user has interacted with, the embedding of the high-order item, and the embedding of the pseudo item. In the federal aggregation stage, the three types of item embeddings (i.e. user score item embedding, pseudo item embedding and high-order item embedding) are uploaded together, so as to hide the list of items that the user has really interacted with.

[0144] As another embodiment, a communication-efficient privacy-preserving recommendation system is also provided, which includes two types of devices, including a plurality of user clients and a central server, and the data transmission mode between the clients and the server is combined Figure 8 , wherein the data transmission between the client and the server in the data preprocessing stage is only performed once, and the transmission of the model parameters in the training needs to be performed multiple times.

[0145] The local device of each user client uses differential privacy technology to perform perturbation processing on its own local interaction data (such as rating data for movies) and uploads to the server (the server cannot judge the real interaction items of the user and the real rating data of the user for the items).

[0146] The server side processes the obtained user disturbance data, denoises the similarity of the noise data in the presence of noise, and obtains the similarity between users; the server uses the calculated user similarity matrix to perform personalized grouping of users with the same interest, extracts high-order information required for model training on the server after user personalized grouping; the server sends the extracted high-order information to each user; repeat the following steps to complete the training of the recommendation model (repeat steps 6-9 multiple times): the server distributes the global model to each client, each client uses local data and received high-order information to train the local model, and each client uploads the updated parameters of the local model to the server, and the server aggregates the received model parameters of each group to update the global model on the server.

[0147] In order to make the above-mentioned embodiments of the present application more understandable to those skilled in the art, the following will be described by taking video recommendation as an example.

[0148] First, the user local device retains the specific ratings of the user to the video, the user local device converts the specific rating record (such as Douban rating 1-5) into an interaction record (0, 1 mark whether watched, 1 for watched, 0 for not watched), then uses differential privacy technology to add disturbance to the interaction record, that is, to randomly flip the 0, 1 record with a certain probability (0 flips to 1, 1 flips to 0), and then uploads the disturbed record to the server. After the server obtains the user's disturbed record, it calculates the user similarity and performs personalized grouping of the users using the method proposed in the present application. After grouping, the high-order information required for model training is extracted for each group of users and distributed. The server distributes the initial recommendation model to each client, and the users in each group work together to train and update the model using their local data, and after multiple rounds of training, each group can obtain a recommendation model with better performance, realizing the prediction of the preferences of the users in the group.

[0149] In summary, the user real rating data server of the prior art cannot be obtained, but it can be inferred that the user has interacted with which items, and the present application innovatively performs personalized grouping of users on the server using user disturbance records, extracts high-order information from the obtained user groups for recommendation model training. It reduces the communication cost of federated training and improves the accuracy of the recommendation model. In the technical solution of the present application, both the protection of real rating data and the protection of user real interaction records can be realized.

[0150] In addition, the present application can be applied to the recommendation of movies, videos, books, music, etc. In model design, other models can also be used in addition to graph neural networks, such as deep learning models, etc.

[0151] Of course, the above description is not a limitation of the present application, and the present application is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the spirit and scope of the present application should be within the scope of the present application.

Claims

1. A privacy-preserving recommendation method with high communication efficiency, characterized in that, Includes the following steps: Step 1: Construct a recommendation model based on graph neural networks to capture user interests and predict user preferences, and perform federated training on the recommendation model. The federal training process includes: The first stage involves user data protection: Each user client perturbs and adds protection to its local interaction data before transmission. This perturbed data is then uploaded to the central server for calculating user behavior similarity and user grouping. Members within each group jointly train a recommendation model, rather than all users training a global model. Simultaneously, the central server's high-order information extraction module extracts high-order embedding information within each group for each user. After completing user grouping and extracting high-order information, the central server sends the high-order information, item embeddings, and model parameters to each group member. User embeddings, item embeddings, and the recommendation model are then trained through federated learning. The high-order information extraction process is as follows: In a federated learning setup, user data is stored on the user's device, for user u n Only user u n Interacted items, i.e., first-order nodes, are locally available; user embeddings retain user interests and contain certain private information, and under strict privacy restrictions, interact with the user. n Embeddings of other users with the same interaction items, i.e., second-order nodes, are unavailable; this will be related to user u. n Interaction items from other users with the same interaction items, i.e., third-order nodes, are added to user u. n During local training, once the server receives the perturbation records uploaded by each user, it can group users with similar preferences and extract the perturbation records for each user. n The available higher-order information consists of two parts: a list of higher-order items and the frequency of occurrence of each higher-order item. This is achieved by utilizing the information from user u. n Enhance user experience by providing interactive items from other users with the same interactive items. n Interest representation; after the server sends the extracted high-level information to each target user, it can perform federated training of the recommendation model; In the second stage, each client uses local data and received higher-order information to train the model locally. After several rounds of model training, the model is uploaded to the central server. The central server aggregates the local models uploaded by each client and updates the global model. The second stage is repeated multiple times until the global model is available. The recommendation model includes an embedding representation layer for users and items, a user interest extraction module based on graph neural networks, and a user preference score calculation layer. In addition, the server side also has a high-order information extraction module. In federated training, depending on the device where the recommendation model is located, the model held by the user client is referred to as the local model, and the model held by the central server is the global model. The recommendation model training task is jointly completed by each user client and the central server. The user client uses local data to train and update the recommendation model, and the central server aggregates and updates the local models uploaded by each user client to obtain the latest global model and the updated distribution. Step 2: For a cold start user, obtain the user embedding and item embedding obtained from federated training; Step 3: Feed the user embeddings and the embeddings of items that the user has not interacted with into the trained recommendation model to obtain the user's interest score for items that have not been interacted with. Step 4: Sort and recommend items according to the user's interest score for each item output by the recommendation model.

2. The efficient privacy-preserving recommendation method for communication according to claim 1, characterized in that, The specific steps for user data protection processing are as follows: Define U = {u1, u2, ..., u...} N } and T={t1,t2,…,t M } represents the user set and the item set, respectively, where N is the number of users, M is the number of items, and u is the number of users. n For project t m The rating is R nm Then the rating matrix formed by all users' ratings of the project is R∈R N×M Each user stores their rating data locally, so the rating matrix is ​​not available to the server; Client c is defined as a local device that stores user rating data and actual user interaction data. Each client c n Associate a user u n Its actual rating data and interaction data are R n- and h n Each user u n There is a local rating record vector Use these rating records to generate interaction data h n h nm Represents interactive data h n The method for converting the m-th bit is as follows: Subsequently, user interaction records after the disturbance Generated through a random flipping mechanism.

3. The communication-efficient privacy-preserving recommendation method according to claim 1, characterized in that, User behavior similarity calculation and user grouping are performed by the server. User behavior similarity is estimated using Jaccard similarity. The similarity function for two users, 1 and 2, is defined as follows: Where h1 represents user 1's interaction data, and h2 represents user 2's interaction data. The key to calculating user similarity lies in the given perturbation vector and In the case of h1∩h2 and h1∪h2, an accurate estimate of the perturbation vector is obtained. and These are the perturbation-added interaction data for h1 and h2, respectively. For i,j∈{0,1}, n ij Let h1[s] = i and h2[s] = j be the number of ordered pairs at the s-th bit position of the real vector. All n ij Constructing vectors Similarly, for i,j∈{0,1}, m ij This indicates that at the s-th bit position of the perturbation vector and The number of ordered pairs, all m ij Constructing vectors Let pr(1→1)=pr(0→0)=q1, representing the probability that the value at a certain position remains the original value is q1, and pr(1→0)=pr(0→1)=q2, representing the probability that the value at a certain position is flipped is q2. Then the cardinality of the union and intersection of vectors h1 and h2 can be estimated using the mean-field model as follows: The probability that the value at a certain position of the original vector remains the original value is q1, and the probability of flipping it is q2. By inverting the matrix, we can obtain n. ij The four unbiased estimators: The number of 0-0 pairs at corresponding positions in the original vectors h1 and h2 is determined by the vector length m and |h1∪h2|, i.e., n 00 =m-|h1∪h2|, then |h1∪h2| can be estimated as: This is an estimate of |h1∪h2|; at the same time, It can be directly used as an estimator of h1∩h2. Let h1∩h2 be the estimated value; Based on this calculation method, using the uploaded perturbation data, the behavioral similarity between any two users can be estimated, generating a user similarity matrix S∈R. N×N The nth row of the similarity matrix S is taken as user u n Behavioral characteristics during the grouping phase, i.e., user u n Similarity to all other users; Then, a clustering algorithm is used to divide the users into different clusters G = {G}. l |l=1,…,C}, where G l Let l be the l-th user group, and C be the number of user groups after clustering.

4. The communication-efficient privacy-preserving recommendation method according to claim 1, characterized in that, Each client has three different types of embeddings: user embedding, rating item embedding, and higher-order item embedding. User embedding is used to preserve the user's own privacy features and is always stored on the local device. Item embedding is a vector representation of item features in the latent space. Rating item embedding can provide information about the user's preferences for specific items. Higher-order item embedding provides information about items that the user interacts with indirectly to enhance the representation of user interests. Project embedding is represented as E t ∈R d×M Maintained by the server, each user u n Each has its own embedded representation Maintained by the user themselves; For user u n Its scoring items are Advanced projects are Its rating items are embedded as Higher-order terms are embedded as K and P represent the total number of user rating items and higher-order items, respectively. The higher-order items are embedded... and rating item embedding and user embedding As input to the local model for training the recommendation model, and for learning user / item embeddings and predicting user preferences, the method is as follows: different GAT layers are used to learn the corresponding embedding weights, and these three types of embeddings are aggregated: Among them, for users and rating items (u n ,t k Attention score s nk for: Where W1∈R d×d It is a linear mapping matrix. For the user's own embedded representation, For the embedding representation of user rating items, the Attention function represents the attention layer; more specifically, the attention layer is defined as... Where a∈R 2d It is a weight vector, and || represents the concatenation operation of two vectors; For user u n The scoring items are used to calculate the attention weight α using the softmax function. nk as follows: Among them, s nk Attention score for users and rating items; Define the weight β of the higher-order term np The calculation is as follows: in Indicates user u n The number of repetitions of the higher-order term p, The user's attention score for higher-level items uses the same calculation method as the user's attention score for rating items; The information from the two types of projects is aggregated as follows: These are the hidden embeddings of the aggregated user rating items and higher-order items, respectively. This represents the embedding of higher-order terms; A self-attention mechanism is used to compute the hidden embeddings obtained from the two classes to obtain the current interest representation e. agg The end user said The calculation method is as follows: H={u n ,agg} H is the set of subscripts used to identify the current e. h The reference to , Att represents attention mechanism computation, e h express and e agg Two parts; When predicting user preferences, given the learned user u n The final embedding representation And the embedded representation of item t, e t Where item t includes interactive items and non-interactive items, then user u n t-scoring of the project The calculation is as follows: Where T represents transpose.

5. The communication-efficient privacy-preserving recommendation method according to claim 1, characterized in that, During federated training, real user-item interaction data is stored on the local device for local model training; user embeddings learned by the local model are only stored on the local device and are not uploaded to the server. Only model parameters and item embeddings are uploaded to the server for aggregation.

6. The communication-efficient privacy-preserving recommendation method according to claim 1, characterized in that, During federated training, protection of user interaction records is achieved using a pseudo-interaction item sampling strategy: in the local client, Q items are sampled as pseudo-items, denoted as... For Q items that users have not interacted with and that do not appear in the higher-order item list, the predicted score of the pseudo-item is obtained by rounding down the current local model. Based on the difference between the predicted score and the rounded score, the pseudo-item embedding can be updated on the local device. There are three types of item embeddings that are updated on the user's local device: the embedding of items that the user has interacted with, the higher-order item embedding, and the pseudo-item embedding. During the federated aggregation phase, three types of item embeddings—user rating item embeddings, pseudo-item embeddings, and higher-order item embeddings—are uploaded together to hide the list of items that the user actually interacts with.

7. A highly efficient privacy-preserving recommendation system, characterized in that, To implement the recommended method according to any one of claims 1-6, the system includes multiple user clients and a central server, and the data transmission between each client and the server is as follows: Each user's local device uses differential privacy technology to perturb its local interaction data before uploading it to the server. The server processes the obtained user perturbation data, and calculates the similarity of the noisy data in the presence of noise to obtain the similarity between users. The server uses the calculated user similarity matrix to personalize user groups based on shared interests. After personalizing user groups, the server extracts the high-order information needed for model training. The server then sends the extracted high-order information to each user. The following steps are repeated to train the complete recommendation model: the server distributes the global model to each client, each client trains its local model using local data and the received high-order information, and each client uploads the updated parameters of its local model to the server. The server then groups and aggregates the received model parameters to update the global model on the server.

Images