A privacy-preserving cross-domain recommendation system based on federated representation learning

Abstract

The application discloses a privacy protection cross-domain recommendation system based on federal representation learning, which comprises the following steps: S1: private update in a single domain, that is, user global and local preferences are fused through a graph migration module, and the global and local preferences are locally updated based on the domain data by using a gradient descent algorithm; S2: after the private update, a federal update process is used to cooperatively learn the global user preferences based on multi-domain data, and the preferences of different domains of a user are personalized and aggregated through an attention mechanism, so that the preferences are adapted to the heterogeneous domain data; and S3: a period synchronization mechanism is used to reduce the communication cost, that is, after a fixed number of private updates, a cross-domain federal update process is performed. The application protects the user privacy information from multiple aspects. In the application, the original interaction data of each domain is locally stored in the private space of the domain and is not uploaded to other domains, so that the risk of privacy leakage can be effectively reduced.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence natural language processing and recommendation systems, and in particular to a privacy-preserving cross-domain recommendation system based on federated representation learning. Background Art

[0002] In modern recommendation systems, developing multi-domain recommendation services to meet diverse user needs has become a technological trend. To improve the multi-domain services of information systems, cross-domain recommendation issues are gaining increasing research and industry attention. The goal is to leverage useful information from other domains to improve the quality of recommendations in the target domain. Typical cross-domain recommendation methods primarily establish connections between domains through overlapping users / items and transfer useful information between these domains. In this study, we focus on collaborative cross-domain recommendation without auxiliary interaction data. Collaborative cross-domain recommendation tends to apply classic machine learning techniques to directly share or indirectly map user / item embeddings or capture common patterns across domains.

[0003] Although existing cross-domain recommendation methods are effective, they usually rely on a strong assumption that all or part of the user-item interaction data is accessible between different domains. However, due to commercial competition and privacy issues, this assumption may not be realistic in practice. For example, application data from different domains usually belong to different companies or departments, and these data cannot be shared directly. In addition, recent data protection regulations, such as GDPR, have imposed strict restrictions on the storage and sharing of privacy-sensitive user data. The above privacy and security issues severely limit the practical application of existing cross-domain recommendation methods. Recently, some studies have proposed cross-domain recommendation models that consider privacy, but these methods either ignore the heterogeneity of data in different domains or fail to achieve consistent improvements for all users. In response to the above shortcomings of existing cross-domain recommendation methods, the present invention proposes a privacy-preserving cross-domain recommendation system based on federated representation learning.

[0004] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the Invention

[0005] The present invention aims to provide a privacy-preserving cross-domain recommendation system based on federated representation learning. The key concept of this invention is to model a user's global preferences across all domains and their local preferences within a specific domain. Specifically, for each domain, a domain-specific user-item interaction graph is constructed based on the user interaction data for that domain. Global and local user nodes are set to model global and local user preferences, and edges connecting the global and local user nodes are added. Based on this user-item graph, the present invention designs a federated cross-domain recommendation model. To learn cross-domain recommendation knowledge in a privacy-preserving manner, each training iteration of the proposed cross-domain recommendation model consists of a private update process within the local domain and a federated update process across multiple domains. During the private update process, a graph transfer module is designed for each domain to perform bidirectional message exchange and propagation, thereby integrating the user's global and local preferences. Then, during the federated update process, each domain applies local differential privacy (LDP) techniques to the learned global user preferences and shares them with other domains. Simultaneously, each domain receives global user preferences from other domains and then locally updates them through a personalized aggregation strategy, enabling domain-specific adaptation of user preferences. In this way, the cross-domain recommendation model proposed in this paper can effectively approximate the multi-domain training process, which directly shares local interaction data in a privacy-preserving manner. Furthermore, this paper proposes a periodic synchronization mechanism to reduce the communication cost associated with learning cross-domain global preferences.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a privacy-preserving cross-domain recommendation system based on federated representation learning, comprising the following steps:

[0008] S1: Private update within a single domain, that is, fusing the user's global and local preferences through the graph migration module, and locally updating the global and local preferences using the gradient descent algorithm based on the domain data;

[0009] S2: After the private update, a federated update process is used to collaboratively learn global user preferences based on multi-domain data. The attention mechanism is used to aggregate user preferences in different domains to adapt to heterogeneous domain data.

[0010] S3: Use a periodic synchronization mechanism to reduce communication costs, that is, perform a cross-domain federated update process after a fixed number of private updates.

[0011] As a further technical solution, the graph migration module in step S1 is specifically:

[0012] For each domain, a domain-specific user-item interaction graph (e.g. Figure 3 As shown), set global and local user nodes to model global and local user preferences, and add edges connecting global and local user nodes; where the global user node of user u is embedded with The original user node of user u is associated with the embedded user node e u Related, e u and are all vectors of dimension m in real space; the graph transmission module can be abstracted as L-layer transformations; at the l-th layer, a bidirectional embedding transmission is first applied to exchange messages between local and global user preferences, which is calculated as follows:

[0013]

[0014]

[0015] in, f T (·,·|β) is the transfer function, β1 and β2 are hyperparameters in the range [0,1] that control the retention rate during transfer, represents the neighbor set of user u in the graph; and denote the local and global representations of user u before transmission, respectively, where the local user representation In sending

[0016]

[0017] Before reaching the l+1 layer, it will be further updated during message propagation. The specific operations of message propagation are as follows:

[0018] in, Represents the neighbor set of item i in the user-item interaction graph, and the item represents Embedding e through learnability i ∈R m To initialize; after the L-layer transformation in the graph transfer module, we concatenate the representations generated by all L layers to obtain the final user representation and item representation as shown below:

[0019]

[0020] Where Concat(·) represents the concatenation operation; the final user representation h u Global and local user preferences, as well as high-order information on the user-item graph, have been encoded.

[0021] As a further technical solution, in step S1, the global and local preferences are locally updated using a gradient descent algorithm based on the in-domain data as follows:

[0022] Given user and item representations, an inner product operation is used to generate a score to predict the likelihood of user u interacting with item i from domain d, which is calculated as follows:

[0023]

[0024] in is the predicted score for user u and item i; then the Bayesian personalized ranking loss is used to update the local and global user embeddings based on the single-domain interaction data, which is defined as:

[0025]

[0026] Where σ(·) is the Sigmoid function, λ controls the strength of L2 regularization, and Θ d is the model parameter of domain d; the training sample pairs are constructed by negative sampling, represents a set of paired training data; in this way, the model locally learns local user embeddings and domain-specific item embeddings for each domain based on the in-domain data.

[0027] As a further technical solution, in step S2, after the private update, a federated update process is used to collaboratively learn global user preferences based on multi-domain data. Specifically:

[0028] During the private update process, each domain d is a global user Maintenance Embed Global user of Use in-domain data for local updates; In order to characterize more comprehensive user preferences, it is necessary to learn cross-domain knowledge to enhance local user preferences; To this end, decentralized federated learning is used to collaboratively update global user embeddings based on data from multiple domains; These global user embeddings will be shared across multiple domains; However, these user embeddings contain private information about user behavior and cannot be directly shared outside the domain for privacy reasons; Before sharing the global user embeddings, local differential privacy technology is applied to the global user embeddings; Specifically, for each global user Add strength λ LDP Laplace noise to obtain the encrypted embedding Each domain d then sends the protected embedded to other domains, while accepting shared embeddings from other domains

[0029] As a further technical solution, in step S2, the global preferences of different fields are personalized and aggregated through the attention mechanism to adapt them to heterogeneous domain data. Specifically:

[0030] In the cross-domain recommendation scenario, in order to consider the domain adaptability of cross-domain knowledge utilization, a personalized preference aggregation strategy is adopted to generate domain-specific global user embeddings for each domain; specifically, based on the attention mechanism, personalized preference aggregation is performed in each domain; when the domain d receives shared embeddings from other domains, The personalized aggregation layer uses a learnable transformation matrix W∈R 2m Calculate the attention coefficient. The specific calculation method is:

[0031]

[0032] Where g(·) is the LeakyReLU activation function, and the softmax function is used to normalize the attention coefficient; the attention coefficient α d,d′ Indicates the importance of the domain knowledge to the domain d; these attention coefficients are used as weights for personalized federated aggregation to generate the global embedding of user u in the domain d. The specific generation method is as follows:

[0033]

[0034] Among them, β3 is a hyperparameter that controls the retention rate in the range of [0,1]. The second term of the equation adaptively combines knowledge from different fields; for users Generated global user embeds Private updates will be made via the graph transfer module in the next round of training.

[0035] As a further technical solution, step S3 is specifically as follows:

[0036] When each domain receives the protected global user embeddings shared by other domains, personalized preference aggregation is applied to generate domain-specific global user embeddings; then, given these global user embeddings and the local user embeddings maintained locally, a private update process is first performed times to optimize the local BPR loss function, and then a global update is called.

[0037] By adopting the above technical solution, the present invention has the following beneficial effects:

[0038] Compared to existing cross-domain recommendation systems, the privacy-preserving cross-domain recommendation system based on federated representation learning proposed in this paper protects user privacy information at multiple levels. First, in our approach, the raw interaction data for each domain is stored locally in the domain's private space and is not uploaded to other domains, effectively reducing the risk of privacy leakage. Second, due to data processing inequality, the only data communicated between domains is the user's global embedding, which contains much less private information than the raw interaction data. Furthermore, the user global embedding encodes the user's global preferences across domains rather than domain-specific preferences, making uploading the global embedding more secure. Third, the user global embedding is updated based on a set of user interactions rather than a single user interaction, which makes it difficult to recover specific interaction histories. Fourth, local differential privacy (LDP) technology is applied to the user global embedding, adding zero-mean Laplace noise to it, increasing the difficulty of inferring private information. Therefore, the proposed method can model users' global preferences for recommendation without leaking their privacy, making it more secure than existing cross-domain recommendation systems.

[0039] Compared to existing privacy-conscious recommendation systems that learn unified cross-domain information for all domains, the method proposed in this paper uses an attention mechanism to perform personalized aggregation of information from different domains, thereby enabling each domain to perform domain-specific adaptation based on the received global user preferences. Due to the heterogeneity of cross-domain data in the real world, this domain-specific adaptation is necessary. In addition, the periodic synchronization mechanism proposed in this paper reduces communication costs. This mechanism performs a cross-domain federated update process after a fixed number of private updates. Compared with existing privacy-conscious cross-domain recommendation systems, the method proposed in this paper has lower communication costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is the overall architecture diagram of the privacy-preserving cross-domain recommendation system based on federated representation learning of the present invention;

[0042] Figure 2 This is a flow chart of the algorithm in one update cycle within a single domain of the present invention;

[0043] Figure 3 A domain-specific user-item interaction graph constructed for the present invention. DETAILED DESCRIPTION

[0044] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0046] Combine Figure 1-2 As shown, this embodiment provides a privacy-preserving cross-domain recommendation system based on federated representation learning, including the following steps:

[0047] S1: Private updates within a single domain. This involves fusing a user's global and local preferences through a graph migration module and updating them locally using a gradient descent algorithm based on in-domain data. The primary purpose of the private update process is to locally fuse a user's global and local preferences and update them based on in-domain data to capture domain-specific knowledge. Specifically, this paper proposes a graph migration module to fuse a user's global and local preferences and to locally update them using a gradient descent algorithm based on in-domain data.

[0048] S2: After the private update, a federated update process is used to collaboratively learn global user preferences based on multi-domain data. The attention mechanism is used to aggregate user preferences in different domains to adapt to heterogeneous domain data.

[0049] S3: Use a periodic synchronization mechanism to reduce communication costs, that is, perform a cross-domain federated update process after a fixed number of private updates.

[0050] As a further technical solution, the graph migration module in step S1 is specifically:

[0051] For each domain, a domain-specific user-item interaction graph (e.g. Figure 3 As shown), set global and local user nodes to model global and local user preferences, and add edges connecting global and local user nodes; where the global user node of user u is embedded with The original user node of user u is associated with the embedded user node e u Related, e u and are all vectors of dimension m in real space; the graph transmission module can be abstracted as L-layer transformations; at the l-th layer, a bidirectional embedding transmission is first applied to exchange messages between local and global user preferences, which is calculated as follows:

[0052]

[0053]

[0054] in, f T (·,·|β) is the transfer function, β1 and β2 are hyperparameters in the range [0,1] that control the retention rate during transfer, represents the neighbor set of user u in the graph; and denote the local and global representations of user u before transmission, respectively, where the local user representation Before being sent to layer l+1, it will be further updated in the message propagation. The specific operation of the message propagation is as follows:

[0055] in, Represents the neighbor set of item i in the user-item interaction graph, and the item represents Embedding e through learnability i ∈R m To initialize; after the L-layer transformation in the graph transfer module, we concatenate the representations generated by all L layers to obtain the final user representation and item representation as shown below:

[0056]

[0057] Where Concat(·) represents the concatenation operation; the final user representation h u Global and local user preferences, as well as high-order information on the user-item graph, have been encoded.

[0058] As a further technical solution, in step S1, the global and local preferences are locally updated using a gradient descent algorithm based on the in-domain data as follows:

[0059] Given user and item representations, an inner product operation is used to generate a score to predict the likelihood of user u interacting with item i from domain d, which is calculated as follows:

[0060]

[0061] in is the prediction score for user u and item i; then Bayesian personalized ranking loss is used,

[0062]

[0063] Update local and global user embeddings based on single-domain interaction data, which are defined as:

[0064]

[0065] Where σ(·) is the Sigmoid function, λ controls the strength of L2 regularization, and Θ d is the model parameter of domain d; the training sample pairs are constructed by negative sampling, represents a set of paired training data; in this way, the model locally learns local user embeddings and domain-specific item embeddings for each domain based on the in-domain data.

[0066] As a further technical solution, in step S2, after the private update, a federated update process is used to collaboratively learn global user preferences based on multi-domain data. Specifically:

[0067] During the private update process, each domain d is a global user Maintenance Embed Global user of Use domain data for local updates; In order to characterize more comprehensive user preferences, it is necessary to learn cross-domain knowledge to enhance local user preferences; To this end, decentralized federated learning is used to collaboratively update global user embeddings based on data from multiple domains; These global user embeddings will be shared across multiple domains; However, these user embeddings contain private information about user behavior and cannot be directly shared outside the domain for privacy reasons; Before sharing the global user embeddings, local differential privacy technology (LDP) is applied to the global user embeddings; Specifically, for each global user Add strength λ LDP Laplace noise to obtain the encrypted embedding Each domain d then sends the protected embedded to other domains, while accepting shared embeddings from other domains

[0068] As a further technical solution, in step S2, these global preferences are adapted to heterogeneous domain data through personalized aggregation, specifically:

[0069] In the cross-domain recommendation scenario, in order to consider the domain adaptability of cross-domain knowledge utilization, a personalized preference aggregation strategy is adopted to generate domain-specific global user embeddings for each domain; specifically, based on the attention mechanism, personalized preference aggregation is performed in each domain; when the domain d receives shared embeddings from other domains, The personalized aggregation layer uses a learnable transformation matrix W∈R 2mCalculate the attention coefficient. The specific calculation method is:

[0070]

[0071] Where g(·) is the LeakyReLU activation function, and the softmax function is used to normalize the attention coefficient; the attention coefficient α d,d′ Indicates the importance of the domain knowledge to the domain d; these attention coefficients are used as weights for personalized federated aggregation to generate the global embedding of user u in the domain d. The specific generation method is as follows:

[0072]

[0073] Among them, β3 is a hyperparameter that controls the retention rate in the range of [0,1]. The second term of the equation adaptively combines knowledge from different fields; for users Generated global user embeds Private updates will be made via the graph transfer module in the next round of training.

[0074] As a further technical solution, step S3 is specifically as follows:

[0075] To reduce communication costs, the present invention additionally proposes a periodic synchronization mechanism to reduce communication costs. Periodic Synchronization. In decentralized federated learning, communication bandwidth is a major bottleneck as clients try to communicate their local update information to other clients. To this end, the present invention proposes a periodic synchronization mechanism to reduce communication costs, which performs a cross-domain federated update process after a fixed number of private updates. Specifically, when each domain receives the protected global user embeddings shared by other domains, personalized preference aggregation is applied to generate domain-specific global user embeddings. Then, given these global user embeddings and local user embeddings maintained locally, the private update process is first performed T times to optimize the local BPR loss function, and then a global update is called.

[0076] In summary, the present invention includes at least the following invention points:

[0077] 1. Graph Migration Module: Based on the user-item interaction graph with global and local user nodes, the graph migration module extends the message passing scheme of traditional graph neural networks, while considering: (1) message exchange between global and local user preferences; (2) message propagation on the user-item graph.

[0078] 2. Personalized aggregation module for global user preferences: To consider the domain adaptability of cross-domain knowledge utilization, the personalized aggregation module performs personalized preference aggregation in each domain based on the attention mechanism, thereby generating domain-specific global user embeddings for each domain.

[0079] 3. Periodic synchronization mechanism: To reduce communication costs, the present invention proposes a periodic synchronization mechanism to reduce communication costs. This mechanism performs a cross-domain federated update process after a fixed number of private updates, thereby reducing the communication frequency and thus reducing communication costs.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A privacy-preserving cross-domain recommendation system based on federated representation learning, characterized by: The steps include: S1: Private update within a single domain, that is, fusing the user's global and local preferences through the graph migration module, and locally updating the global and local preferences using the gradient descent algorithm based on the domain data; S2: After the private update, a federated update process is used to collaboratively learn global user preferences based on multi-domain data. The attention mechanism is used to aggregate user preferences in different domains to adapt to heterogeneous domain data. S3: Use a periodic synchronization mechanism to reduce communication costs, that is, perform a cross-domain federated update process after a fixed number of private updates.

2. The privacy-preserving cross-domain recommendation system based on federated representation learning according to claim 1 is characterized in that The graph migration module in step S1 is specifically: For each domain, we construct a domain-specific user-item interaction graph based on the user interaction data of that domain, set global and local user nodes to model global and local user preferences, and add edges connecting global and local user nodes; where the global user node of user u is connected to the embedding The original user node of user u is associated with the embedded user node e u Related, e u and are all vectors of dimension m in real space; the graph transmission module is abstracted as L-layer transformation; at the l-th layer, a bidirectional embedding transmission is first applied to exchange messages between local and global user preferences, which is calculated as follows: in, f T (·,·|β) is the transfer function, β1 and β2 are hyperparameters in the range [0,1] that control the retention rate during transfer, represents the neighbor set of user u in the graph; and denote the local and global representations of user u before transmission, respectively, where the local user representation Before being sent to layer l+1, it will be further updated in the message propagation. The specific operation of the message propagation is as follows: in, Represents the neighbor set of item i in the user-item interaction graph, and the item represents Embedding e through learnability i ∈R m To initialize; after the L-layer transformation in the graph transfer module, we concatenate the representations generated by all L layers to obtain the final user representation and item representation as shown below: Where Concat(·) represents the concatenation operation; the final user representation h u Global and local user preferences, as well as high-order information on the user-item graph, have been encoded.

3. The privacy-preserving cross-domain recommendation system based on federated representation learning according to claim 2 is characterized in that In step S1, the global and local preferences are updated locally using the gradient descent algorithm based on the in-domain data as follows: Given user and item representations, an inner product operation is used to generate a score to predict the likelihood of user u interacting with item i from domain d, which is calculated as follows: in is the predicted score for user u and item i; then the Bayesian personalized ranking loss is used to update the local and global user embeddings based on the single-domain interaction data, which is defined as: Where σ(·) is the Sigmoid function, λ controls the strength of L2 regularization, and Θ d is the model parameter of domain d; the training sample pairs are constructed by negative sampling, represents a set of paired training data; in this way, the model locally learns local user embeddings and domain-specific item embeddings for each domain based on the in-domain data.

4. The privacy-preserving cross-domain recommendation system based on federated representation learning according to claim 1, characterized in that In step S2, after the private update, a federated update process is used to collaboratively learn global user preferences based on multi-domain data. Specifically: During the private update process, each domain d is a global user Maintenance Embed Global user of Use in-domain data for local updates; in order to characterize more comprehensive user preferences, it is necessary to learn cross-domain knowledge to enhance local user preferences; To this end, decentralized federated learning is adopted to collaboratively update global user embeddings based on data from multiple domains; These global user embeddings will be shared across multiple domains; however, these user embeddings contain private information about user behavior and cannot be directly shared outside the domain for privacy reasons; before sharing the global user embeddings, local differential privacy techniques are applied to the global user embeddings; specifically, for each global user Add strength λ LDP Laplace noise to obtain the encrypted embedding Each domain d then sends the protected embedded to other domains, while accepting shared embeddings from other domains 5. The privacy-preserving cross-domain recommendation system based on federated representation learning according to claim 1 is characterized in that In step S2, these global preferences are adapted to heterogeneous domain data through personalized aggregation as follows: In the cross-domain recommendation scenario, to consider the domain adaptability of cross-domain knowledge utilization, a personalized preference aggregation strategy is adopted to generate domain-specific global user embeddings for each domain; Specifically, based on the attention mechanism, personalized preference aggregation is performed in each field; When domain d accepts shared embeds from other domains The personalized aggregation layer uses a learnable transformation matrix W∈R 2m Calculate the attention coefficient. The specific calculation method is: Where g(·) is the LeakyReLU activation function, and the softmax function is used to normalize the attention coefficient; the attention coefficient α d,d′ Indicates the importance of the domain knowledge to the domain d; these attention coefficients are used as weights for personalized federated aggregation to generate the global embedding of user u in the domain d. The specific generation method is as follows: Among them, β3 is a hyperparameter that controls the retention rate in the range of [0,1]. The second term of the equation adaptively combines knowledge from different fields; for users Generated global user embeds Private updates will be made via the graph transfer module in the next round of training.

6. The privacy-preserving cross-domain recommendation system based on federated representation learning according to claim 1, characterized in that Step S3 is specifically as follows: When each domain receives the protected global user embeddings shared by other domains, personalized preference aggregation is applied to generate domain-specific global user embeddings; Then, given these global user embeddings and local user embeddings maintained locally, a private update process is first performed T times to optimize the local BPR loss function, and then a global update is called.

Images