A Federated Knowledge Graph Representation Learning Method and System Based on a Reconstruction Network

Abstract

The present invention discloses a federated knowledge graph representation learning method based on a reconstruction network, including initially defining a federated knowledge graph; constructing a global model M located on a server S; reconstructing a neural network of a defined client entity; obtaining the global model from the server and restoring the low-dimensional entity vector therein to a local entity vector through an entity reconstruction network; and performing federated training by combining the server global model, the local reconstruction neural network of the client, and a knowledge graph representation learning loss function. The present invention enables the knowledge graph representation learning model to adaptively learn the features of local data according to the characteristics of the knowledge graph, and at the same time, its low-dimensional entity vector can also effectively reduce the communication complexity between the server and the client, achieving more effective federated knowledge graph learning.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular, to a federated knowledge graph representation learning method and system based on a reconstruction network. Background Art

[0002] Existing knowledge graph federated learning models are mostly based on a server-client architecture. A global model is defined on the server side and distributed to the client for training, and finally, the global models of all clients are aggregated as the final model. However, the above models ignore the knowledge heterogeneity of the knowledge graph on different clients. Therefore, some methods use two separate models, a local model and a global model, to alleviate data heterogeneity. Among them, the local model usually associates with the global model by means of tensor decomposition, dictionary indexing, linear mapping, etc. However, such methods are limited by the association operation and are only applicable to specific graph structures and cannot be extended to knowledge graphs in more fields. A knowledge graph is composed of entities (nodes) and relationships (edges) between entities. Its structured data form can effectively represent a large amount of general / domain knowledge and can provide a concise indexing method. Knowledge graph representation learning maps the entities and relationships it contains into a low-dimensional vector space and ensures that the vector representations of entities and relationships can reflect the structural information of the knowledge graph. The vector representations of entities and relationships further improve the convenience of using the knowledge graph in downstream tasks (such as recommendation systems, knowledge discovery, question answering systems) and support tasks including graph completion, reasoning, error correction, etc.

[0003] Generally, a knowledge graph is stored in a large graph database. However, in real life, many knowledge graphs are scattered across multiple clients, such as the account information of users in different banks and the purchase records of customers on different platforms. Moreover, due to reasons such as information security and data restrictions, clients cannot share their data with other clients. In this case, since a single client only contains a partial knowledge graph, traditional knowledge graph representation learning methods cannot be directly applied. To solve this problem, the federated learning architecture is introduced into knowledge graph representation learning to achieve representation learning without accessing the complete data.

[0004] At the same time, knowledge graph representation learning needs to map the knowledge graph into a low-dimensional vector space. For the scenario where the knowledge graph is distributed across multiple clients, the representation learning of the knowledge graph can be achieved without accessing all the data through a federated learning-based framework. However, the data type of the knowledge graph is usually graph data, which is very different from the images, texts, etc. processed by traditional federated learning. Therefore, the federated learning model needs to be designed specifically.

[0005] Existing federal knowledge graph representation learning is divided into two main types. The first type adopts a client-client federated learning architecture, that is, a client can share its local knowledge graph model with other trusted clients, and each client can use its local data and the models obtained from other clients to complete training. If the connection relationship between clients is more than one minimum spanning tree containing all clients, the models of each client can finally obtain complete knowledge graph information.

[0006] The second type adopts a server-client federated learning architecture, which additionally introduces a server that can obtain the models of all clients. During the learning process, the server defines a global model and distributes the global model to each client. The client trains the global model according to the local knowledge graph and returns the trained global model to the server. Subsequently, the server uses a model aggregation algorithm to aggregate all the received global models into a unified global model. Considering that the knowledge graph shows high heterogeneity among different clients, some methods additionally define local models independent of the global model. Thus, the global model learns the features shared among different clients, while the local model learns the unique features of the local knowledge graph. These local models usually associate with the global model in ways such as tensor decomposition, dictionary indexing, and linear mapping to ensure the consistency of local and global information.

[0007] However, these association methods are limited by their operational characteristics and can only learn the association relationships on specific types of graph structures and cannot be extended to other types of knowledge graphs. At the same time, the definition of these association methods is difficult and the calculation is complex, which will reduce the training efficiency of the entire model. Summary of the Invention

[0008] One of the purposes of the present invention is to provide a federated knowledge graph representation learning method based on a reconstruction network to solve the problems of high communication complexity, low learning effect, and poor adaptability between the server and the client in distributed knowledge graph representation learning under the federated learning architecture.

[0009] The present invention is realized through the following technical solutions. A federated knowledge graph representation learning method based on a reconstruction network includes the following steps:

[0010] S100, perform an initial definition on the federated knowledge graph. The initial definition includes defining the data format, type, and learning scenario of the federated knowledge graph, and defining the server and the client according to the differences in the learning scenario; S200, construct a global model M located at the server S, where M is the global model, is a set of entity representation vectors, R is a set of relationship representation vectors; define the relationship representation vector as an n-dimensional vector, that is, r ∈ R n, the representation vector of the global entity is a vector with less than n dimensions, that is where E is the set of entities, and R is the set of relationships is the low-dimensional global entity representation vector is the dimension of the low-dimensional global entity vector, and n is the dimension of the local entity vector represents any entity; S300, reconstruct the neural network of the defined client entity, where each defined client has its own reconstruction network, that is, each defined client C i has a reconstruction network f i (*), through f i (*) reconstructs the unique features of the local knowledge graph based on the global entity information; S400, obtain the global model from the server, and restore the global low-dimensional entity vector in it to the local entity vector through the entity reconstruction network, that is where e is the local entity vector is the reconstruction network is the global low-dimensional entity vector; S500, perform federated training by combining the server global model, the local reconstruction neural network of the client, and the knowledge graph representation learning loss function. The training process is carried out for c rounds until the model converges, and the result of the model training is verified through the head entity or tail entity prediction problem.

[0011] It should be noted that since the complexity of the global model is linearly related to the dimension of the entity vector, a lower-dimensional global entity vector can effectively reduce the parameters of the global model, and the number of relationships in the knowledge graph is usually much smaller than that of entities, so there is no need to particularly reduce the vector dimension of the relationships. Therefore, from this perspective, the present invention adopts constructing a low-dimensional global entity vector in the server in step 200 to reduce the parameters in the global model and reduce the communication complexity between the server and the client. At the same time, by restoring the global low-dimensional entity vector to the local entity vector through the entity reconstruction network in step S400, the global low-dimensional entity vector representing global information and having low representation ability is mapped to a higher-dimensional vector with local knowledge graph features and strong representation ability, ensuring the accuracy of the training data and avoiding sacrificing the performance of the model while reducing the communication complexity between the server and the client.

[0012] Furthermore, the defined server and client may include that the defined server is S and the defined client is C, where C = (C1, C2,..., C m ), and m is the number of clients.

[0013] Further, the federated knowledge graph data can be integrated into a federated knowledge graph dataset, which is represented as G=(E, R, T). Among them, E is the set of entities, R is the set of relationships, and T is the set of triples. Among them, the triple is represented as q=(h, r, t), where q is a single unit group, h is the head entity, t is the tail entity, and r is the relationship between entities. From the above content, the local knowledge graph of the i-th client can be represented as G i 。

[0014] It should be noted that in the representation learning of the federated knowledge graph, entities and relationships are mapped to low-dimensional vectors, that is, e, r ∈ R n , where n is the dimension of the vector, e is the entity vector, r is the vector of the relationship between entities, and R is the set of relationships.

[0015] Further, the step S100 may further include the step of standardizing the data in the federated knowledge graph dataset. The standardization process can adopt distributed Min-Max standardization. Specifically, each participating node locally calculates the minimum and maximum values of its dataset; through the secure multi-party calculation method, aggregates the local minimum and maximum values of each node, so as to calculate the global minimum and maximum values; distributes the global minimum and maximum values to each participating node through the secure communication mechanism; each node uses the global minimum and maximum values to standardize its local data.

[0016] It should be noted that the federated knowledge graph dataset contains a large number of data values of each client at different times. These noisy data need to be removed before training these data. If these data are not processed and the global model operation is performed, it is very likely that the final result will be incorrect or convergence cannot be formed. Therefore, standardizing the data in the federated knowledge graph dataset can effectively improve the consistency of the data in the federated knowledge graph dataset, and distributed Min-Max standardization can scale the data to a fixed range, making the value range of the data fixed, which helps to improve the training efficiency of the federated knowledge graph model. At the same time, it can retain the distribution shape and relative size of the original data, and will not affect the final training result. And distributed Min-Max standardization has extremely high security and can ensure the security of the communication information between the server and the client.

[0017] Further, the step S200 may further include adding a similarity constraint to the entity vector, that is where H is the entity similarity function, e i is the local entity vector with index i, e j is the local entity vector with index j, σ 2 is the parameter of the RBF kernel function, and exp is the natural base number, is the global low-dimensional entity vector with index i, is the global low-dimensional entity vector with index j.

[0018] It should be noted that, since the complexity of the global model has a linear relationship with the dimension of the entity vector, a lower-dimensional global entity vector is adopted, which effectively reduces the parameters of the global model while also resulting in a reduction in the specificity of the entire model. However, the reduction in specificity is not beneficial for the training effect. Therefore, in order to ensure that the entity vector can have sufficient specificity, the present invention adds a similarity constraint to the entity vector here. The purpose is to ensure that the entity vectors in M can have specificity. Through this constraint, it can be ensured that the low-dimensional representation vectors of each entity have large differences, thereby ensuring the accuracy of the final training result.

[0019] Furthermore, the reconstruction network can be a multi-layer graph attention network.

[0020] Furthermore, the calculation method of the entity representation in the l-th layer of the multi-layer attention network is where N is the set of neighbors of e in the knowledge graph, e l is the entity in the l-th layer of the multi-layer attention network, is the entity with index i in the (l - 1)-th layer of the multi-layer attention network, and α i is the attention weight of the i-th neighbor to e.

[0021] Furthermore, the calculation method of α i is: ATT(e i , e) = h T (We i + Ue),

[0022] where e i is the local entity vector with index i, e j is the local entity vector with index j, N is the set of neighbors of e in the knowledge graph, h, W, and U are parameters that can be learned by the neural network, h T is the transpose of h, and We i and Ue are matrix multiplications.

[0023] Furthermore, in S400, the client can be provided with a compression network that can compress the local entity into a low-dimensional entity vector again, that is, by setting the compression network, the reversibility of the entire process can be ensured, and the accuracy of the entire training result can be ensured.

[0024] Further, step S500 may further include the following sub-steps: S510, the server sends the global model M to each client; S520, the client C i performs the operation of converting the global entity vector into a local entity vector using e and r on the local knowledge graph G i and performs multiple rounds of training according to the learning objective function loss to obtain the updated e', r' and f(*); S530, the client re-transforms the trained e' into a low-dimensional entity vector through t-distributed stochastic neighbor embedding (t-SNE) and then, together with r', serves as the global model trained by the client, that is uploads it to the server; S540, on the server side, aggregates the global models uploaded by all clients, that is where M is the aggregated global model, is the updated global parameter of the i-th client.

[0025] Further, model aggregation on the server side may further include the following steps: S541, the server side receives the low-dimensional entity vector re-transformed through the local compression network S542, uses the parameter mini-batch stochastic gradient descent algorithm for weighted averaging, updates the global model and optimizes the low-dimensional entity vector to obtain the formula for optimizing and updating the global model parameters, wt+1 = wt + 1 / N ∑k=1K nkΔwk, where wt is the global model parameter of the t-th round, wt+1 is the global model parameter of the (t + 1)-th round, N is the total data volume, Δwk is the model parameter update on the k-th client, and nk is the local data volume on the k-th client; S543, the server side distributes the updated global model to all clients, the clients train the model according to the local data and generate local model parameter updates, and send the updated parameters to the server; S544, the server performs weighted averaging on the updated parameters sent by all clients to obtain the updated global model parameters.

[0026] It should be noted that the formula for optimizing and updating the global model parameters can be obtained through the following steps. Assume there are K clients, the local data volume on the k-th client is nk, then the total data volume is N = ∑k=1K nk, the model parameter update on the k-th device is Δwk, and the formula for updating the global model parameters can be deduced as wt+1 = wt + 1 / N ∑k=1K nkΔwk.

[0027] Further, the tail entity prediction model can be The head entity prediction model can be g = ||h + r - t||p , which is a triple score. Here, g is the triple score, p is the norm (usually 2), h is the head entity, r is the relation, is the candidate tail entity.

[0028] Furthermore, based on the tail entity prediction model and the head entity prediction model, the learning objective function of the client is: loss = ∑q∈D,q - ∈D - {[g(q) - γ1] + + ∈[γ2 - g(q - )] + , where q is a triple, q - is the negative sample triple, g(q) is the score of triple q, g(q - ) is the score of the negative sample triple q - , D is the knowledge graph triple set, D - is the knowledge graph triple negative sample set, γ1 is the positive sample boundary constant, γ2 is the negative sample boundary constant, and the negative samples can be obtained by standard random negative sampling.

[0029] On the other hand, the present invention provides a federated knowledge graph representation learning system based on a reconstruction network, including an initial definition unit, a global model construction unit, a neural network reconstruction unit, a vector transformation unit, and a training unit.

[0030] Among them, the initial definition unit is configured to initially define the federated knowledge graph. The initial definition includes defining the data format, type, and learning scenario of the federated knowledge graph, and defining the server and the client according to the differences in the learning scenarios.

[0031] The global model construction unit is connected to the initial definition unit and is configured to construct the global model M located on the server S, where M is the global model, is the set of entity representation vectors, and R is the set of relation representation vectors; the relation representation vector is defined as an n-dimensional vector, that is, r ∈ R n , and the representation vector of the global entity is a vector lower than n dimensions, that is, where E is the set of entities, R is the set of relations, is the low-dimensional global entity representation vector, n is the dimension of the low-dimensional global entity vector, n is the dimension of the local entity vector, represents any entity.

[0032] The neural network reconstruction unit is connected to the global model construction unit and is configured to reconstruct the neural network of the defined client entity, where each defined client has its own reconstruction network, that is, each defined client C i has a reconstruction network f i (*), and through f i (*) reconstructs the unique features of the local knowledge graph based on the global entity information.

[0033] The global entity vector is connected to the neural network reconstruction unit and is configured to obtain the global model from the server and restore the low-dimensional entity vector in it to the local entity vector through the entity reconstruction network, that is where e is the local entity vector, is the reconstruction network; is the low-dimensional entity vector.

[0034] The training and validation unit is connected to the vector transformation unit and is configured to perform federated training by combining the server global model, the local reconstruction neural network of the client, and the knowledge graph representation learning loss function. The training process is carried out for c rounds until the model converges, and the results of the model training are verified through the head entity or tail entity prediction problem.

[0035] Furthermore, the training and validation unit may further include a model distribution subunit, a function training subunit, a vector update subunit, and a global model aggregation unit.

[0036] Among them, the model distribution subunit is configured to the server to send the global model M to each client.

[0037] The function training subunit is connected to the model distribution subunit and is configured to, for client C i execute the operation of converting the global entity vector into the local entity vector and perform multiple rounds of training on the local knowledge graph G using e and r according to the learning objective function loss to obtain the updated e', r' and f(*). i

[0038] The vector update subunit is connected to the function training subunit and is configured to the client to re-transform the trained e' into the low-dimensional entity vector through t-distributed stochastic neighbor embedding (t-SNE) and then, together with r', serve as the global model trained by the client, that is upload it to the server.

[0039] The global model aggregation unit is connected to the vector update subunit and is configured to aggregate all the global models uploaded by the clients on the server side, that is Where M is the aggregated global model, are the global parameters updated by the i-th client.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] 1. The present invention ensures the specificity of entity vectors by adding a similarity constraint to the entity vectors. Through this constraint, it can be ensured that the low-dimensional representation vectors of each entity have large differences, thus ensuring the accuracy of the final training result. At the same time, a global model with high representation ability is provided through the specific low-dimensional global representation vectors;

[0042] 2. The present invention reconstructs a neural network locally through a global model by constructing a global model, and constructs a local model that can adapt to the local knowledge graph from the global model, and combines the loss function of knowledge graph representation learning. After multiple rounds of training, the final federated knowledge graph representation learning model is obtained, realizing the reduction of the communication complexity between the server and the client.

[0043] 3. The present invention improves the training efficiency of the federated knowledge graph model by standardizing the data in the federated knowledge graph dataset, and can retain the distribution shape and relative size of the original data at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, form a part of this application, and do not constitute a limitation to the embodiments of the present invention. In the drawings:

[0045] Figure 1 is a flowchart provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations.

[0047] Embodiment 1

[0048] Figure 1 A flowchart of this embodiment is shown. This embodiment provides a federated knowledge graph representation learning method based on a reconstruction network.

[0049] Step 1: Initially define the federated knowledge graph. The initial definition includes defining the data format, type, and learning scenario of the federated knowledge graph, and defining the server and the client according to the differences in the learning scenario.

[0050] In this embodiment, the server can be defined as S, and the client can be defined as C, where C = (C1, C2,..., C m ), and m is the number of clients.

[0051] Then, the federated knowledge graph data is integrated into a federated knowledge graph dataset, which is represented as G = (E, R, T). Among them, E is the set of entities, R is the set of relationships, and T is the set of triples. Among them, the triple is represented as q = (h, r, t), where q is a single unit group, h is the head entity, t is the tail entity, and r is the relationship between entities. From the above, the local knowledge graph of the i-th client can be represented as G i .

[0052] In the federated knowledge graph representation learning, entities and relationships are mapped to low-dimensional vectors, that is, e, r ∈ R n , where n is the dimension of the vector, e is the local entity vector, r is the relationship between entities, and R is the set of relationships. By mapping entities and relationships to low-dimensional vectors, the representation method can effectively assist in efficiently processing and learning the information in the knowledge graph.

[0053] It should be noted that in this embodiment, through the operation in step 1, the federated knowledge graph data can be integrated into a federated knowledge graph dataset, allowing each client to retain its own local knowledge graph data, including entities, relationships, and triples, without uploading the data to the central server. The entire federated knowledge graph dataset is composed of the local knowledge graphs of multiple clients, which can not only utilize their respective data resources but also protect data privacy and security.

[0054] However, having each client retain its own local knowledge graph data will also make the data of each client more chaotic, posing a huge challenge to the processing ability of the server. Moreover, a large amount of data will also cause congestion in the communication between the server and the client. Therefore, to solve this technical problem, this embodiment also includes the step of standardizing the data in the federated knowledge graph dataset. Specifically, the federated knowledge graph dataset contains a large number of data values of each client at different times, and these noisy data need to be removed before training these data. If global model operations are performed without processing these data, it is very likely that the final result will be incorrect or convergence cannot be achieved. Therefore, standardizing the data in the federated knowledge graph dataset can effectively improve the consistency of the data in the federated knowledge graph dataset, and distributed Min-Max standardization can scale the data to a fixed range, making the value range of the data fixed, which helps to improve the training efficiency of the federated knowledge graph model. At the same time, it can retain the distribution shape and relative size of the original data and will not affect the final training result. Moreover, distributed Min-Max standardization has extremely high security and can ensure the security of the communication information between the server and the client.

[0055] Therefore, in this embodiment, the data is standardized by using distributed Min-Max standardization, which specifically includes: each participating node locally calculates the minimum and maximum values of its dataset; through a secure multi-party computing method, the local minimum and maximum values of each node are aggregated to calculate the global minimum and maximum values; the global minimum and maximum values are distributed to each participating node through a secure communication mechanism; each node uses the global minimum and maximum values to standardize its local data.

[0056] Step 2: Construct a global model M located at server S, where M is the global model, is the vector of the entity collection, and R is the relationship set; define the representation vector of the entity as a vector with a lower dimension than n, that is where, is the global low-dimensional entity vector, R is the relationship set, is the dimension of the global entity vector, n is the dimension of the local entity vector, represents any entity, and E is the set of entities.

[0057] Add a similarity constraint to the entity vector, that is where H is the entity similarity, e i is the local entity vector with index i, e j is the local entity vector with index j, σ 2 is the parameter of the RBF kernel function, and exp is the natural base, is the global entity vector with index i, is the global entity vector with index j.

[0058] It should be noted that in this step, since the complexity of the global model is linearly related to the dimension of the entity vector, a lower-dimensional global entity vector is adopted while effectively reducing the parameters of the global model, it will also lead to a reduction in the specificity of the entire model. However, the reduction in specificity is not good for the training effect. Therefore, in order to ensure that the entity vector can have sufficient specificity, the present invention adds a similarity constraint to the entity vector here. The purpose is to ensure that the entity vectors in M can have specificity. Through this constraint, it can be ensured that the low-dimensional representation vectors of each entity have large differences, thereby ensuring the accuracy of the final training result.

[0059] At the same time, since the complexity of the global model is linearly related to the dimension of the entity vector, a lower-dimensional global entity vector is adopted can effectively reduce the parameters of the global model. The number of relationships in the knowledge graph is usually much smaller than that of entities, so there is no need to specifically reduce the vector dimension of the relationships. Therefore, from this perspective, the present invention constructs a global model in the server in step 2 to reduce the parameters in the global model and reduce the communication complexity between the server and the client. Through the model construction step in step 2, a global model is constructed. While mapping the entity to a low-dimensional vector, a similarity constraint is added to ensure that the similarity between entities is maintained while reducing the dimension. Through the operations in step 2, it is possible to more efficiently process and store the knowledge graph data while maintaining the necessary information integrity.

[0060] Step 3: Reconstruct the neural network of the defined client entities, where each defined client has its own reconstruction network, that is, each defined client C i has a reconstruction network f i (*), and through f i (*) reconstructs the unique features of the local knowledge graph based on the global entity information.

[0061] In this embodiment, the reconstruction network can be a multi-layer graph attention network.

[0062] Specifically, the calculation method of the entity representation in the l-th layer of the multi-layer attention network can be where N is the set of neighbors of e in the knowledge graph, and e l is the entity in the l-th layer of the multi-layer attention network, is, α i is the attention weight of the i-th neighbor to e.

[0063] α i is calculated as follows: ATT(e i , e) = h T (We i + Ue),

[0064] where e i is the local entity vector with index i, e j is the local entity vector with index j, N is the set of neighbors of e in the knowledge graph, h, W, U are parameters that can be learned by the neural network, and h T is the transpose of h.

[0065] Step 4: Obtain the global model from the server, and restore the low-dimensional entity vector in it to the local entity vector through the entity reconstruction network, that is where e is the local entity vector, is the restoration function; is the low-dimensional entity vector.

[0066] Meanwhile, the client is equipped with a compression model that can compress the local entity vector into the global low-dimensional entity vector again, that is, re-transform it into the low-dimensional entity vector by using t-distributed stochastic neighbor embedding (t-SNE) Then, together with r', it is used as the global model for client training, that is Uploaded to the server, setting the compression model can ensure the reversibility of the whole process and ensure the accuracy of the whole training result.

[0067] In this step, restoring the low-dimensional entity vector to the local entity vector through the entity reconstruction network maps the low-dimensional entity vector representing global information and with low ability to a higher-dimensional vector with local knowledge graph features and strong representation ability, ensuring the accuracy of the training data, and while reducing the communication complexity between the server and the client, ensuring that the effect of machine learning does not decrease.

[0068] Step 5: Conduct federated training by combining the server global model, the local reconstruction neural network of the client, and the knowledge graph representation learning loss function. The training process is carried out for c rounds until the model converges, and the result of the model training is verified through the head entity or tail entity prediction problem.

[0069] Specifically, the tail entity prediction model can be The head entity prediction model can be g = ||h + r - t|| p , which is the triple score and is used to measure the matching degree between the entity and the relationship. Among them, g is the triple score, p is the norm (usually 2), h is the head entity, r is the relationship, as the candidate tail entity.

[0070] Finally, based on the tail entity prediction model and the head entity prediction model, the learning objective function of the client is obtained as loss = ∑q∈G,q - ∈G{[g(q) - γ1] + + ∈[γ2 - g(q - )] +}, where q is a triple, q - is a negative sample triple, g(q) is the score of the triple, g(q - ) is the score of the negative sample triple q - , G is the set of knowledge graph triples, G - is the set of negative samples of knowledge graph triples, γ1 is the positive sample boundary constant, γ2 is the negative sample boundary constant, ∈ is the balance weight of positive and negative samples, [*] + is the positive part function, defined as: Among them, negative samples can be obtained by standard random negative sampling.

[0071] Specifically, the model training specifically includes the following sub-steps:

[0072] 1) The server sends the global model M to each client.

[0073] 2) Each client C i performs the following operations to convert the global entity vector into a local entity vector and uses e and r to perform multiple rounds of training on the local knowledge graph G i according to the learning objective function loss to obtain the updated e', r' and f(*).

[0074] 3) The client re-transforms the trained e' into a low-dimensional entity vector through t-distributed stochastic neighbor embedding and then, together with r', serves as the global model trained by the client.

[0075] 4) Aggregate the global models uploaded by all clients on the server side.

[0076] Specifically, the global model aggregation on the server side specifically includes: 1. The server side receives the low-dimensional entity vector re-transformed through the local compression network 2. Use the parameter mini-batch stochastic gradient descent algorithm to perform weighted averaging, update the global model, and optimize the low-dimensional entity vectors; 3. The server distributes the updated global model to all clients. The clients train the model based on local data, generate local model parameter updates, and send the updated parameters to the server; 4. The server performs weighted averaging on the updated parameters sent by all clients to obtain the updated global model parameters.

[0077] It should be noted that the formula for optimizing and updating the global model parameters in this embodiment can be obtained through the following steps. Suppose there are K clients, the amount of local data on the k-th client is nk, then the total data volume is N = ∑k=1K nk. The model parameter update on the k-th device is Δwk, then the formula for updating the global model parameters can be deduced as wt+1 = wt + 1 / N ∑k=1K nkΔw.

[0078] In step 5, the knowledge graph representation learning process under the federated learning framework is described by performing loop operations on steps 1-4 until the result converges, including each client training the local data respectively, uploading the optimization results of the local model to the server through the mechanism of compression and restoration, and the server aggregating and updating these results, finally achieving the convergence of the model.

[0079] In this embodiment, through the above steps, the generalization ability and accuracy of the model are improved by collaborative training of the data of multiple clients. At the same time, since the data of each client does not need to be directly transmitted to the server, only the compressed and optimized parameters are used for communication, protecting the privacy of the data. More importantly, the combination of local training of clients and aggregation of the global model realizes the personalization and overall optimization of the model, thereby improving the expressiveness and applicability of the model.

[0080] In this embodiment, by defining the global model as the benchmark model and the local model as the reconstructed neural network with the benchmark model as the input, with the flexibility of the neural network, the knowledge graph representation vector used for local data is adaptively reconstructed from the global benchmark model according to the characteristics of the client knowledge graph, thus improving the effect of the entire federated knowledge graph representation learning.

[0081] In this embodiment, through the specially designed global entity / relationship model and local entity reconstruction network, it is possible to flexibly aggregate the similar features between different client knowledge graphs and distinguish the special features of a single client. And through the above design, the knowledge graph representation learning model can adaptively learn the features of local data according to the characteristics of the knowledge graph, and at the same time its low-dimensional entity vectors can also effectively reduce the communication complexity between the server and the client, realizing more effective federated knowledge graph learning.

[0082] Example 2

[0083] This embodiment provides a federated knowledge graph representation learning system based on a reconstruction network. The system includes an initial definition unit, a global model construction unit, a neural network reconstruction unit, a vector transformation unit, and a training unit.

[0084] Among them, the initial definition unit is configured to initially define the federated knowledge graph. Specifically, the initial definition includes defining the data format, type, and learning scenario of the federated knowledge graph, and defining the server and the client according to the differences in the learning scenario.

[0085] The initial definition unit is configured to initially define the federated knowledge graph. The initial definition includes defining the data format, type, and learning scenario of the federated knowledge graph, and defining the server and the client according to the differences in the learning scenario.

[0086] The global model construction unit is connected to the initial definition unit and is configured to construct a global model M located in the server S. Among them, M is the global model. is a set of entity representation vectors, and R is a set of relationship representation vectors; the relationship representation vector is defined as an n-dimensional vector, that is, r ∈ R. n , the representation vector of the global entity is a vector lower than n dimensions, that is, Among them, E is the set of entities, and R is the set of relationships. is the low-dimensional global entity representation vector. is the dimension of the low-dimensional global entity vector, n is the dimension of the local entity vector. represents any entity.

[0087] The neural network reconstruction unit is connected to the global model construction unit and is configured to reconstruct the neural network of the defined client entity. Each defined client has its own reconstruction network, that is, each defined client C. i has a reconstruction network f. i (*), through f. i (*) reconstructs the unique features of the local knowledge graph based on the global entity information.

[0088] The global entity vector is connected to the neural network reconstruction unit and is configured to obtain the global model from the server and restore the low-dimensional entity vector in it to the local entity vector through the entity reconstruction network, that is, Among them, e is the local entity vector. is the reconstruction network. is the low-dimensional entity vector.

[0089] The training and validation unit is connected to the vector transformation unit and is configured to perform federated training by combining the server global model, the client's local reconstruction neural network, and the knowledge graph representation learning loss function. The training process is carried out for c rounds until model convergence is achieved, and the results of model training are verified through head entity or tail entity prediction questions.

[0090] Specifically, the training and validation unit may further include a model distribution subunit, a function training subunit, a vector update subunit, and a global model aggregation unit.

[0091] Among them, the model distribution subunit is configured to the server to send the global model M to each client.

[0092] The function training subunit is connected to the model distribution subunit and is configured to, for client C i execute the operation of converting the global entity vector into a local entity vector and perform multiple rounds of training on the local knowledge graph G i using e and r according to the learning objective function loss to obtain the updated e', r' and f(*).

[0093] The vector update subunit is connected to the function training subunit and is configured to the client to re-transform the trained e' into a low-dimensional entity vector through t-distributed stochastic neighbor embedding (t-SNE) and then, together with r', as the global model trained by the client, that is upload it to the server.

[0094] The global model aggregation unit is connected to the vector update subunit and is configured to aggregate all the global models uploaded by the clients on the server side, that is where M is the aggregated global model, is the updated global parameter of the i-th client.

[0095] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A federated knowledge graph representation learning method based on a reconstruction network, characterized in that The described federated knowledge graph representation learning method includes: S100. Initially define the federated knowledge graph. The initial definition includes defining the data format, type, and learning scenario of the federated knowledge graph, and defining the server and client according to the differences in the learning scenario. S200, construct the global model M located in the server S, Among them, M is the global model, is a set of low-dimensional global entity representation vectors, and R is a set of relation representation vectors; define the relation representation vector as an n-dimensional vector, that is, r ∈ R n , and the low-dimensional global entity representation vector is a vector lower than n dimensions, that is where E is a set of entities, R is a set of relation representation vectors, is a low-dimensional global entity representation vector, is the dimension of the low-dimensional global entity representation vector, n is the dimension of the local entity vector, represents any entity; S300, reconstruct the neural network of the client entity after the reconstruction definition, where each defined client has its own reconstruction network, that is, each defined client C i has a reconstruction network f i (*), through f i (*) reconstruct the unique features of the local knowledge graph based on the low-dimensional global entity representation vector; S400, obtain the global model from the server, and restore the low-dimensional global entity representation vector in it to the local entity vector through the entity reconstruction network, that is where e is the local entity vector, is the reconstruction network; is the low-dimensional global entity representation vector; S500. Conduct federated training by combining the server global model, the client's local reconstruction network, and the knowledge graph representation learning loss function. The training process is carried out for c rounds until model convergence is achieved, and the results of model training are verified through head entity or tail entity prediction problems.

2. The federated knowledge graph representation learning method based on a reconstruction network according to claim 1, wherein, The defined server and client include: the defined server is S, and the defined client is C, where C = (C1, C2,..., C m ), and m is the number of clients; Integrate the federal knowledge graph data into a federal knowledge graph dataset, and the federal knowledge graph dataset is represented as G = {G1, C2, …, G m} = (E, R, T), where G i is the local knowledge graph representation of the i-th client, E is the set of entities, R is the set of relationship representation vectors, and T is the set of triples; The triple in the knowledge graph is represented as q=(h,r,t), where q represents the triple, h is the head entity, t is the tail entity, and r is the relationship between the entities.

3. The federated knowledge graph representation learning method based on a reconstruction network according to claim 1, wherein The step S200 further includes adding a similarity constraint to the entity vector, that is Among them, H is the similarity function between two entities, e i is the local entity representation vector with index i, e j is the local entity representation vector with index j, σ 2 is the parameter of the RBF kernel function, and exp is the natural base.

4. The federated knowledge graph representation learning method based on a reconstruction network according to claim 1, wherein The reconstruction network f i (*) is a multi-layer graph attention network for reconstructing the unique features of the local knowledge graph based on the global entity vector.

5. The federated knowledge graph representation learning method based on the reconstruction network according to claim 4, wherein The calculation method of the entity representation in the l-th layer of the multi-layer attention network is: where N is the set of neighbors of e in the knowledge graph, and e l is the vector of the l-th layer in the multi-layer attention network, is the vector of the local entity neighbor with index k in the (l - 1)-th layer, and α k is the attention weight of the k-th neighbor of e to e.

6. The federated knowledge graph representation learning method based on a reconstruction network according to claim 1, wherein The step S500 further includes the following sub-steps: S510. The server sends the global model M to each client. S520, Client C i Perform the operation of converting the low-dimensional global entity representation vector into a local entity vector using e and r on the local knowledge graph G i and perform multiple rounds of training according to the learning objective function loss to obtain the updated e', r' and f(*); In S530, the client re-transforms the trained e' into a low-dimensional global entity representation vector through t-distributed stochastic neighbor embedding Then, together with r', it serves as the global model trained by the client; S540. Aggregate the global models uploaded by all clients on the server side.

7. The method for federated knowledge graph representation learning based on a reconstruction network according to claim 1, characterized in that In the verification of the model training results through head entity or tail entity prediction problems, The tail entity prediction model is The head entity prediction model is g = ||h + r - t|| p , which is a triple score Among them, g is the triple score, p is the norm, h is the head entity, r is the relationship, and t is the candidate tail entity.

8. The federated knowledge graph representation learning method based on a reconstruction network according to claim 7, wherein Based on the tail entity prediction model and the head entity prediction model, the learning objective function of the client is: loss = ∑q∈G,q - ∈G - {[g(q) - γ1] + + ∈[γ2 - g(q - )] +}, Among them, q is a triple, q - is a negative sample triple, g(q) is the score of triple q, g(q - ) is the triple score of negative sample q - , G is the set of knowledge graph triples, G - is the set of negative samples of knowledge graph triples, γ1 is the positive sample boundary constant, γ2 is the negative sample boundary constant, ∈ is the balance weight of positive and negative samples, [*] + is the positive part function, defined as:

9. A federated knowledge graph representation learning system based on a reconstruction network, characterized in that The described federated knowledge graph representation learning system includes: an initial definition unit, a global model construction unit, a neural network reconstruction unit, a vector transformation unit, and a training unit. Among them, The initial definition unit is configured to initially define the federated knowledge graph. The initial definition includes defining the data format, type, and learning scenario of the federated knowledge graph, and defining the server and client according to the differences in the learning scenario. The global model construction unit is connected to the initial definition unit and is configured to construct the global model M located on the server S. Among them, M is the global model, is the set of low-dimensional global entity representation vectors, and R is the set of relation representation vectors; the relation representation vector is defined as an n-dimensional vector, that is, r ∈ R n , and the low-dimensional global entity representation vector is a vector with a dimension lower than n, that is, where E is the set of entities, R is the set of relation representation vectors, is the low-dimensional global entity representation vector, is the dimension of the low-dimensional global entity vector, n is the dimension of the low-dimensional global entity representation vector, represents any entity; The neural network reconstruction unit is connected to the global model construction unit and is configured to reconstruct the neural network of the defined client entity, where each defined client has its own reconstruction network, that is, each defined client C i has a reconstruction network f i (*), and through f i (*) reconstructs the unique features of the local knowledge graph based on the low-dimensional global entity representation vector; The global entity vector is connected to the neural network reconstruction unit and is configured to obtain the global model from the server and restore the low-dimensional global entity representation vector therein to the local entity vector through the entity reconstruction network, that is where e is the local entity vector, is the reconstruction network; is the low-dimensional global entity representation vector; The training verification unit is connected to the vector transformation unit and is configured to conduct federated training by combining the server global model, the client's local reconstruction network, and the knowledge graph representation learning loss function. The training process is carried out for c rounds until model convergence is achieved, and the results of model training are verified through head entity or tail entity prediction problems.

10. The federated knowledge graph representation learning system based on the reconstruction network according to claim 9, characterized in that, The training verification unit further includes a model distribution sub-unit, a function training sub-unit, a vector update sub-unit, and a global model aggregation unit. Among them, The model distribution sub-unit is configured to have the server send the global model M to each client. The function training subunit is connected to the model distribution subunit and is configured to, for client C i perform the operation of converting the low-dimensional global entity representation vector into a local entity vector using e and r on the local knowledge graph G i to perform multiple rounds of training according to the learning objective function loss to obtain the updated e', r' and f(*); The vector update subunit is connected to the function training subunit and is configured to re-transform the trained e' into a low-dimensional global entity representation vector by t-distributed stochastic neighbor embedding (t-SNE) for the client. Then, together with r', it serves as the global model trained by the client, that is, uploaded to the server. The global model aggregation unit is connected to the vector update subunit and is configured to aggregate the global models uploaded by all clients on the server side, that is where M is the aggregated global model, is the global parameter updated by the i-th client.

Images