Method and device for training graph encoder based on knowledge graph

By comparative learning, optimized graph encoder and aligned with text encoder, the problem of fusion of graph modal and text modal in knowledge graph graph is solved, the quality of feature extraction and expression ability are improved, and the application of knowledge graph in downstream tasks is promoted. .

CN118917401APending Publication Date: 2024-11-08ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411081997.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The mixing of graph structure information and semantic information of entities or relationships in the knowledge graph leads to difficulty in feature extraction and representation, and it is difficult for the prior art to effectively integrate graph modes and text modes.

Method used

Through comparison learning, the pre-trained text encoder is used to optimize the graph encoder, align it with the text encoder, realize the fusion of graph mode and text mode, and update the graph encoder through the comparison loss of sub-graph and description text, so that the graph encoder can be made It is aligned with the text encoder in the same vector space.

Benefits of technology

It improves the feature extraction quality and expression ability of knowledge graph, enhances the processing ability of knowledge graph data, and promotes the performance of applications such as intelligent search, common sense question and answer, and personalized recommendation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917401A_ABST
    Figure CN118917401A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a method and device for training a graph encoder based on a knowledge graph. The knowledge graph comprises a plurality of nodes recorded in a triple form and connecting edges among the nodes. The method comprises the following steps: sampling nodes and connecting edges from the knowledge graph according to a preset search strategy to obtain a plurality of sub-graphs; and inputting the first sub-graph into the graph encoder to obtain a graph representation corresponding to the first sub-graph. And determining a first description text based on the text corpus of each triad in the second sub-graph. And inputting the first description text into a text encoder to obtain a corresponding text representation. And determining a contrast loss of a first sample pair formed by the first sub-graph and the first description text based on a representation difference between the graph representation and the text representation. And updating the graph encoder according to the respective comparison loss of the plurality of sample pairs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of machine learning and knowledge graph technology, and in particular, to a method and apparatus for training a graph encoder based on a knowledge graph. Background Art

[0002] Knowledge graph is a data structure based on graph structure data that can express entities and the relationships between entities. It plays a key role in many fields, especially in depicting the complex structural relationships between knowledge entities. Some data sets that can reflect the relationships between various entities in the real world, such as relationships in social networks, molecular structure relationships, etc., are often carried in the form of knowledge graphs.

[0003] The use of knowledge graphs is inseparable from feature extraction, that is, representing the entities and relationships contained in the knowledge graph in the form of vectors. However, the knowledge graph contains both graph structure information and semantic information of the entities or relationships themselves. The mixture of different types of information brings challenges to the representation of knowledge graphs.

[0004] Therefore, we hope to have an improved solution to better extract features and characterize the knowledge graph. Summary of the invention

[0005] One or more embodiments of the present specification describe a method and device for training a graph encoder based on a knowledge graph, which effectively utilizes the text processing capabilities of a pre-trained text encoder for knowledge graph data through comparative learning, optimizes the graph encoder to align it with the text encoder, improves the degree of fusion between the graph encoder and the text encoder in knowledge graph processing, and realizes the fusion of graph modality and text modality in feature processing of the knowledge graph.

[0006] According to a first aspect, a method for training a graph encoder based on a knowledge graph is provided, wherein the knowledge graph includes a plurality of nodes recorded in the form of triples and the connecting edges therebetween; the method includes:

[0007] According to a preset search strategy, nodes and connecting edges are sampled from the knowledge graph to obtain several subgraphs.

[0008] The first sub-graph is input into the graph encoder to obtain a graph representation corresponding to the first sub-graph.

[0009] Based on the text corpus of each triple in the second subgraph, a first description text is determined.

[0010] The first description text is input into a text encoder to obtain a corresponding text representation.

[0011] Based on the representation difference between the image representation and the text representation, a contrast loss of a first sample pair consisting of a first sub-image and a first description text is determined.

[0012] The graph encoder is updated according to respective contrast losses of a plurality of sample pairs.

[0013] According to one embodiment, the text encoder is a text encoder obtained after pre-training.

[0014] According to one embodiment, determining the contrast loss of a first sample pair consisting of a first sub-image and a first description text based on the representation difference between the image representation and the text representation includes:

[0015] The graph representation is projected using a first mapping neural network to obtain a first projection.

[0016] The text representation is projected using a second mapping neural network to obtain a second projection, where the second projection and the first projection are in the same vector space.

[0017] Based on a difference between the first projection and the second projection, the characterizing difference is determined.

[0018] According to an implementation of the above embodiment, the parameters of the first mapping neural network and the second mapping neural network are updated according to the contrast loss.

[0019] According to one embodiment, determining the contrast loss of a first sample pair consisting of a first sub-image and a first description text includes:

[0020] If the first sub-image and the second sub-image are the same sub-image, the contrast loss is positively correlated with the representation difference.

[0021] If the first sub-image and the second sub-image are different sub-images, the contrast loss is negatively correlated with the representation difference.

[0022] According to one embodiment, the multiple sample pairs include positive sample pairs and negative sample pairs, wherein the description text in the positive sample pairs is used to describe the corresponding sub-image, and the description text in the negative sample pairs is not used to describe the corresponding sub-image.

[0023] According to one implementation, the preset search strategy includes:

[0024] Starting from the target node, a breadth-first or depth-first traversal search is performed according to a preset search depth and / or search width.

[0025] According to one implementation, the preset search strategy includes:

[0026] Starting from the target node, a random walk search is performed for a predetermined number of steps.

[0027] According to one implementation, determining the first description text includes:

[0028] For any of the triples, the description text of the entity represented by the node and the description text of the semantic association represented by the connecting edge are concatenated to serve as the text corpus.

[0029] The text corpora corresponding to each triple in the second subgraph are concatenated to obtain the first description text.

[0030] According to one implementation, obtaining the graph representation corresponding to the first subgraph includes:

[0031] The first subgraph is encoded using the graph encoder to obtain a node representation of each node.

[0032] Based on the node representations of the respective nodes, a pooling operation is performed to obtain the graph representation.

[0033] According to a second aspect, a device for training a graph encoder based on a knowledge graph is provided, wherein the knowledge graph includes a plurality of nodes recorded in the form of triples and connecting edges therebetween; the device includes:

[0034] A subgraph acquisition module is configured to sample nodes and connecting edges from the knowledge graph according to a preset search strategy to obtain a plurality of subgraphs;

[0035] a sub-graph encoding module, configured to input the first sub-graph into the graph encoder to obtain a graph representation corresponding to the first sub-graph;

[0036] A subgraph description module, configured to determine a first description text based on the text corpus of each triple in the second subgraph;

[0037] A text encoding module, configured to input the first description text into a text encoder to obtain a corresponding text representation;

[0038] A loss determination module, configured to determine a contrast loss of a first sample pair consisting of a first sub-image and a first description text based on a representation difference between the image representation and the text representation;

[0039] The encoder optimization module is configured to update the image encoder according to the contrast loss of each of the multiple sample pairs.

[0040] According to a third aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0041] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein an executable code is stored in the memory, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0042] In the method and device provided in the embodiments of this specification, through comparative learning, the text representation and graph representation of the knowledge graph are used to align the graph encoder with the pre-trained text encoder to achieve the fusion of the graph modality and the text modality. The text representation used is a sub-graph text representation obtained by encoding the natural language description corresponding to the sub-graph of the knowledge graph using a pre-trained text encoder. The graph representation used is a sub-graph representation obtained by encoding the sub-graph of the knowledge graph using a graph encoder. Then, based on the difference between the sub-graph text representation and the sub-graph representation, the graph encoder is updated so that the graph encoder is continuously aligned with the pre-trained text encoder, improving the degree of fusion between the two, and finally achieving multimodal fusion in the same vector space.

[0043] By using this training method to train the graph encoder, it is possible to effectively achieve the alignment and fusion of the graph modality and the text modality, improve the quality of feature extraction of the knowledge graph, enhance the expression ability, and promote the application of the knowledge graph in downstream tasks (for example, intelligent search, common sense question and answer, personalized recommendation, etc.). The text encoder is good at processing natural language text, exploring semantic expressions and the relative importance between word units, while the graph encoder performs well in capturing the structural information of the knowledge graph data. In the training method provided in the embodiment of this specification, the degree of fusion can be improved while maintaining the unique capabilities of each encoder, so that the advantages of the two complement each other and the processing ability of the knowledge graph data is improved. At the same time, the training method provided in the embodiment of this specification is not limited to the alignment between a certain node in the knowledge graph data and its description text, but uses the whole graph representation of the subgraph and its overall description text to align the encoder, so that it can align the encoder on different subgraphs of multiple knowledge graphs, improving the generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of a knowledge graph structure applicable to the embodiments of this specification;

[0046] Figure 2 A schematic diagram of an implementation scenario provided according to an embodiment of this specification;

[0047] Figure 3 A flow chart of a method for training a graph encoder based on a knowledge graph according to an implementation;

[0048] Figure 4A A schematic diagram of a subgraph sampling process using breadth-first search according to an implementation scenario;

[0049] Figure 4B A schematic diagram of a subgraph sampling process using breadth-first search according to an implementation;

[0050] Figure 5A A schematic diagram of a subgraph sampling process using a depth-first search according to an implementation scenario;

[0051] Figure 5B A schematic diagram of a subgraph sampling process using a depth-first search according to an embodiment;

[0052] Figure 6 A schematic diagram for generating a triple sentence according to an embodiment;

[0053] Figure 7 A device for training a graph encoder based on a knowledge graph is provided according to an embodiment of this specification. DETAILED DESCRIPTION

[0054] The solution provided by the embodiments of this specification is described in detail below in conjunction with the accompanying drawings.

[0055] Figure 1 The following is a schematic diagram of the knowledge graph structure applicable to the embodiments of this specification. As shown in the figure, the knowledge graph contains multiple entities, such as eating breakfast, chopsticks, spoon, dining table, etc. Each entity corresponds to a node in the knowledge graph. Nodes with a certain association relationship are connected by directed edges, which show the relationship between the two connected nodes. In the knowledge graph, the relationship between entities can be of multiple types and multiple directions. Figure 1 For example, the directed edge from the node "eat breakfast" to the node "chopsticks" represents the relationship type "use" between them; the directed edge from the node "chopsticks" to the node "dining table" represents the relationship type "placed" between them; there is a directed edge between the node "chopsticks" and the node "spoon" that points to each other, representing the relationship type "matching" between them, and so on.

[0056] In the recording method of the knowledge graph, the conventional recording method of graph structure data can be used, for example, the adjacency matrix, to record the topological structure of the knowledge graph. More commonly, the knowledge graph is recorded in a triple manner, which records each connecting edge in the knowledge graph and the two nodes connected by the edge as a triple. It can be understood that a triple includes two nodes and a connecting edge, where the head node (h) is the starting node connected by the connecting edge, and the tail node (t) is the target node connected by the connecting edge, and the connecting edge represents the relationship between the head node and the tail node (r). Therefore, a triple can be expressed as 〈h, r, t〉. For example, in Figure 1 The part circled by the dotted circle can be expressed as a triplet 〈eat breakfast, use, chopsticks〉.

[0057] The above is a brief introduction to the structure and recording method of the knowledge graph. It is not difficult to see that the knowledge graph contains both textual expressions of entities and expressions of associations between different entities, that is, topological structures. Between textual expressions and topological structures, there may be complementary information. When using the knowledge graph, it is necessary to extract features of the entities and the relationships between entities contained in the knowledge graph and represent them in the form of vectors. The textual representation extracted from the knowledge graph can express the semantic features of each entity, and the graph representation can express the topological structure features between entities. When extracting features from the knowledge graph, it is necessary to fuse the representations of the two different modalities to complement each other's expression advantages in order to improve the expressiveness of the knowledge graph representation.

[0058] Graph encoders have become the preferred technology for processing and analyzing graph structured data because of their ability to transfer and integrate information between nodes. Using graph encoders to process knowledge graph data can obtain feature representations that can effectively support various downstream tasks. However, although graph encoders can well capture the structural information of knowledge graph data and explore the associations and structural characteristics between data, they are limited in the feature expression of nodes. On the other hand, in recent years, a variety of pre-trained text encoders, including large language models, have made significant progress and have shown excellent performance in a variety of natural language processing tasks such as text classification and intelligent question answering. These text encoders are good at processing natural language text, exploring semantic expressions and the relative importance between word units, but they are insufficient in capturing topological structure information between nodes. In order to combine the two, complement their respective advantages, and enhance the ability to extract representations of knowledge graphs, it is necessary to align the graph encoder and the text encoder in the vector space, and realize cross-modal data processing and understanding of knowledge graphs while maintaining the unique capabilities of each encoder, and improve the feature extraction performance and generalization ability of the model.

[0059] Figure 2Schematic diagram of an implementation scenario of an embodiment disclosed in this specification. First, subgraph sampling is performed in the knowledge graph to obtain several subgraphs. Then, the sampled subgraphs are Figure 1 The aspect is input into the graph encoder for encoding, and the graph representation corresponding to each subgraph is obtained (in the figure, e i Identify the graph representation corresponding to the i-th subgraph); on the one hand, as the input of the text encoder, the corresponding description text obtained by text processing of each subgraph is input into the text encoder for encoding, and the text representation corresponding to each subgraph is obtained (in the figure, h j represents the text representation corresponding to the jth subgraph). Next, based on the obtained graph representations and text representations, the graph encoder and the text encoder are compared and learned to obtain the contrast loss. Finally, the graph encoder is updated according to the contrast loss.

[0060] The goal of the above contrastive learning is to minimize the difference between the representation of the graph and the text representation generated based on the same subgraph, and to maximize the difference between the representation of the graph and the text representation generated based on different subgraphs. Figure 2 , as mentioned above, in the figure with e i Identify the graph representation corresponding to the i-th subgraph, with h j Represents the text representation corresponding to the j-th subgraph. When i=j, the graph representation and the text representation are extracted based on the same subgraph, and the goal of contrastive learning is to minimize the difference between their representations, that is, to minimize the values ​​of each element on the diagonal of the matrix on the right side of the figure (for example, e1·h1, e2·h2, etc.). When i≠j, the graph representation and the text representation are extracted based on different subgraphs, and the goal of contrastive learning is to maximize the difference between their representations, that is, to maximize the values ​​of each element outside the diagonal of the matrix on the right side of the figure (for example, e1·h2, e2·h1, etc.).

[0061] It is understandable that in this embodiment, the difference between two representation vectors is determined by using the vector dot product method, but this does not limit the method for determining vector similarity. In other embodiments, the difference between two representation vectors may also be determined by using methods such as cosine similarity and Euclidean distance, which are not limited in the embodiments of this specification.

[0062] In this embodiment, a subgraph is extracted from the knowledge graph, and a graph representation is extracted based on the subgraph, so that the graph representation can express the complete topological structure characteristics of the subgraph. At the same time, a text representation is extracted based on the description text generated by the subgraph, so that the text semantic features contained in the subgraph can be fully extracted, forming a semantically equivalent graph-text representation pair with the graph representation. By comparing the graph representation and the text representation, the representation differences between them can be discovered, thereby determining the update direction of the graph encoder parameters. After multiple rounds of comparative learning, the graph encoder and the text encoder can be aligned to the same vector space.

[0063] The above is a general description of a method for training a graph encoder based on a knowledge graph provided in an embodiment of this specification. The technical solutions provided by one or more embodiments of this specification will be described in detail below in conjunction with the accompanying drawings.

[0064] Figure 3 A flow chart of a method for training a graph encoder based on a knowledge graph according to an embodiment is shown. It can be understood that the method can be performed by any device, equipment, platform, or device cluster with computing and processing capabilities. In this embodiment, the knowledge graph includes multiple nodes recorded in the form of triples and the connecting edges therebetween. The structure of the knowledge graph and the meanings of the nodes and connecting edges contained therein have been introduced in the previous text and will not be repeated here.

[0065] The specific implementation method of each step in this embodiment will be described in detail below.

[0066] Step S301, according to a preset search strategy, sample nodes and connecting edges from the knowledge graph to obtain several subgraphs.

[0067] Since there are associations between nodes representing entities in the knowledge graph, if irregular random sampling of nodes and connecting edges is used when sampling subgraphs, the directed edges between nodes will be lost, destroying the association information between nodes in the subgraph and affecting the quality of subsequent representation extraction. Therefore, in the subgraph sampling process, it is necessary to follow a certain search strategy to preserve the complete association relationship between nodes in the subgraph as much as possible.

[0068] In this step, taking into account the topological structure between nodes in the knowledge graph and the graph neural network message passing mechanism relied on for graph encoding in subsequent steps, in one embodiment, breadth-first search or depth-first search is selected for subgraph sampling, which can preserve the associated edges and corresponding associated nodes of the sampled nodes to the greatest extent possible.

[0069] According to one implementation, the search strategy is to start with the target node and perform a breadth-first search. Figure 4AThe subgraph sampling process using breadth-first search in the implementation is shown. In the given exemplary knowledge graph, there are 11 nodes representing entities. Node 1 is used as the sampling starting point. According to the directed edges of the knowledge graph, breadth-first search is performed in sequence (the search process is indicated by circle numbers in the figure). A subgraph can be sampled. According to the search sampling order, the nodes contained in the subgraph are: node 1, node 5, node 3, node 2, node 6, node 10, node 11, node 8, node 4, node 9.

[0070] In one scenario of this implementation, in order to avoid the sampled subgraph being too large, the breadth-first search performed on the knowledge graph is performed based on certain search rules, and the search rules include a pre-set search depth and / or search width. Figure 4B The figure shows a subgraph sampling process in this scenario, using a breadth-first search method with search rules for subgraph sampling. In the given exemplary knowledge graph, there are 11 nodes representing entities. Node 1 is used as the sampling starting point. Breadth-first search is performed in sequence according to the directed edges of the knowledge graph. The search rule is set to search width less than or equal to 2 and search depth less than or equal to 2. In the figure, the search process is marked with circle numbers and the stop symbol is The search step indicated does not meet the search rule and stops. For example, when searching from node 6 to node 4 in the figure, the search step cannot be executed because the search depth of the search step is 3, which exceeds the search rule setting of 2. Thus, a subgraph with controllable scale can be obtained, as shown on the right side of the figure. The subgraph sampled under the search rule in this example contains the following nodes according to the search sampling order: node 1, node 5, node 3, node 6, node 10.

[0071] According to another implementation, the search strategy is to start with the target node and perform a depth-first search. Figure 5A The subgraph sampling process using depth-first search in the implementation is shown. In the given exemplary knowledge graph, there are 11 nodes representing entities. Node 1 is used as the sampling starting point. According to the directed edges of the knowledge graph, depth-first search is performed in sequence (the search process is indicated by circle numbers in the figure). A subgraph can be sampled. According to the search sampling order, the nodes contained in the subgraph are: node 1, node 5, node 6, node 4, node 9, node 10, node 11, node 3, node 2, and node 8.

[0072] In one scenario of this implementation, in order to avoid the sampled subgraph being too large, the depth-first search performed on the knowledge graph is performed based on certain search rules, and the search rules include a pre-set search depth and / or search width. Figure 5BThe figure shows a subgraph sampling process in this scenario, using a depth-first search method with search rules for subgraph sampling. In the given exemplary knowledge graph, there are 11 nodes representing entities. Node 1 is used as the sampling starting point. According to the directed edges of the knowledge graph, depth-first search is performed in sequence. The search rule is set to search width less than or equal to 2 and search depth less than or equal to 2. In the figure, the search process is marked with circle numbers and the stop symbol is The search step indicated does not meet the search rule and stops. For example, when searching from node 6 to node 4 in the figure, the search step cannot be executed because the search depth of the search step is 3, which exceeds the search rule setting of 2. Thus, a subgraph with controllable scale can be obtained, as shown on the right side of the figure. The subgraph sampled under the search rule in this example contains the following nodes according to the search sampling order: node 1, node 5, node 6, node 10, node 3.

[0073] In the above description of the implementation methods of various embodiments, for the search rules, two conditions, search width and search depth, are used to jointly constrain the sub-graph sampling process. In other implementations, any one of the search width and search depth can also be used as a constraint condition for the sub-graph sampling process, which is not limited in the embodiments of this specification.

[0074] The above steps describe the process of applying search strategies to sample subgraphs in the knowledge graph. In addition to the breadth-first search and depth-first search listed above, other graph traversal methods can also be used to sample nodes and connecting edges of the knowledge graph to obtain subgraphs. For example, in other implementations, a random walk search method can be used to sample subgraphs of the knowledge graph. At the same time, in order to avoid the sampled subgraph being too large, the random walk search is based on certain search rules. In a specific scenario, the search rule is a limit on the number of random walk steps. Therefore, in this scenario, any node in the knowledge graph is used as a visit node, and a neighbor node is randomly sampled from its neighbors as the next visit node. If the neighbor node has not been sampled, the neighbor node is added to the node sequence, and this process is repeated until the length of the sampled node sequence meets the preset condition (limit on the number of walk steps) to obtain a subgraph.

[0075] It should be noted that no matter which sampling method is used, the sampling of nodes in the process has a certain degree of randomness. For example, node 1 has three neighbor nodes, node 2, node 3, and node 5. These neighbor nodes can all be used as the next sampling nodes of node 1, and the selection of which node as the sampling node for the next search is random. Therefore, for the same knowledge graph, with a certain node as the starting node, under the same subgraph sampling method and search rules, the multiple subgraphs obtained by executing subgraph sampling multiple times may not be the same. The subgraph sampling process cited above is only used as an example to illustrate the subgraph sampling process in this step, and does not represent a limitation on the sampling results.

[0076] In addition, in the various implementations described above, only a specific node in the knowledge graph is used as the target node for subgraph sampling. In actual business scenarios, subgraph sampling of the knowledge graph is not limited to a certain node. Any node can be used as the target node to implement subgraph sampling, or pre-circled nodes of interest can be used as target nodes for subgraph sampling, thereby obtaining multiple subgraphs.

[0077] Through the above steps, several subgraphs can be obtained from the knowledge graph by subgraph sampling. In step S303, the first subgraph is input into the graph encoder to obtain the graph representation corresponding to the first subgraph.

[0078] In this step, the sub-image for encoding processing may be one or more. When multiple sub-images are input into the image encoder, the encoder encodes each sub-image in the same manner to ensure the stability of the obtained sub-image representation. Therefore, in this step, one sub-image (the first sub-image) is used as an example for explanation. In the subsequent steps, the first sub-image will also be used as an example to explain each step in detail.

[0079] The first subgraph is one of the subgraphs obtained in the previous step. In a specific implementation, the graph encoder is used to encode the first subgraph to obtain node representations of each node; then, based on the node representations of each node, a pooling operation is performed to obtain the graph representation. The pooling operation can be maximum pooling, average pooling, etc.

[0080] It should be understood that the graph encoder can be any graph neural network (GNN) that uses deep learning to directly learn graph structure data and generate representation vectors, such as graph convolution networks (GCN), graph attention networks (GAT), etc. The embodiments of this specification do not specifically limit the selection of the graph encoder.

[0081] Generally, when the first subgraph is input into the graph encoder, the graph encoder aggregates and represents each node according to the initial features of each node in the first subgraph and the graph structure information of the subgraph (node ​​connection relationship, or triples), and obtains the subgraph representation of the entire subgraph by fusing the representations of each node.

[0082] Next, based on the several sub-graphs obtained by sub-graph sampling in the above steps, text encoding processing is performed to obtain the text representation corresponding to each sub-graph.

[0083] In step S305, a first description text is determined based on the text corpus of each triple in the second subgraph.

[0084] Similar to the previous step, in this step, the sub-graph used for processing can be one or more. When extracting the description text of multiple sub-graphs and inputting the text encoder for encoding, each sub-graph is processed in the same way to ensure the stability of the obtained sub-graph text representation. Therefore, in this step, one sub-graph (the second sub-graph) is used as an example for explanation. In the subsequent steps, the second sub-graph will also be used as an example to explain each step in detail.

[0085] The second subgraph is one of several subgraphs obtained in the aforementioned steps, and the first description text is a natural language text used to describe the graph content of the second subgraph, which can be obtained based on the natural language description texts corresponding to each triple in the second subgraph. In a specific implementation, the first description text is obtained specifically in the following manner. For any triple in the second subgraph, the description text of the entity represented by the node and the description text of the semantic association represented by the connecting edge are spliced ​​to obtain the sentence corresponding to the triple as the text corpus. Then the text corpus corresponding to each triple in the second subgraph is spliced ​​to form a paragraph. This paragraph can serve as the first description text corresponding to the second subgraph.

[0086] Specifically, there are many ways to generate sentences based on the nodes and connecting edges in the triples. For example, in actual application scenarios, different sentence generation templates can be selected to splice the elements in the triples according to the category of the knowledge graph, the characteristics of the subgraph, or the semantics contained in the triples. Generally speaking, the generated sentence structure is a "subject + predicate + object + object complement" structure, and each part is taken from the elements in the triples, including but not limited to the type and name of the elements and the relationship between the elements. Figure 6 The triples contained in an exemplary subgraph are shown. Taking this graph as an example, the generation method of triple sentences is briefly introduced.

[0087] According to one implementation, Figure 6Take the triple "node 1, edge 1, node 2" in the example. The sentence generation uses the relationship type of the edge in the triple as the predicate, the name of the tail node as the object, and the name of the head node as the object complement. This triple can generate the sentence "Use chopsticks to eat breakfast".

[0088] According to one implementation, Figure 6 Take the triple "node 1, edge 2, node 3" in the example. The sentence generation uses the name of the head node in the triple as the subject, the relationship type of the edge as the predicate, and the name of the tail node as the object. This triple can generate the sentence "eating breakfast can fill your stomach".

[0089] According to one implementation, Figure 6 Take the triple "node 1, edge 1, node 2" in the example. The sentence generation uses the relationship type of the edge in the triple as the predicate, the type of the tail node as the object, and the type of the head node as the object complement. This triple can generate the sentence "use cutlery to eat".

[0090] It should be understood that the several embodiments listed above are merely illustrative examples to illustrate the generation method of triple sentences. In actual scenarios, there are many different ways to generate triple sentences, which are not limited to this and are not listed one by one in the embodiments of this specification.

[0091] Thus, for each triple contained in the second subgraph, a sentence can be generated, and then the obtained sentences can be concatenated to form a paragraph. This paragraph is the description text corresponding to the second subgraph, that is, the first description text.

[0092] The above is a brief introduction to the method of generating a subgraph description text. Next, the text description corresponding to the subgraph is input into the text encoder, and the text representation corresponding to the subgraph can be determined through the encoding process. The text representation is represented in the form of a vector.

[0093] That is, in step S307, the first description text is input into a text encoder to obtain a corresponding text representation.

[0094] Similar to the encoding process of the graph representation in the previous step, the text representation can be a global representation of the description text corresponding to the second subgraph. For example, the description text is encoded using a text encoder to obtain the encoding of each token, and the encoding of the [CLS] token is taken as the representation of the entire description text to represent the overall characteristics of the description text. Other fusion algorithms can also be used to obtain the text representation of the entire text based on the encoding of each token.

[0095] In one embodiment, the text encoder is a pre-trained text encoder, which can fully utilize the training accumulation of historical text corpus to the text encoder, obtain a text representation with stronger expressive ability, and avoid training the text encoder from scratch.

[0096] Through the above steps, several subgraphs are obtained based on the knowledge graph, and the graph representation and text representation corresponding to each subgraph are determined accordingly. Next, based on the representation difference between the graph representation and the text representation, the graph encoder is trained so that the graph encoder is continuously aligned with the text encoder, and finally a corresponding relationship is established between different data modalities to achieve multimodal fusion.

[0097] In step S309 , based on the representation difference between the image representation and the text representation, the contrast loss of a first sample pair consisting of a first sub-image and a first description text is determined.

[0098] In one example, the first sub-image and the second sub-image represented by the first description text are the same sub-image, and the first sample pair is a positive sample pair. In constructing a positive sample pair, any sub-image and its corresponding text representation combination can be taken as a positive sample pair.

[0099] In another example, the first sub-image and the second sub-image represented by the first description text are different sub-images, and the first sample pair is a negative sample pair. In the construction of the negative sample pair, any sub-image can be taken and combined with the text representation of any other sub-image as a negative sample pair.

[0100] In this step, the determination of the contrastive loss is related to the type of sample pairs.

[0101] When the first sample pair is a positive sample pair, the contrast loss is positively correlated with the representation difference, that is, the greater the representation difference between the image representation and the text representation, the greater the contrast loss, and vice versa.

[0102] When the first sample pair is a negative sample pair, the contrast loss is negatively correlated with the representation difference, that is, the greater the representation difference between the image representation and the text representation, the smaller the contrast loss, and vice versa.

[0103] In one implementation of this embodiment, the graph representation and the text representation can be projected into the same vector space for vector comparison to determine the representation difference. This implementation is particularly suitable for situations where the graph representation and the text representation have different dimensions and cannot be directly compared. When the dimensions of the two are the same, the vector comparison can be performed directly, or a projection conversion can be performed first and then the comparison can be performed.

[0104] Specifically, the process of projection conversion includes projecting the graph representation using a first mapping neural network to obtain a first projection; projecting the text representation using a second mapping neural network to obtain a second projection, wherein the second projection and the first projection are in the same vector space. Then, the representation difference can be determined based on the difference between the first projection and the second projection.

[0105] According to one implementation, the contrast loss is determined by a loss function. For example, the InfoNCE loss function may be used to determine the contrast loss.

[0106] Thus, after the contrast losses of the plurality of sample pairs are determined, in step S311, the graph encoder is updated according to the contrast losses of the plurality of sample pairs.

[0107] In one embodiment, a plurality of sample pairs constitute a batch of training samples, which include several positive sample pairs and several negative sample pairs. According to the sum or mean of the contrast losses of each sample pair, the training loss of the batch can be obtained. With the goal of minimizing the training loss, the encoder to be trained is updated. As mentioned above, for positive sample pairs, the contrast loss is positively correlated with their representation difference, and for negative sample pairs, the contrast loss is negatively correlated with the representation difference. Therefore, minimizing the training loss means that the representation difference of the positive sample pairs is minimized and the representation difference of the negative sample pairs is maximized. In this way, the graph representation and the text representation of the same subgraph are aligned or close to each other, and the graph representation and the text representation of different subgraphs are far away from each other.

[0108] In some embodiments, the text encoder may be a pre-trained encoder. In order to prevent the text encoder from overfitting and losing intrinsic text information, in this step, only the parameters of the graph encoder are updated, and the parameters of the text encoder are not updated.

[0109] In one implementation of this embodiment, based on the contrast loss, while updating the graph encoder, the parameters of the first mapping neural network and the second mapping neural network in the above steps are also updated.

[0110] This embodiment trains the graph encoder based on the knowledge graph, introduces the text encoder, and aligns it with the graph encoder in multimodality. By aligning the encoders of two different modalities, it is possible to prepare for the fusion of features in different modalities, realize cross-modal representation processing and understanding, and enable the graph encoder to understand the knowledge graph from the perspective of text, thereby enhancing the expressiveness of the model and improving the explanatory power of the extracted representations.

[0111] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] Figure 7 A device for training a graph encoder based on a knowledge graph is provided according to an embodiment of this specification. The device 700 is deployed in a computing device, which can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. Figure 3 The method embodiment shown corresponds to the embodiment shown, wherein the knowledge graph includes multiple nodes recorded in the form of triples and the connecting edges between them. The device 700 includes:

[0113] The subgraph acquisition module 701 is configured to sample nodes and connecting edges from the knowledge graph according to a preset search strategy to obtain a plurality of subgraphs.

[0114] A sub-graph encoding module 702 is configured to input the first sub-graph into the graph encoder to obtain a graph representation corresponding to the first sub-graph;

[0115] A subgraph description module 703 is configured to determine a first description text based on the text corpus of each triple in the second subgraph;

[0116] A text encoding module 704, configured to input the first description text into a text encoder to obtain a corresponding text representation;

[0117] A loss determination module 705 is configured to determine a contrast loss of a first sample pair consisting of a first sub-image and a first description text based on a representation difference between the image representation and the text representation;

[0118] The encoder optimization module 706 is configured to update the image encoder according to the contrast loss of each of the multiple sample pairs.

[0119] According to another embodiment of the present specification, there is also provided a computer program product, including a computer program / instruction, which is executed by a processor to implement the aforementioned combination Figure 3 The steps of the method.

[0120] According to another embodiment, the present specification further provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the aforementioned combination is implemented. Figure 3 The steps of the method.

[0121] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0122] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for training a graph encoder based on a knowledge graph, wherein the knowledge graph includes a plurality of nodes recorded in the form of triples and the connecting edges between them; the method comprises: According to a preset search strategy, nodes and connecting edges are sampled from the knowledge graph to obtain a plurality of subgraphs; Inputting the first sub-graph into the graph encoder to obtain a graph representation corresponding to the first sub-graph; Determine a first description text based on the text corpus of each triple in the second subgraph; Inputting the first description text into a text encoder to obtain a corresponding text representation; Determining a contrast loss of a first sample pair consisting of a first sub-image and a first description text based on a representation difference between the image representation and the text representation; The graph encoder is updated according to respective contrast losses of a plurality of sample pairs.

2. The method according to claim 1, wherein: The text encoder is a text encoder obtained after pre-training.

3. The method according to claim 1, wherein: The determining, based on the representation difference between the image representation and the text representation, a contrast loss of a first sample pair consisting of a first sub-image and a first description text comprises: Projecting the graph representation using a first mapping neural network to obtain a first projection; Projecting the text representation using a second mapping neural network to obtain a second projection, wherein the second projection and the first projection are in the same vector space; Based on a difference between the first projection and the second projection, the characterizing difference is determined.

4. The method according to claim 3, further comprising: According to the contrast loss, parameters of the first mapping neural network and the second mapping neural network are updated.

5. The method according to claim 1, wherein: Determining a contrast loss of a first sample pair consisting of a first sub-image and a first description text, including: If the first sub-image and the second sub-image are the same sub-image, the contrast loss is positively correlated with the representation difference; If the first sub-image and the second sub-image are different sub-images, the contrast loss is negatively correlated with the representation difference.

6. The method according to claim 1, wherein the plurality of sample pairs include positive sample pairs and negative sample pairs, wherein: The description text in the positive sample pair is used to describe the corresponding sub-image, while the description text in the negative sample pair is not used to describe the corresponding sub-image.

7. The method according to claim 1, wherein: The preset search strategy includes: Starting from the target node, a breadth-first or depth-first traversal search is performed according to a preset search depth and / or search width.

8. The method according to claim 1, wherein: The preset search strategy includes: Starting from the target node, a random walk search is performed for a predetermined number of steps.

9. The method according to claim 1, wherein: The determining of the first description text includes: For any of the triples, concatenate the description text of the entity represented by the node and the description text of the semantic association represented by the connecting edge to form the text corpus; The text corpora corresponding to each triple in the second subgraph are concatenated to obtain the first description text.

10. The method according to claim 1, wherein: The obtaining of the graph representation corresponding to the first subgraph includes: Encoding the first subgraph using the graph encoder to obtain node representations of each node; Based on the node representations of the respective nodes, a pooling operation is performed to obtain the graph representation.

11. A device for training a graph encoder based on a knowledge graph, wherein the knowledge graph includes a plurality of nodes recorded in the form of triples and the connecting edges between them; the device comprises: A subgraph acquisition module is configured to sample nodes and connecting edges from the knowledge graph according to a preset search strategy to obtain a plurality of subgraphs; a sub-graph encoding module, configured to input the first sub-graph into the graph encoder to obtain a graph representation corresponding to the first sub-graph; A subgraph description module, configured to determine a first description text based on the text corpus of each triple in the second subgraph; A text encoding module, configured to input the first description text into a text encoder to obtain a corresponding text representation; A loss determination module, configured to determine a contrast loss of a first sample pair consisting of a first sub-image and a first description text based on a representation difference between the image representation and the text representation; The encoder optimization module is configured to update the image encoder according to the contrast loss of each of the multiple sample pairs.

12. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

13. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 10 is implemented.