Correlation prediction based on heterogeneous graph learning

Abstract

Methods, apparatuses, and computer program products for relevance prediction based on heterogeneous graph learning are presented. A query and a keyword can be received. A heterogeneous graph corresponding to the query and the keyword can be generated. Query graph structure information for the query can be inferred from the heterogeneous graph. A query representation of the query can be generated based on the query graph structure information. Keyword graph structure information for the keyword can be inferred from the heterogeneous graph. A keyword representation of the keyword can be generated based on the keyword graph structure information. Relevance between the query and the keyword can be predicted based on the query representation and the keyword representation.

Description

Background Technology

[0001] With the development of network technology and the growth of online information, recommender systems are playing an increasingly important role in many online services. Different recommender systems exist based on the content being recommended, such as product recommender systems, news recommender systems, video recommender systems, and book recommender systems. These systems can typically perform personalized recommendations for users. For example, a recommender system usually predicts content that a user might be interested in based on their query and recommends it to them. Summary of the Invention

[0002] This invention is provided to introduce a set of concepts, which will be further described in the following detailed description. This invention is not intended to identify key or essential features of the protected subject matter, nor is it intended to limit the scope of the protected subject matter.

[0003] Embodiments of this disclosure provide methods, apparatus, and computer program products for relevance prediction based on heterogeneous graph learning. The system can receive queries and keywords. It can generate a heterogeneous graph corresponding to the query and the keywords. Query graph structure information for the query can be inferred from the heterogeneous graph. A query representation of the query can be generated based on the query graph structure information. Keyword graph structure information for the keywords can be inferred from the heterogeneous graph. A keyword representation of the keywords can be generated based on the keyword graph structure information. The relevance between the query and the keywords can be predicted based on the query representation and the keyword representation.

[0004] It should be noted that one or more of the above aspects include the features described in detail below and specifically pointed out in the claims. The following description and drawings set forth certain illustrative features of the one or more aspects in detail. These features merely indicate various ways in which the principles of each aspect can be employed, and this disclosure is intended to include all such aspects and their equivalents. Attached Figure Description

[0005] The following description will take into account several aspects disclosed, which are provided to illustrate rather than limit the aspects disclosed.

[0006] Figure 1 An exemplary heterogeneous diagram according to an embodiment of this disclosure is shown.

[0007] Figure 2 An exemplary process for generating heterogeneous graphs according to embodiments of the present disclosure is shown.

[0008] Figure 3 An exemplary process for constructing an original heterogeneous graph according to an embodiment of this disclosure is shown.

[0009] Figure 4 An exemplary process for generating an enhanced heterogeneous graph according to an embodiment of this disclosure is shown.

[0010] Figure 5 An exemplary process for pruning an enhanced heterogeneous graph according to an embodiment of this disclosure is shown.

[0011] Figure 6 An exemplary process for relevance prediction based on heterogeneous graph learning according to embodiments of this disclosure is shown.

[0012] Figure 7 An exemplary process for inferring query graph structure information for a query, according to embodiments of the present disclosure, is illustrated.

[0013] Figure 8 An exemplary process for generating a query representation according to an embodiment of this disclosure is shown.

[0014] Figure 9 An exemplary process for performing graph encoding operations according to embodiments of the present disclosure is shown.

[0015] Figure 10 An exemplary process for generating a query representation based on an embedded vector sequence, according to an embodiment of the present disclosure, is shown.

[0016] Figure 11 An exemplary process for performing correlation prediction using a lightweight correlation prediction model, according to embodiments of this disclosure, is illustrated.

[0017] Figure 12 This is a flowchart of an exemplary method for relevance prediction based on heterogeneous graph learning, according to embodiments of the present disclosure.

[0018] Figure 13 An exemplary apparatus for relevance prediction based on heterogeneous graph learning, according to embodiments of the present disclosure, is shown.

[0019] Figure 14 An exemplary apparatus for relevance prediction based on heterogeneous graph learning, according to embodiments of the present disclosure, is shown. Detailed Implementation

[0020] This disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that this discussion of embodiments is merely intended to enable those skilled in the art to better understand and thus implement the embodiments of this disclosure, and is not intended to teach any limitation on the scope of this disclosure.

[0021] Machine learning models can be used to predict the relevance between user queries and keywords for candidate content items, and to determine which content items to recommend to the user based on the predicted relevance. In this paper, a content item refers to an individual item with specific content. For example, a news article, a movie, a video, or a product can be considered a content item. A query can be a search query from the user, which reflects the user's intent. Keywords can be keywords for candidate content items to be recommended to the user. By predicting the relevance between the query and keywords, it can be determined whether the corresponding candidate content items match the user's intent. In this paper, the machine learning model used to predict the relevance between queries and keywords is called a relevance prediction model. The relevance prediction model can first generate query representations of the query and keyword representations of the keywords, and then predict the relevance between the query and keywords based on the generated query and keyword representations. Queries or keywords often contain only a few words or phrases, and therefore contain only limited semantic information. The user's historical click behavior can be used as supplementary information to enrich the query and keyword representations. Currently, historical click behavior includes click behavior between queries and keywords. For example, for a given query, multiple content items can be presented, each with corresponding keywords. When a user clicks on a content item, a click relationship is considered to exist between the query and the keyword of that content item. A homogeneous graph can be constructed based on this click relationship. For example, the query and keyword can be set as query nodes and keyword nodes, respectively. When a click relationship exists between a query and a keyword, an edge can exist between the corresponding query node and keyword node. Existing relevance prediction models can model homogeneous graphs. For example, they can identify the neighboring nodes of query nodes and keyword nodes from the homogeneous graph and use the information from the neighboring nodes to generate query representations of the query and keyword representations of the keyword accordingly. However, existing relevance prediction models, when modeling homogeneous graphs, treat nodes in the graph as having the same attributes and only consider one type of relationship. This approach offers limited improvement to the relevance prediction task.

[0022] This disclosure proposes a relevance prediction method based on heterogeneous graph learning. It can receive queries and keywords. Queries can be search queries from users. Keywords can be keywords for candidate content items to be recommended to users. A heterogeneous graph corresponding to the received queries and keywords can be generated. Unlike homogeneous graphs used in traditional relevance prediction tasks, the heterogeneous graph according to this disclosure can contain a set of nodes with multiple node types and a set of edges with multiple relationship types. Node types can include, for example, query nodes, keyword nodes, title nodes, entity nodes, etc. Relationship types can include, for example, query-keyword click relationships, query-title click relationships, query-entity extraction relationships, keyword-entity extraction relationships, keyword-keyword co-occurrence relationships, etc. Compared to homogeneous graphs, heterogeneous graphs can contain more comprehensive information and richer semantics, which can more effectively improve the performance of relevance prediction tasks. Query graph structure information for queries and keyword graph structure information for keywords can be inferred from the heterogeneous graph. Query representations of queries and keyword representations of keywords can be generated based on the query graph structure information and keyword graph structure information, respectively. The generated query representation and keyword representation can be used to predict the relevance between the query and the keywords.

[0023] This disclosure proposes constructing heterogeneous graphs based on search logs, knowledge graphs, etc. Heterogeneous graphs can be constructed using various historical user click behaviors extracted from search logs and various entities extracted from knowledge graphs. The constructed original heterogeneous graph can contain more comprehensive information and richer semantics, which helps alleviate the graph sparsity problem. However, historical click behaviors are often random and volatile. For example, users may frequently click on popular content items while rarely clicking on less popular ones. Such behavior generates long-tail nodes with only a few graph connections, causing the sparsity problem to persist. This disclosure proposes performing graph augmentation operations on the heterogeneous graph. Graph augmentation aims to generate edges relevant to the relevance prediction task by mining the complex relationships between node semantic information and the graph topology, thereby further mitigating the sparsity of the heterogeneous graph. Additionally, users may mistakenly click on unwanted content items. Such behavior generates redundant graph connections irrelevant to the relevance prediction task, resulting in a noisy graph topology. This disclosure proposes performing graph pruning operations on the heterogeneous graph. Graph pruning aims to prune heterogeneous graphs based on the importance of neighboring nodes relative to the central node. This approach removes edges irrelevant to the relevance prediction task while retaining useful edges. The process employs a coarse-to-fine strategy to learn task-relevant graph connections. First, edges relevant to the relevance prediction task are generated in a coarse manner, and then edges irrelevant to the task are removed in a fine manner. This method leverages the semantic information and topological structure of the node set in the heterogeneous graph, significantly mitigating the sparsity and noise problems associated with heterogeneous graphs.

[0024] To reveal the diverse topologies in heterogeneous graphs, embodiments of this disclosure define meta-paths. In this document, the path used to connect two nodes is referred to as a meta-path. Depending on the type of nodes being connected, there can be various meta-path types, such as query-keyword type, query-entity type, query-title type, query-title-query type, query-keyword-query type, keyword-keyword type, keyword-entity type, keyword-query-keyword type, etc.

[0025] The relevance between a query and keywords can be predicted using a relevance prediction model based on heterogeneous graph learning, according to embodiments of this disclosure. The relevance prediction model based on heterogeneous graph learning includes a graph encoder. The graph encoder learns a heterogeneous graph by mining the complex relationships between semantic information of nodes and the graph topology, and encodes the learned textual semantics and structural features into query and keyword representations, thereby generating enhanced query and keyword representations. The graph encoder can aggregate information from neighboring nodes around a node into the representation of that node based on the topology of the heterogeneous graph. The graph encoder can include intra-metapath encoders, inter-metapath encoders, node encoders, etc. Intra-metapath encoders enable information transfer between nodes belonging to the same metapath. Inter-metapath encoders enable information transfer between nodes spanning different metapaths. Node encoders can aggregate information from the same metapath and information from different metapaths into the representation of the corresponding node, for example, into the query representation of a query node or the keyword representation of a keyword node, thereby generating enhanced query or keyword representations. Using query and keyword representations generated in this way to predict the relevance between a query and keywords can significantly improve the relevance prediction results.

[0026] Heterogeneous graphs can be used Marked as ,in It is a set of nodes, and It is a set of edges. A heterogeneous graph can be defined using two mapping functions. For example, node type mapping functions and link type mapping function ,in It is a collection of node types, and It is a set of relation types.

[0027] Node type collection This can include, for example, query nodes. Keyword nodes Title node Entity nodes Query nodes, keyword nodes, and title nodes are set up based on components extracted from search logs. Search logs can be logs associated with a search engine that record user search behavior. Queries, keywords, and titles can be extracted from search logs. Queries can be search queries from users. Keywords can be keywords for content items. Content items can include news, videos, movies, books, music, web pages, product information, etc. Titles can be the titles of web pages found in response to a query. The extracted queries, keywords, and titles can be set as query nodes, keyword nodes, and title nodes, respectively. Entity nodes are set up based on components extracted from knowledge graphs. A knowledge graph is a structured semantic knowledge base that describes concepts and their relationships in the physical world in symbolic form. Knowledge graphs contain a large number of entities and relationships between entities. Entities associated with queries or keywords can be extracted from knowledge graphs. The extracted entities can be set as entity nodes. This will be discussed later in conjunction with... Figure 3 This illustrates an exemplary process for setting up nodes.

[0028] Relation types are also called edge types. A set of relation types. This can include, for example, query-keyword click relationships. Query - Title Click Relationship Query - Entity Relationship Extraction Keyword-entity extraction relationship Keyword-keyword co-occurrence relationship wait.

[0029] Query-keyword click relationships describe the relationship between queries and keywords. Multiple queries and multiple click keywords for each query can be obtained from search logs. In this article, query click keywords refer to the keywords of content items that were clicked by users among the multiple content items presented for that query. Multiple click keywords for a query can be sorted based on click frequency. A query-keyword click relationship can exist between the query and a predetermined number of top-ranking click keywords, such as the top 5.

[0030] The query-title click relationship describes the relationship between a query and its titles. Multiple queries and multiple clicked titles for each query can be obtained from search logs. In this article, the clicked titles of a query refer to the titles of the web pages that were clicked by the user among the multiple web pages retrieved for that query. The multiple clicked titles of a query can be sorted based on click frequency. A query-title click relationship can exist between the query and a predetermined number of the top-ranked clicked titles, such as the top 5.

[0031] Query-entity extraction relationships describe the relationships between queries and entities. Multiple entities associated with each query can be extracted from the knowledge graph. Each entity has a corresponding confidence score. Multiple entities in a query can be ranked based on their confidence scores. A query-entity extraction relationship can exist between the query and a predetermined number of entities in the top-ranking list, such as the top three.

[0032] Keyword-entity extraction relationships describe the relationships between keywords and entities. Multiple entities associated with each keyword can be extracted from the knowledge graph. Each keyword has a corresponding confidence score. Multiple entities associated with a keyword can be ranked based on their confidence scores. A keyword can have a keyword-entity extraction relationship with a predetermined number of entities ranked at the top, for example, the top three.

[0033] Keyword-keyword co-occurrence relationships describe the relationships between keywords. Content providers typically indicate a set of keywords. Two keywords co-occurring may mean that the two keywords are similar. For a given keyword, its co-occurring keywords can be ranked based on their co-occurrence frequency. A keyword can have a keyword-keyword co-occurrence relationship with a predetermined number of keywords ranking in the top position, such as the top 5.

[0034] To reveal the diverse topologies in heterogeneous graphs, embodiments of this disclosure define meta-paths. A meta-path can be defined as a path. (abbreviated as) ), which describes the nodes With nodes The composite relationship between them ,in It is a compound operator for relations. Depending on the type of the nodes being connected, it can have various meta-path types, such as query-keyword. Type, Query - Entity Type, Query - Title Type, Query - Title - Query Type, Query - Keyword - Query Type, Keywords - Keywords Type, Keywords - Entity Type, Keywords - Query - Keywords Types, etc.

[0035] Figure 1 An exemplary heterogeneous diagram 100 according to an embodiment of the present disclosure is shown.

[0036] Heterogeneous graph 100 contains four node types, such as query nodes. Keyword nodes Title node Entity nodes For example, node 102 is a query node. Nodes 104, 112, and 114 are keyword nodes. Node 106 is the title node. Nodes 108 and 110 are physical nodes. ,etc.

[0037] In addition, Heterogeneous Graph 100 includes five relationship types, such as query-keyword click relationship. Query - Title Click Relationship Query - Entity Relationship Extraction Keyword-entity extraction relationship Keyword co-occurrence relationship For example, the relationship type between node 102 and node 104 is a query-keyword click relationship. The relationship between node 102 and node 106 is a query-title click relationship. The relationship type between node 102 and node 108 is a query-entity retrieval relationship. The relationship between node 104 and node 110 is a keyword-entity extraction relationship. The relationship between node 112 and node 114 is a keyword-keyword co-occurrence relationship. ,etc.

[0038] The following will be combined Figures 2 to 5 This illustrates an exemplary process for generating heterogeneous graphs. It should be understood that... Figure 1 The heterogeneous graph 100 shown is merely an example of a heterogeneous graph. Depending on the specific application requirements, heterogeneous graphs can have any other structure and can include more or fewer elements.

[0039] Figure 2 An exemplary process 200 for generating a heterogeneous graph according to an embodiment of the present disclosure is shown. In process 200, a heterogeneous graph 212 may be generated by a heterogeneous graph generation module 210 based on a search log 202 and / or a knowledge graph 204.

[0040] The heterogeneous graph generation module 210 includes a graph building unit 220. The graph building unit 220 is designed to incorporate various types of nodes and edges into the graph. The original heterogeneous graph 222 can be constructed using the graph building unit 220, based on the search log 202 and / or the knowledge graph 204. Figure 3 An exemplary process 300 for constructing an original heterogeneous graph according to an embodiment of this disclosure is illustrated. Process 300 can be performed by... Figure 2 The graph construction unit 220 in the middle is executed.

[0041] At position 302, a set of components can be extracted from search logs and / or knowledge graphs. The set of components may include historical query sets, historical keyword sets, historical title sets, etc., extracted from the search logs. These set of components may be extracted from various historical click behaviors of users contained in the search logs. Preferably, the set of components may also include entity sets corresponding to historical query sets, entity sets corresponding to historical keyword sets, etc., extracted from the knowledge graph.

[0042] At position 304, the component set can be set as a node set. For example, historical queries can be set as query nodes, historical keywords as keyword nodes, historical titles as title nodes, entities as entity nodes, and so on. Then, the set of edges between the node sets can be determined.

[0043] At point 306, for every two nodes in the node set, it can be determined whether a relationship exists between the two components corresponding to those two nodes. Relationships can include, for example, query-keyword click relationships, query-title click relationships, query-entity extraction relationships, keyword-entity extraction relationships, keyword-keyword co-occurrence relationships, etc.

[0044] At point 308, in response to determining that a relationship exists between the two components, an edge can be determined to exist between the two nodes. The edge type between the two nodes can correspond to the relationship type between the two components corresponding to those two nodes.

[0045] Steps 306 and 308 can be executed for every two nodes in the node set, so that at 310, the set of edges between the node sets can be obtained.

[0046] At position 312, the set of nodes and the set of edges can be combined to form the original heterogeneous graph. Preferably, nodes without any edges can be removed from the original heterogeneous graph.

[0047] Through process 300, a primitive heterogeneous graph can be constructed using various historical click behaviors of users extracted from search logs and various entities extracted from knowledge graphs. The constructed primitive heterogeneous graph can contain more comprehensive information and richer semantics, which helps to alleviate the graph sparsity problem. However, historical click behaviors are often random and volatile. For example, users may often click on some popular and trending content items, while rarely clicking on less popular content items. Such behavior will generate some long-tail nodes with only a few graph connections, which makes the sparsity problem still exist. Embodiments of this disclosure propose to perform graph augmentation operations on the primitive heterogeneous graph.

[0048] Return to reference Figure 2After obtaining the original heterogeneous graph 222, an enhanced heterogeneous graph 232 can be generated using the graph enhancement unit 230 in the heterogeneous graph generation module 210. The graph enhancement unit 230 aims to generate potential edges relevant to the relevance prediction task by mining the complex relationships between node semantic information and the graph topology, thereby further mitigating the sparsity of the heterogeneous graph. The enhanced heterogeneous graph 232 can be generated using the graph enhancement unit 230 based on the semantic information and / or topology of the node set of the original heterogeneous graph 222.

[0049] Figure 4 An exemplary process 400 for generating an enhanced heterogeneous graph according to an embodiment of this disclosure is illustrated. Process 400 can be performed by… Figure 2 The graph enhancement unit 230 in the diagram performs the enhancement. Different relationship types may have unique meanings and information content. Based on this, embodiments of this disclosure propose a relationship type-based enhancement strategy. The original heterogeneous graph may include multiple original partial graphs corresponding to multiple relationship types. In this document, an original partial graph refers to a partial graph of the original heterogeneous graph for a specific relationship type. The original partial graph may contain a subset of nodes corresponding to that relationship type. For example, a query-keyword click relationship... The corresponding original partial graph can include a subset of nodes consisting of query nodes and keyword nodes. This can be used to analyze the query-keyword click relationship. The original partial diagram is labeled as ,in It is the number of query nodes in the node subset, and This represents the number of keyword nodes in the node subset. In process 400, for each of the multiple relation types, an augmented partial graph corresponding to that relation type can be generated. Subsequently, multiple augmented partial graphs corresponding to multiple relation types can be combined into an augmented heterogeneous graph. Steps 402 to 406 will use the relation type as the query-keyword click relationship. For example, we will illustrate an exemplary process for generating an enhanced partial graph.

[0050] At position 402, a semantic partial graph corresponding to a given relation type can be generated based on the semantic similarity between two nodes in a subset of nodes of the original partial graph corresponding to that relation type. The semantic partial graph can be labeled as... Semantic part graphs can be used to capture the semantic similarity between two corresponding nodes. For example, for query-keyword click relationships... The semantic partial graph can capture query nodes With keyword nodes Semantic similarity between them. Query nodes with similar semantics. and keyword nodes There may be connections between them. A query node can be generated first. Embedding vector and keyword nodes Embedded vector As shown in the following formula: in, It can be any pre-learned text encoder. As an example, This could be a TwinBert model fine-tuned through relevance labeling. Embedded vectors. and embedding vector These are for query nodes. and keyword nodes of Dimensional embedding vector.

[0051] Subsequently, the embedding vector can be added. and embedding vector Applying metric learning to generate query-keyword click relationships Corresponding semantic part diagram As shown in the following formula: in, It is a hyperparameter used to control sparsity. It is used for measurement and Between The head-weighted cosine similarity function is shown in the following formula: in, It is the Hadamard product, and It is a learnable vector used to indicate the importance of different dimensions.

[0052] It should be understood that other methods can also be used for measurement. and The similarity between them. For example, for an original heterogeneous graph containing a large number of nodes, it can also be measured. and The ordinary cosine similarity between them can be efficiently calculated using the Approximate Nearest Neighbors (ANN) search method.

[0053] At position 404, a propagation partial graph corresponding to the relationship type can be generated based on the semantic similarity between two nodes of the same type in a subset of nodes and the topology of the original partial graph. Nodes of the same type with similar semantics may have similar neighboring nodes. The propagation partial graph can capture the associations and topology between nodes. The following example illustrates the process of generating a query propagation partial graph using a query node as an example. First, the semantic matrix between query nodes can be calculated. As shown in the following formula: in, The form is the same as formula (4), but it has different characteristics. Alternatively, It can also be a regular cosine similarity measurement function.

[0054] Subsequently, a query propagation graph can be obtained. As shown in the following formula:

[0055] Original partial image It can reflect the original connections between nodes based on user historical behavior. Through the query propagation graph, the semantic similarity between query nodes can be propagated between nodes through such original connections.

[0056] This can be achieved by generating a query propagation graph as described above. In a similar manner, generate a keyword propagation part diagram. .

[0057] At position 406, it can be based on the original partial graph. Semantic Part Diagram and propagation part diagram and This generates an enhanced partial graph corresponding to the relationship type. Continue by using the relationship type as the query keyword and clicking on the relationship. For example, generate an enhanced partial graph corresponding to this relationship type. It can be represented by the following formula: in, It is a combination function. As an example, It can be an efficient summation operation.

[0058] Steps 402 to 406 can be performed for other relation types, so that at 408, multiple enhanced partial graphs corresponding to multiple relation types can be obtained.

[0059] At position 410, multiple augmented partial graphs can be combined into an augmented heterogeneous graph.

[0060] Through process 400, the sparse original heterogeneous graph can be enhanced to obtain a denser enhanced heterogeneous graph, thereby further mitigating the sparsity of the heterogeneous graph. However, this denser enhanced heterogeneous graph may contain some noise. For example, a user might mistakenly click on some unwanted content items. Such behavior generates redundant edges irrelevant to the relevance prediction task, resulting in a noisy graph topology. According to embodiments of this disclosure, the enhanced heterogeneous graph can be pruned. The graph pruning operation aims to prune the enhanced heterogeneous graph based on the importance of neighboring nodes relative to the central node. This approach can remove edges irrelevant to the relevance prediction task while retaining useful edges.

[0061] Return to reference Figure 2 After the enhanced heterogeneous graph 232 is generated by the graph enhancement unit 230, it can be pruned by the graph pruning unit 240 in the heterogeneous graph generation module 210 to obtain the heterogeneous graph 212. The enhanced heterogeneous graph 232 can be pruned based on the semantic information and / or topological structure of its node set. The graph pruning unit 240 aims to remove noisy edges that are irrelevant to the relevance prediction task while retaining useful edges. The enhanced heterogeneous graph 232 can be pruned by the graph pruning unit 240 based on the importance of neighboring nodes relative to the central node in its node set.

[0062] To reveal the diverse topologies in heterogeneous graphs, embodiments of this disclosure define a set of neighbor nodes based on meta-paths. For example, given a node... and a meta-path type This node Meta-path type The set of neighboring nodes Refers to the path type as Metapath and Node The set of connected nodes. At this point, the nodes... These can also be referred to as central nodes. An enhanced heterogeneous graph's node set can include multiple central nodes. Each central node can have various meta-path types. Corresponding set of multiple neighbor nodes Noisy or redundant neighbor nodes can be removed from each set of neighbor nodes through pruning operations. Figure 5 An exemplary process 500 for pruning an enhanced heterogeneous graph according to an embodiment of the present disclosure is shown.

[0063] At position 502, the most important subset of neighboring nodes for the central node can be identified from the set of neighboring nodes based on a meta-path type. The following uses the central node as the query node. And the meta path type is query-keyword Let's take an example to illustrate. We can query nodes... Meta-path type The set of neighboring nodes is labeled as The set of neighboring nodes can be calculated using the following formula. Keyword nodes Relative to the central node, i.e., the query node The importance of : (8) Among them, the function It can be implemented as a multi-layer feedforward network with an output dimension of 1.

[0064] The above method aims to learn important neighboring nodes by incorporating the semantic information and topological structure of the node. This can be achieved by utilizing... From the set of neighboring nodes Sampling for query nodes The most important pre-ordered quantity, for example One, neighboring nodes.

[0065] Preferably, in order to make the sampling process differentiable, the Gumbel-Softmax technique can be introduced to generate discrete samples, as shown in the following formula: in, It is used to control the - The temperature at which the softmax function approximates the discrete categorical distribution. At this point, it can be used... From the set of neighboring nodes Sample the query node The most important pre-ordered quantity, for example One, neighboring nodes.

[0066] The sampled The neighboring nodes are combined to form a subset of the neighboring nodes that are most important to the central node. At position 504, the neighboring nodes that are not in this subset can be identified. Then, at position 506, the edges between the identified neighboring nodes and the central node can be removed. The graph with the edges removed can be called a subgraph.

[0067] The graph pruning operation described above aims to prune heterogeneous graphs based on the importance of neighboring nodes relative to the central node. This approach removes edges irrelevant to the relevance prediction task while retaining useful edges. The process employs a coarse-to-fine strategy to learn task-relevant graph connections, first generating edges relevant to the relevance prediction task in a coarse manner, and then removing edges irrelevant to the relevance prediction task from the graph in a fine manner. This method leverages the semantic information and topological structure of the node set in the heterogeneous graph, significantly mitigating the sparsity and noise problems of heterogeneous graphs.

[0068] It should be understood that the above text, in combination with... Figures 2 to 5 The described process for generating heterogeneous graphs is merely exemplary. Depending on the specific application requirements, the steps in the process for generating heterogeneous graphs can be replaced or modified in any way, and the process may include more or fewer steps. For example, although both search log 202 and knowledge graph 204 are shown in process 200, in some embodiments, it is possible to construct the original heterogeneous graph 222 based solely on search log 202. Furthermore, although the heterogeneous graph generation module 210 in process 200 includes a graph construction unit 220, a graph enhancement unit 230, and a graph pruning unit 240, in some embodiments, the heterogeneous graph generation module 210 may not include the graph enhancement unit 230 or the graph pruning unit 240. When the heterogeneous graph generation module 210 does not include the graph enhancement unit 230, the graph pruning unit 240 can prune the original heterogeneous graph 222 constructed by the graph construction unit 210 to obtain the final heterogeneous graph 212. When the heterogeneous graph generation module 210 does not include the graph pruning unit 240, the enhanced heterogeneous graph 232 generated by the graph enhancement unit 230 can be directly used as the final heterogeneous graph 212. Furthermore, the specific order or hierarchy of the steps in processes 200 to 500 is merely exemplary, and the processes for generating heterogeneous graphs can be performed in an order different from the described order.

[0069] Figure 6 An exemplary process 600 for relevance prediction based on heterogeneous graph learning according to an embodiment of this disclosure is shown. In process 600, a query 602 and a keyword 604 may be received. The keyword 604 is a keyword for candidate content items. Candidate content items may be content items from a set of candidate content items that can be recommended to a user. Candidate content items may include news, videos, movies, books, music, web pages, product information, etc. The relevance 612 between query 602 and keyword 604 can be predicted by a relevance prediction model 610 based on heterogeneous graph learning.

[0070] First, a heterogeneous graph 622 corresponding to query 602 and keyword 604 can be generated. For example, the heterogeneous graph 622 can be generated using the heterogeneous graph generation module 620 in the heterogeneous graph learning-based relevance prediction model 610, based on search log 606 and / or knowledge graph 608. The heterogeneous graph generation module 620 can correspond to... Figure 2 The heterogeneous graph generation module 210 in the middle.

[0071] Subsequently, the graph structure information 632 for query 602 can be inferred from the heterogeneous graph 622 by the graph structure inference module 630 in the heterogeneous graph-based relevance prediction model 610. Figure 7 An exemplary process 700 for inferring query graph structure information for a query according to an embodiment of this disclosure is shown. Process 700 can be performed by... Figure 6 The graph structure inference module 630 is executed. At 702, the meta-path to which the query node corresponding to the query belongs can be inferred based on the heterogeneous graph. The query node may belong to one or more meta-paths. For example, the query node... The metapath to which it belongs can be tagged as ,in This refers to the number of meta-paths. At position 704, the set of nodes based on meta-paths can be inferred from the heterogeneous graph. When one or more meta-paths exist, the set of nodes based on each meta-path can be inferred, thus obtaining one or more node sets. Meta-path-based... The set of nodes is labeled as ,in This represents the number of nodes included in the node set. At point 706, query graph structure information can be generated based on the metapath and node set. This graph structure information can be marked as... ,in This represents the central node, i.e., the query node. .

[0072] Return to reference Figure 6 After inferring the graph structure information 632 for query 602, a query representation 646 for query 602 can be generated based on the graph structure information 632. Figure 8 An exemplary process 800 for generating a query representation according to an embodiment of the present disclosure is shown. Process 800 can be performed by… Figure 6 The text encoder 640, graph encoder 642, pooling module 644, etc. are executed.

[0073] At position 802, the metapath to which the query node belongs and its neighboring nodes based on the metapath can be obtained from the query graph structure information. A query node may belong to one or more metapaths. Each metapath may have one or more neighboring nodes.

[0074] At position 804, a sequence of tokens corresponding to the metapath can be generated based on the query and neighbor nodes. In this document, a token refers to a basic linguistic unit that constitutes text in different languages. To enable information transmission at different levels, embodiments of this disclosure propose two special types of tokens: local tokens [TRA] for transmitting information within the same metapath, and global tokens [TER] for transmitting information across different metapaths. First, query text tokens for the query and neighbor text tokens for neighbor nodes can be obtained through tokenization. Next, query local tokens [TRA] for the query can be added before the query text tokens. Similarly, neighbor local tokens [TRA] for neighbor nodes can be added before neighbor text tokens. Additionally, each metapath can have an associated global token [TER]. The global tokens, query local tokens, query text tokens, neighbor local tokens, and neighbor text tokens of the metapath can be combined to form a token sequence corresponding to the metapath.

[0075] At point 806, an initial embedding vector sequence corresponding to the metapath can be generated based on the word chunk sequence corresponding to the metapath. For example, the initial embedding vector sequence can be generated using text encoder 640.

[0076] At point 808, an embedding vector sequence corresponding to the meta-path can be generated by iteratively performing graph encoding operations based on the initial embedding vector sequence. By performing graph encoding operations, the previous embedding vector sequence corresponding to the meta-path can be updated to the current embedding vector sequence corresponding to the meta-path. For example, refer to... Figure 6 It can be done through multiple, for example ( Multiple graph encoders 642 are stacked together to iteratively perform graph encoding operations. The graph encoders 642 include intra-metapath encoders 642-1, inter-metapath encoders 642-2, and node encoders 642-3. Intra-metapath encoder 642-1 captures shared features of the same metapath to enable information transfer between nodes belonging to the same metapath. Inter-metapath encoder 642-2 enables information transfer between nodes spanning different metapaths. Node encoder 642-3 aggregates information from the same metapath and information from different metapaths into the representation of the corresponding node. This will be discussed later in conjunction with... Figure 9 This illustrates an exemplary process for performing graph encoding operations.

[0077] At point 810, a query representation can be generated based on the embedded vector sequence. For example, this can be achieved using pooling module 644, which generates the query representation based on the embedded vector sequence. This will be discussed later. Figure 10This illustrates an exemplary process for generating query representations based on embedded vector sequences.

[0078] The above process can generate a query representation of the query. For example, refer to... Figure 6 This can generate a query representation 646 for query 602. A keyword representation 666 for keyword 604 can be generated through a process similar to that used to generate query representation 646. The graph structure inference module 650 can infer the keyword graph structure information 652 for keyword 604 based on the heterogeneous graph 622. The structure and function of the graph structure inference module 650 can be similar to those of the graph structure inference module 630. Subsequently, the text encoder 660, The graph encoder 662 and pooling module 664 generate a keyword representation 666 for keyword 604 based on keyword graph structure information 652. The structure and function of the text encoder 660, graph encoder 662, and pooling module 664 can be similar to those of the text encoder 640, graph encoder 642, and pooling module 644, respectively. The graph encoder 662 includes an intra-metapath encoder 662-1, an inter-metapath encoder 662-2, and a node encoder 662-3, etc. The structure and function of the intra-metapath encoder 662-1, inter-metapath encoder 662-2, and node encoder 662-3 can be similar to those of the intra-metapath encoder 642-1, inter-metapath encoder 642-2, and node encoder 642-3, respectively.

[0079] After generating the query representation 646 and keyword representation 666, the prediction layer 670 can predict the relevance 612 between query 602 and keyword 604 based on the query representation 646 and keyword representation 666. The relevance 612 can be defined as cosine similarity, which can be further viewed as a probability score. After both the query representation 646 and the keyword representation 666 are normalized, the cosine similarity can be equated to the Euclidean distance and can be efficiently solved using the ANN search method.

[0080] The cross-entropy loss function can be chosen as the objective function for training the heterogeneous graph-based relevance prediction model 610, as shown in the following formula:

[0081] Figure 9 An exemplary process 900 for performing graph encoding operations according to an embodiment of this disclosure is shown. Process 900 can be performed by… Figure 6 One of the graph encoders, 642, is executed.

[0082] In procedure 900, the query node belongs to two meta-paths: a first-meta path and a second-meta path. As an example, the type of the first-meta path is query-title. This metapath contains query nodes and title nodes. The type of the second metapath is query-entity. This metapath contains query nodes and entity nodes. It should be understood that when a query node belongs to another type of metapath, or to multiple metapaths, graph encoding operations can be performed using a process similar to process 900. Process 900 can be used to update the previous embedding vector sequence 902 corresponding to the first metapath to the current embedding vector sequence 904, and the previous embedding vector sequence 906 corresponding to the second metapath to the current embedding vector sequence 908.

[0083] Can belong to metapath nodes The The previous embedding vectors of the word chunks are labeled as follows: . That is, when Time is a node The previous embedding vector of the local word chunk [TRA]. With metapath The previous embedding vector of the associated global word block [TER] can be labeled as .

[0084] The previous embedding vector sequence 902 corresponding to the first metapath includes previous embedding vectors corresponding to query nodes on the metapath, such as a previous local embedding vector 910 for the local term [TRA] of the query node and two previous text embedding vectors 912 and 914 for the two text terms of the query node. The previous embedding vector sequence 902 also includes a previous global embedding vector 916 for the global term [TER] associated with the first metapath. The previous embedding vector sequence 902 also includes previous embedding vectors corresponding to header nodes on the metapath, such as a previous local embedding vector 918 for the local term [TRA] of the header node and two previous text embedding vectors 920 and 922 for the two text terms of the header node.

[0085] The previous embedding vector sequence 906 corresponding to the second meta-path includes previous embedding vectors corresponding to query nodes on the meta-path, such as a previous local embedding vector 924 for the local term [TRA] of the query node and two previous text embedding vectors 926 and 928 for the two text terms of the query node. The previous embedding vector sequence 906 also includes a previous global embedding vector 930 for the global term [TER] associated with the second meta-path. The previous embedding vector sequence 906 also includes previous embedding vectors corresponding to entity nodes on the meta-path, such as a previous local embedding vector 932 for the local term [TRA] of the entity node and two previous text embedding vectors 934 and 936 for the two text terms of the entity node.

[0086] A self-attention mechanism can be used to generate intermediate global embedding vectors for global blocks and multiple intermediate local embedding vectors for multiple local blocks, based on the previous global embedding vectors of the global block associated with the meta-path and the multiple previous local embedding vectors of the multiple local blocks associated with the meta-path. Given a meta-path... and belonging to this metapath Node set It is possible to collect the previous global embedding vectors of the global chunks [TER] associated with the metapath and the previous local embedding vectors of the local chunks [TRA] of multiple nodes to form a metapath intramatrix: ,in yes The number of nodes in the matrix. Then, the matrix within the metapath can be analyzed. Multi-head self-attention is employed to pass information between nodes belonging to the same meta-path. For any attention head, information passing within a meta-path can be defined by the following formula: in, Among them, matrix ,matrix sum matrix It is a learnable variable.

[0087] The learned matrix can be labeled as This matrix contains information related to metapaths. The intermediate global embedding vector of the corresponding global word block [TER] and multiple intermediate local embedding vectors of multiple local word chunks [TRA] . This represents the intermediate local embedding vector of the local chunk [TRA] obtained through the encoder within the metapath, which can capture local heterogeneity within a single metapath. It can be compared with the metapath... The intermediate global embedding vector of the corresponding global word block [TER] Consider as a metapath The embedding vectors of other word chunks can contain contextual information centered around different nodes through a self-attention mechanism.

[0088] For example, for a first metapath, a metapath encoder 938 based on a self-attention mechanism can generate intermediate local embedding vectors 940, 942, and 944 corresponding to the first metapath, based on previous local embedding vectors 910, 916, and 918. Similarly, for a second metapath, a metapath encoder 946 based on a self-attention mechanism can generate intermediate local embedding vectors 948, 950, and 952 corresponding to the second metapath, based on previous local embedding vectors 924, 930, and 932. The metapath encoders 938 and 946 can correspond to... Figure 6 The metapath encoder 642-1.

[0089] Next, a self-attention mechanism can be used to generate the current global embedding vector of the global word block [TER] based on the intermediate global embedding vector of the global word block [TER]. For each metapath... This allows us to collect the metapath. The intermediate global embedding vector of the global word block [TER] obtained by the metapath encoder To form a matrix between metapaths: Metapath matrix Each row in the matrix represents the embedding vector of the corresponding metapath. This can be applied to the matrix between metapaths. The multi-head self-attention operation is employed, as shown in the following formula: in, This is the pattern attention proposed in the embodiments of this disclosure, which can be used to further reveal the relationships between different meta-paths. (Element) Metapath Patterns and metapaths The edit distance between patterns. The edit distance can be mapped to a learnable scalar and used as a bias term in the self-attention module. Additionally, in the above formula... , and as follows: Among them, matrix ,matrix sum matrix It is a learnable variable.

[0090] The learned matrix can be labeled as . Represents the metapath obtained through the metapath encoder. The current global embedding vector. Through the inter-metapath encoder, information can be transmitted across different metapaths, thus enabling the encoding of global heterogeneity across different metapaths.

[0091] For example, the current global embedding vector 956 and the current global embedding vector 958 can be generated by the meta-path encoder 954 based on the intermediate global embedding vector 942 and the intermediate global embedding vector 952.

[0092] Subsequently, for each node in the metapath, a self-attention mechanism can be used to generate the current local embedding vector and the current text embedding vector of that node, based on the current global embedding vector corresponding to the metapath, the intermediate local embedding vector of that node, and the previous text embedding vector. Given a node... It can be used for metapaths The corresponding current global embedding vector ,node intermediate local embedding vector and previous text embedding vectors Combine them to form a node matrix: ,in It is a node The number of text blocks.

[0093] Can be used for node matrix A multi-head self-attention operation is employed. This multi-head self-attention operation can be similar to Equation 12. The difference is that, in order to ensure that different nodes on the same meta-path can share the same global embedding vector, the global embedding vector can remain unchanged after this multi-head self-attention operation.

[0094] For example, for a query node in the first meta-path, a node encoder 960 based on a self-attention mechanism can generate the current local embedding vector 962 and the current text embedding vectors 964 and 966 of the query node based on the intermediate local embedding vector 940, the previous text embedding vectors 926 and 928, and the current global embedding vector 956. To ensure that different nodes on the same meta-path can share the same global embedding vector, the global embedding vector can remain unchanged after passing through the node encoder. Similarly, a node encoder 968 based on a self-attention mechanism can generate the current local embedding vector 970 and the current text embedding vectors 972 and 974 of the title node. Furthermore, for a query node in the second meta-path, a node encoder 976 based on a self-attention mechanism can generate the current local embedding vector 978 and the current text embedding vectors 980 and 982 of the query node. A node encoder 982 based on a self-attention mechanism can generate the current local embedding vector 984 and the current text embedding vectors 986 and 988 of the entity node. Node encoders 960, 968, 976, and 982 can correspond to Figure 6 The node encoder 642-3 in the middle.

[0095] Finally, the current global embedding vector corresponding to the meta-path, along with multiple current local embedding vectors and multiple current text embedding vectors corresponding to multiple nodes in the meta-path, can be combined into a current embedding vector sequence corresponding to the meta-path. For example, for the first meta-path, the current global embedding vector 956, the current local embedding vector 962 corresponding to the query node, and the current text embedding vectors 964 and 966, as well as the current local embedding vector 970 corresponding to the title node, and the current text embedding vectors 972 and 974, can be combined into a current embedding vector sequence 904.

[0096] In the above process, each word block can focus on other word blocks in the same meta-path via the local word block [TRA], thereby achieving local heterogeneity. Additionally, each word block can focus on other word blocks in other meta-paths via the global word block [TER], thereby achieving global heterogeneity. Through the local word block [TRA] and global word block [TER] proposed according to embodiments of this disclosure, information within the same meta-path and information spanning different meta-paths can be fused together. In this way, hierarchical heterogeneity can be deeply integrated into the semantic representation of nodes. Furthermore, each text word block only needs to focus on other word blocks within the same node. This can significantly reduce the number of learnable parameters and accelerate model training.

[0097] The current embedding vector sequence corresponding to each meta-path, obtained through the last graph encoding operation, can be considered as the embedding vector sequence corresponding to that meta-path. A query representation can be generated based on this embedding vector sequence. For example, local embedding vectors for the local term [TRA] for the query can be extracted from the embedding vector sequence. A query representation can be generated based on the extracted local embedding vectors. When a query node belongs to only one meta-path, one embedding vector sequence corresponding to that meta-path can be obtained. The local embedding vector for the local term [TRA] extracted from this embedding vector sequence can be considered as the query representation. When a query node belongs to multiple meta-paths, multiple embedding vector sequences corresponding to multiple meta-paths can be obtained. Local embedding vectors for the local term [TRA] can be extracted from each embedding vector sequence, resulting in multiple local embedding vectors. A query representation can be generated based on these multiple local embedding vectors.

[0098] Figure 10 An exemplary process 1000 for generating a query representation based on an embedded vector sequence, according to an embodiment of the present disclosure, is shown. Process 1000 can be performed by… Figure 6 The pooling module 644 executes. In process 1000, the query node belongs to two meta-paths: the first meta-path and the second meta-path. As an example, the type of the first meta-path is query-title. This metapath contains query nodes and title nodes. The type of the second metapath is query-entity. This metapath contains query nodes and entity nodes. It should be understood that when the query node belongs to other types of metapaths, or to multiple metapaths, a query representation can be generated through a process similar to process 1000.

[0099] Embedded vector sequences 1002 and 1004 correspond to the first-order path and the second-order path, respectively. Embedded vector sequences 1002 and 1004 can be obtained through... Figure 6 The last graph encoder 642 in the process is used to obtain the embedding vector sequence 1002 and the embedding vector sequence 1004. In other words, the embedding vector sequence 1002 and the embedding vector sequence 1004 can be obtained through the last execution. Figure 9 The process 900 in the middle is obtained. A local embedding vector 1006 for the query node targeting the local term [TRA] can be extracted from the embedding vector sequence 1002. Similarly, a local embedding vector 1008 for the query node targeting the local term [TRA] can be extracted from the embedding vector sequence 1004. Local embedding vectors 1006 and 1008 are provided to the pooling module 1010. The pooling module 1010 can correspond to... Figure 6Pooling module 644. Pooling module 1010 can perform pooling operations on local embedding vectors 1006 and 1008 to generate query representation 1012. It should be understood that generating query representations through pooling operations is only one example of generating query representations; other operations can also be used to generate query representations. Keyword representations can be generated using a process similar to that used to generate query representations.

[0100] Return to reference Figure 6 The aforementioned heterogeneous graph learning-based relevance prediction model 610 can predict the relevance 612 between query 602 and keyword 604. The heterogeneous graph learning-based relevance prediction model 610 includes graph encoders 642 and 662. Graph encoders 642 or 662 can learn a heterogeneous graph 622 by mining the complex relationships between node semantic information and the graph topology, and can encode the learned text semantics and structural features into a query representation 646 or a keyword representation 666, thereby generating an enhanced query representation 646 or keyword representation 666. Graph encoders 642 or 662 can aggregate information from neighboring nodes around a node into the representation of that node based on the topology of the heterogeneous graph 622. Each graph encoder can include an intra-metapath encoder, an inter-metapath encoder, a node encoder, etc. The intra-metapath encoder enables information transfer between nodes belonging to the same metapath. The inter-metapath encoder enables information transfer between nodes spanning different metapaths. The node encoder can aggregate information from the same meta-path and information from different meta-paths into the representation of the corresponding node, such as the query representation 646 of the query node or the keyword representation 666 of the keyword node, thereby generating an enhanced query representation or keyword representation. Using the query representation 646 and keyword representation 666 generated in this way to predict the relevance between query 602 and keyword 604 can significantly improve the relevance prediction results.

[0101] It should be understood that the above text, in combination with... Figures 6 to 10 The described process for relevance prediction based on heterogeneous graph learning is merely exemplary. Depending on the specific application requirements, the steps in the process can be replaced or modified in any way, and the process may include more or fewer steps. Furthermore, the specific order or hierarchy of the steps in processes 600 to 1000 is only exemplary, and the process for relevance prediction based on heterogeneous graph learning can be performed in an order different from the described order. Moreover, Figure 6 The heterogeneous graph learning-based correlation prediction model 610 shown is merely an example of a correlation prediction model based on heterogeneous graph learning. Depending on the specific application requirements, the heterogeneous graph learning-based correlation prediction model can have any other structure and can include more or fewer layers.

[0102] According to the heterogeneous graph learning-based relevance prediction model of embodiments of this disclosure, for example Figure 6 The heterogeneous graph learning-based relevance prediction model 610 can generate query and keyword representations by fusing rich contextual information related to queries and keywords. Because it includes multiple layers of graph encoders, this model may experience significant prediction latency, especially for new queries. A knowledge distillation strategy can be employed, treating the heterogeneous graph learning-based relevance prediction model as a teacher model and using it to learn a student model, such as a lightweight relevance prediction model. A large number of query-keyword pairs can be sampled from search logs and provided to the teacher model. Each query-keyword pair includes a query and a keyword. The teacher model can predict the relevance between the query and the keyword. The predicted relevance can be used as pseudo-labels, along with the corresponding query and keyword, to generate training samples for training the lightweight relevance prediction model. The trained lightweight relevance prediction model can then be deployed to perform relevance prediction tasks online.

[0103] Figure 11 An exemplary process 1100 for relevance prediction using a lightweight relevance prediction model according to an embodiment of the present disclosure is illustrated. In process 1100, a lightweight relevance prediction model 1110 is used to predict the relevance 1112 between query 1102 and keyword 1104. The lightweight relevance prediction model 1110 includes an embedding layer 1120 for query 1102 and... ( ) Transformer layer 1130, embedding layer 1140 for keyword 1104 and There is a transformer layer 1150 and a prediction layer 1160. It should be understood that... Figure 11 The lightweight correlation prediction model 1110 shown is merely an example of a lightweight correlation prediction model that can be deployed to perform correlation prediction tasks online. Depending on the specific application requirements, the lightweight correlation prediction model can have any other structure and can include more or fewer layers.

[0104] Figure 12 This is a flowchart of an exemplary method 1200 for relevance prediction based on heterogeneous graph learning according to embodiments of the present disclosure.

[0105] At position 1210, queries and keywords can be received.

[0106] At position 1220, a heterogeneous graph corresponding to the query and the keyword can be generated.

[0107] At position 1230, the query graph structure information for the query can be inferred based on the heterogeneous graph.

[0108] At position 1240, a query representation of the query can be generated based on the query graph structure information.

[0109] At position 1250, keyword graph structure information for the keyword can be inferred based on the heterogeneous graph.

[0110] At position 1260, a keyword representation of the keyword can be generated based on the keyword graph structure information.

[0111] At point 1270, the relevance between the query and the keyword can be predicted based on the query representation and the keyword representation.

[0112] In one implementation, the keywords may be keywords targeting candidate content items. The candidate content items may include at least one of news, videos, movies, books, music, web pages, and product information.

[0113] In one implementation, generating the heterogeneous graph may include: constructing an original heterogeneous graph based on search logs and / or a knowledge graph; generating an enhanced heterogeneous graph based on the semantic information and / or topological structure of the node set of the original heterogeneous graph; and pruning the enhanced heterogeneous graph based on the semantic information and / or topological structure of the node set of the enhanced heterogeneous graph to obtain the heterogeneous graph.

[0114] In one implementation, constructing the original heterogeneous graph may include: extracting a set of components from the search log and / or the knowledge graph; setting the set of components as a set of nodes; determining a set of edges between the set of nodes; and combining the set of nodes and the set of edges into the original heterogeneous graph.

[0115] The component set may include at least one of the following: a historical query set, a historical keyword set, and a historical title set extracted from the search log; and / or an entity set corresponding to the historical query set and / or an entity set corresponding to the historical keyword set extracted from the knowledge graph.

[0116] The determination of the edge set may include, for every two nodes in the node set: determining whether there is a relationship between the two components corresponding to the two nodes; and in response to determining that there is a relationship between the two components, determining that there is an edge between the two nodes.

[0117] The relationship may include at least one of the following: query-keyword click relationship, query-title click relationship, query-entity extraction relationship, keyword-entity extraction relationship, and keyword-keyword co-occurrence relationship.

[0118] The original heterogeneous graph may include multiple original partial graphs corresponding to multiple relation types. The generation of the enhanced heterogeneous graph may include: for each original partial graph corresponding to a relation type, generating an enhanced partial graph corresponding to the relation type based on the semantic information and / or topological structure of a subset of nodes in the original partial graph; and combining the multiple enhanced partial graphs corresponding to the multiple relation types into the enhanced heterogeneous graph.

[0119] The generation of the enhanced partial graph may include: generating a semantic partial graph corresponding to the relation type based on the semantic similarity between two nodes in the node subset; generating a propagation partial graph corresponding to the relation type based on the semantic similarity between two nodes of the same type in the node subset and the topology of the original partial graph; and generating the enhanced partial graph corresponding to the relation type based on the original partial graph, the semantic partial graph, and the propagation partial graph.

[0120] The node set of the enhanced heterogeneous graph may include multiple center nodes. Each center node has multiple neighbor node sets based on various meta-path types. The pruning of the enhanced heterogeneous graph includes, for each of the multiple neighbor node sets: identifying a subset of neighbor nodes that are most important to the center node; determining neighbor nodes in the neighbor node sets that are not in the subset of neighbor nodes; and removing edges between the determined neighbor nodes and the center node.

[0121] In one implementation, the inferred query graph structure information may include: inferring the metapath to which the query node corresponding to the query belongs based on the heterogeneous graph; inferring the node set based on the metapath based on the heterogeneous graph; and generating the query graph structure information based on the metapath and the node set.

[0122] In one implementation, generating the query representation may include: obtaining from the query graph structure information the metapath to which the query node corresponding to the query belongs and the neighboring nodes based on the metapath; generating a word chunk sequence corresponding to the metapath based on the query and the neighboring nodes; generating an initial embedding vector sequence corresponding to the metapath based on the word chunk sequence; generating an embedding vector sequence corresponding to the metapath by iteratively performing graph encoding operations based on the initial embedding vector sequence; and generating the query representation based on the embedding vector sequence.

[0123] The generated word block sequence may include: obtaining query text blocks of the query and neighbor text blocks of the neighbor nodes through word segmentation; adding query local word blocks for the query before the query text blocks; adding neighbor local word blocks for the neighbor nodes before the neighbor text blocks; and combining the global word blocks of the metapath, the query local word blocks, the query text blocks, the neighbor local word blocks, and the neighbor text blocks into the word block sequence corresponding to the metapath.

[0124] The graph encoding operation may include: generating an intermediate global embedding vector of the global word block and multiple intermediate local embedding vectors of the multiple local word blocks associated with the meta-path, based on the previous global embedding vectors of the global word block associated with the meta-path and multiple previous local embedding vectors of the multiple local word blocks associated with the meta-path, using a self-attention mechanism; generating a current global embedding vector based on the intermediate global embedding vectors, using a self-attention mechanism; generating a current local embedding vector and a current text embedding vector of the node, based on the current global embedding vector, the intermediate local embedding vectors of the node, and the previous text embedding vectors, using a self-attention mechanism; and combining the current global embedding vector and the multiple current local embedding vectors and multiple current text embedding vectors of the multiple nodes in the meta-path into a current embedding vector sequence corresponding to the meta-path.

[0125] The step of generating the query representation based on the embedding vector sequence may include: extracting the local embedding vector of the query from the embedding vector sequence; and generating the query representation based on the local embedding vector.

[0126] In one implementation, method 1200 may further include: generating training samples for training a lightweight relevance prediction model based on the query, the keywords, and the relevance.

[0127] It should be understood that method 1200 may also include any step / process for relevance prediction based on heterogeneous graph learning according to embodiments of the present disclosure as described above.

[0128] Figure 13 An exemplary apparatus 1300 for relevance prediction based on heterogeneous graph learning according to an embodiment of the present disclosure is shown.

[0129] The apparatus 1300 may include: a query and keyword receiving module 1310 for receiving queries and keywords; a heterogeneous graph generation module 1320 for generating a heterogeneous graph corresponding to the query and the keywords; a query graph structure information inference module 1330 for inferring query graph structure information for the query based on the heterogeneous graph; a query representation generation module 1340 for generating a query representation of the query based on the query graph structure information; a keyword graph structure information inference module 1350 for inferring keyword graph structure information for the keywords based on the heterogeneous graph; a keyword representation generation module 1360 for generating a keyword representation of the keywords based on the keyword graph structure information; and a relevance prediction module 1370 for predicting the relevance between the query and the keywords based on the query representation and the keyword representation. Furthermore, the apparatus 1300 may also include any other modules configured for relevance prediction based on heterogeneous graph learning according to embodiments of the present disclosure as described above.

[0130] Figure 14 An exemplary apparatus 1400 for relevance prediction based on heterogeneous graph learning according to an embodiment of the present disclosure is shown.

[0131] The apparatus 1400 may include: a processor 1410; and a memory 1420 storing computer-executable instructions. When executed, the computer-executable instructions cause the processor 1410 to: receive a query and keywords; generate a heterogeneous graph corresponding to the query and keywords; infer query graph structure information for the query based on the heterogeneous graph; generate a query representation of the query based on the query graph structure information; infer keyword graph structure information for the keywords based on the heterogeneous graph; generate a keyword representation of the keywords based on the keyword graph structure information; and predict the relevance between the query and the keywords based on the query representation and the keyword representation.

[0132] In one implementation, generating the heterogeneous graph may include: constructing an original heterogeneous graph based on search logs and / or a knowledge graph; generating an enhanced heterogeneous graph based on the semantic information and / or topological structure of the node set of the original heterogeneous graph; and pruning the enhanced heterogeneous graph based on the semantic information and / or topological structure of the node set of the enhanced heterogeneous graph to obtain the heterogeneous graph.

[0133] In one implementation, generating the query representation may include: obtaining from the query graph structure information the metapath to which the query node corresponding to the query belongs and the neighboring nodes based on the metapath; generating a word chunk sequence corresponding to the metapath based on the query and the neighboring nodes; generating an initial embedding vector sequence corresponding to the metapath based on the word chunk sequence; generating an embedding vector sequence corresponding to the metapath by iteratively performing graph encoding operations based on the initial embedding vector sequence; and generating the query representation based on the embedding vector sequence.

[0134] It should be understood that the processor 1410 may also perform any other steps / processes of the method for relevance prediction based on heterogeneous graph learning according to the embodiments of the present disclosure as described above.

[0135] Embodiments of this disclosure provide a computer program product for relevance prediction based on heterogeneous graph learning, comprising a computer program executed by a processor to: receive a query and keywords; generate a heterogeneous graph corresponding to the query and keywords; infer query graph structure information for the query based on the heterogeneous graph; generate a query representation of the query based on the query graph structure information; infer keyword graph structure information for the keywords based on the heterogeneous graph; generate a keyword representation of the keywords based on the keyword graph structure information; and predict the relevance between the query and the keywords based on the query representation and the keyword representation. Furthermore, the computer program may also be executed to perform any other steps / processes for implementing the method for relevance prediction based on heterogeneous graph learning according to embodiments of this disclosure as described above.

[0136] Embodiments of this disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform any operation of the method for relevance prediction based on heterogeneous graph learning according to embodiments of this disclosure as described above.

[0137] It should be understood that all operations in the methods described above are merely exemplary, and this disclosure is not limited to any operation in the methods or the order of such operations, but should cover all other equivalent variations under the same or similar concept. Furthermore, unless otherwise specified or clearly indicated from the context as relating to the singular form, the articles “a” and “an” as used in this specification and the appended claims should generally be interpreted as meaning “one” or “one or more”.

[0138] It should also be understood that all modules in the apparatus described above can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0139] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. As an example, the processors, any portions of processors, or any combination of processors given in this disclosure can be implemented using microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic units, discrete hardware circuits, and other suitable processing components configured to perform the various functions described in this disclosure. The functionality of the processors, any portions of processors, or any combination of processors given in this disclosure can be implemented using software executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[0140] Software should be broadly considered to refer to instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, procedures, functions, etc. Software may reside on a computer-readable medium. Computer-readable media may include, for example, memory, which may be, for example, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical disks, smart cards, flash memory devices, random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), registers, or removable disks. Although memory is shown as separate from the processor in several aspects set forth in this disclosure, memory may also reside within the processor, such as in caches or registers.

[0141] The above description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described herein, known or likely to be known by one of ordinary skill in the art, are expressly incorporated herein and covered by the claims.

Claims

1. A method for relevance prediction based on heterogeneous graph learning, comprising: Receive queries and keywords; Generate a heterogeneous graph corresponding to the query and the keywords; Based on the heterogeneous graph, infer the query graph structure information for the query; Based on the query graph structure information, a query representation of the query is generated, wherein generating the query representation of the query includes: The metapath to which the query node corresponding to the query belongs and the neighbor nodes based on the metapath are obtained from the query graph structure information. Based on the query and the neighboring nodes, generate a sequence of word blocks corresponding to the meta-path; Based on the word block sequence, an initial embedding vector sequence corresponding to the meta-path is generated; Based on the initial embedding vector sequence, an embedding vector sequence corresponding to the meta-path is generated by iteratively performing graph coding operations; Extract the local embedding vectors of the query's local word chunks from the embedding vector sequence, wherein the local word chunks convey information within the same meta-path; and The query representation is generated based on the local embedding vector; Based on the heterogeneous graph, infer the keyword graph structure information for the keyword; Based on the keyword graph structure information, a keyword representation of the keyword is generated; and Based on the query representation and the keyword representation, the relevance between the query and the keyword is predicted.

2. The method according to claim 1, wherein, The keywords are keywords for candidate content items, and the candidate content items include at least one of news, videos, movies, books, music, web pages, and product information.

3. The method according to claim 1, wherein, The generation of heterogeneous graphs includes: Construct the original heterogeneous graph based on search logs and / or knowledge graphs; Based on the semantic information and / or topological structure of the node set of the original heterogeneous graph, an enhanced heterogeneous graph is generated; and Based on the semantic information and / or topological structure of the node set of the enhanced heterogeneous graph, the enhanced heterogeneous graph is pruned to obtain the heterogeneous graph.

4. The method according to claim 3, wherein, The construction of the original heterogeneous graph includes: Extract the component set from the search log and / or the knowledge graph; Set the component set as a node set; Determine the set of edges between the node sets; and The set of nodes and the set of edges are combined to form the original heterogeneous graph.

5. The method according to claim 4, wherein, The component set includes: At least one of the historical query set, historical keyword set, and historical title set extracted from the search log; and / or The entity set corresponding to the historical query set and / or the entity set corresponding to the historical keyword set extracted from the knowledge graph.

6. The method according to claim 4, wherein, The determined edge set includes, for every two nodes in the node set: Determine whether a relationship exists between the two components corresponding to the two nodes; as well as In response to determining that a relationship exists between the two components, an edge exists between the two nodes.

7. The method according to claim 6, wherein, The relationships include at least one of the following: query-keyword click relationship, query-title click relationship, query-entity extraction relationship, keyword-entity extraction relationship, and keyword-keyword co-occurrence relationship.

8. The method according to claim 3, wherein, The original heterogeneous graph includes multiple original partial graphs corresponding to multiple relation types, and the generation of the enhanced heterogeneous graph includes: For each original partial graph corresponding to a relation type, an enhanced partial graph corresponding to that relation type is generated based on the semantic information and / or topological structure of a subset of nodes in the original partial graph; and The multiple enhanced partial graphs corresponding to the multiple relation types are combined into the enhanced heterogeneous graph.

9. The method according to claim 8, wherein, The generated enhanced partial graph includes: Based on the semantic similarity between two nodes in the node subset, a semantic partial graph corresponding to the relationship type is generated; Based on the semantic similarity between two nodes of the same type in the node subset and the topological structure of the original partial graph, a propagation partial graph corresponding to the relationship type is generated; and Based on the original partial graph, the semantic partial graph, and the propagation partial graph, an enhanced partial graph corresponding to the relation type is generated.

10. The method according to claim 3, wherein, The node set of the enhanced heterogeneous graph includes multiple central nodes, each central node having multiple neighbor node sets based on various meta-path types, and the pruning of the enhanced heterogeneous graph includes, for each of the multiple neighbor node sets: Identify the subset of neighbor nodes that are most important to the central node from the set of neighbor nodes; Identify the neighbor nodes in the set of neighbor nodes that are not in the subset of neighbor nodes; as well as Remove the edges between the identified neighboring nodes and the central node.

11. The method according to claim 1, wherein, The inference query graph structure information includes: Based on the heterogeneous graph, infer the metapath to which the query node corresponding to the query belongs; Based on the heterogeneous graph, infer the set of nodes based on the meta-path; and Based on the metapath and the set of nodes, the query graph structure information is generated.

12. The method according to claim 1, wherein, The generated word block sequence includes: By performing word segmentation, the query text block of the query and the neighbor text block of the neighbor node are obtained; Add a local query term block for the query before the query text term block. The local query term block is a local term block for the query. Add a local neighbor term for the neighbor node before the neighbor text term, wherein the local neighbor term is a local term for the neighbor node; and The global word block of the metapath, the query local word block, the query text word block, the neighbor local word block, and the neighbor text word block are combined into the word block sequence corresponding to the metapath, and the global word block transmits information across different metapaths.

13. The method according to claim 1, wherein, The graph encoding operation includes: By using a self-attention mechanism, an intermediate global embedding vector of the global word block and multiple intermediate local embedding vectors of the multiple local word blocks are generated based on the previous global embedding vector of the global word block associated with the meta-path and multiple previous local embedding vectors of the multiple local word blocks associated with the meta-path. The current global embedding vector is generated based on the intermediate global embedding vector using a self-attention mechanism. For each node in the metapath, a self-attention mechanism is used to generate the node's current local embedding vector and current text embedding vector based on the current global embedding vector, the node's intermediate local embedding vector, and the previous text embedding vector; and The current global embedding vector, along with multiple current local embedding vectors and multiple current text embedding vectors of multiple nodes in the metapath, are combined to form a current embedding vector sequence corresponding to the metapath.

14. The method according to claim 1, further comprising: Based on the query, the keywords, and the relevance, training samples are generated for training a lightweight relevance prediction model.

15. An apparatus for relevance prediction based on heterogeneous graph learning, comprising: processor; as well as A memory storing computer-executable instructions, which, when executed, cause the processor to: Receive queries and keywords. Generate a heterogeneous graph corresponding to the query and the keywords. Based on the heterogeneous graph, infer the query graph structure information for the query. Based on the query graph structure information, a query representation of the query is generated, wherein generating the query representation of the query includes: From the query graph structure information, obtain the metapath to which the query node corresponding to the query belongs and the neighbor nodes based on the metapath. Based on the query and the neighboring nodes, a sequence of word chunks corresponding to the meta-path is generated. Based on the word block sequence, an initial embedding vector sequence corresponding to the meta-path is generated. Based on the initial embedding vector sequence, an embedding vector sequence corresponding to the meta-path is generated by iteratively performing graph encoding operations. Extract the local embedding vectors of the query's local word chunks from the embedding vector sequence, wherein the local word chunks convey information within the same meta-path, and The query representation is generated based on the local embedding vector. Based on the heterogeneity graph, infer the keyword graph structure information for the keyword. Based on the keyword graph structure information, a keyword representation of the keyword is generated, and Based on the query representation and the keyword representation, the relevance between the query and the keyword is predicted.

16. The apparatus according to claim 15, wherein, The generation of heterogeneous graphs includes: Construct the original heterogeneous graph based on search logs and / or knowledge graphs; Based on the semantic information and / or topological structure of the node set of the original heterogeneous graph, an enhanced heterogeneous graph is generated; and Based on the semantic information and / or topological structure of the node set of the enhanced heterogeneous graph, the enhanced heterogeneous graph is pruned to obtain the heterogeneous graph.

17. A computer program product for relevance prediction based on heterogeneous graph learning, comprising a computer program executed by a processor for: Receive queries and keywords; Generate a heterogeneous graph corresponding to the query and the keywords; Based on the heterogeneous graph, infer the query graph structure information for the query; Based on the query graph structure information, a query representation of the query is generated, wherein, The query representation that generates the query includes: The metapath to which the query node corresponding to the query belongs and the neighbor nodes based on the metapath are obtained from the query graph structure information. Based on the query and the neighboring nodes, generate a sequence of word blocks corresponding to the meta-path; Based on the word block sequence, an initial embedding vector sequence corresponding to the meta-path is generated; Based on the initial embedding vector sequence, an embedding vector sequence corresponding to the meta-path is generated by iteratively performing graph coding operations; Extract the local embedding vectors of the query's local word chunks from the embedding vector sequence, wherein the local word chunks convey information within the same meta-path; and The query representation is generated based on the local embedding vector; Based on the heterogeneous graph, infer the keyword graph structure information for the keyword; Based on the keyword graph structure information, a keyword representation of the keyword is generated; and Based on the query representation and the keyword representation, the relevance between the query and the keyword is predicted.

Images