An evaluation method for large-scale language models based on dynamic graphs
By generating dynamic graphs and designing question-answer pairs, combined with spatiotemporal decoupling prompts, the understanding ability of large language models on dynamic graphs is evaluated. This solves the problem that existing technologies cannot evaluate spatiotemporal understanding ability and improves the model's spatial-temporal understanding ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2023-10-20
- Publication Date
- 2026-05-26
AI Technical Summary
Existing research has failed to effectively assess the spatial and temporal understanding capabilities of large language models on dynamic graphs, and thus cannot improve their understanding capabilities in the spatiotemporal dimensions.
By generating dynamic graphs, including node sets, edge sets, and timestamp information, question-answer pairs are designed and fed into a large language model. Spatiotemporal decoupling prompts are used to guide the model to output predicted answers, and its comprehension ability is evaluated.
It enables the evaluation of the spatiotemporal information understanding ability of large language models on dynamic graphs, and improves the model's understanding ability in the space-time dimension.
Smart Images
Figure CN117609439B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to an evaluation method for large language models based on dynamic graphs. Background Technology
[0002] Large Language Models (LLMs) are a class of deep learning-based artificial intelligence models trained with massive amounts of data and computational resources, capable of understanding and generating natural language. LLMs are widely used in various tasks, leading to increasing attention on their ability to process network data. Therefore, a method for evaluating the data processing capabilities of large language models is urgently needed. Summary of the Invention
[0003] In view of the above problems, this disclosure provides an evaluation method for large language models based on dynamic graphs to overcome or at least partially solve the above problems.
[0004] A first aspect of this disclosure provides an evaluation method for large-scale language models based on dynamic graphs, the method comprising:
[0005] Generate a dynamic graph, which includes a node set, an edge set, and timestamp information, wherein the timestamp information represents the generation time of each edge in the edge set;
[0006] Based on the dynamic graph, a question-answer pair for the dynamic graph is generated, wherein the question in the question-answer pair includes at least: a question about understanding spatiotemporal information;
[0007] The animated graph and the question are input into a large language model to be evaluated to obtain the predicted answer output by the large language model.
[0008] Based on the predicted answers and the answers in the question-answer pairs of the dynamic graph, the evaluation results of the large language model's ability to understand the dynamic graph data are obtained.
[0009] Optionally, the step of inputting the dynamic graph and the question into a large language model to be evaluated, and obtaining the predicted answer output by the large language model, includes:
[0010] Obtain spatiotemporal decoupling prompts;
[0011] The spatiotemporal decoupling prompt, the dynamic graph, and the question are input into the large language model to be evaluated to obtain the predicted answer output by the large language model.
[0012] Optionally, generating the dynamic graph includes:
[0013] Obtain the trained animation generator;
[0014] The generation parameters of the dynamic graph generator are set, including the number of nodes and the generation probability of edges between node pairs. The number of nodes is used to control the size of the generated dynamic graph, and the generation probability is used to control the density of the generated dynamic graph.
[0015] An initial dynamic graph is generated using a dynamic graph generator with the generated parameters set. The initial dynamic graph includes a node set and an edge set.
[0016] Each edge in the initial dynamic graph is assigned a timestamp to obtain the dynamic graph.
[0017] Optionally, assigning timestamps to each edge in the initial dynamic graph to obtain the dynamic graph includes:
[0018] Set a time range and obtain multiple timestamps from the time range;
[0019] Each timestamp obtained from the time range is assigned to a different edge in the initial dynamic graph to obtain the dynamic graph.
[0020] Optionally, the spatiotemporal information understanding problem includes any one or more of the following:
[0021] Do the three selected nodes in the dynamic graph form a path ordered by time?
[0022] What are the target paths in the dynamic graph? The target paths are: paths sorted by time, starting from the selected nodes in the dynamic graph.
[0023] What is the sorting result when the edges of the dynamic graph are sorted by time?
[0024] Optionally, the problem may also include any one or more of the following: the problem of understanding time information and the problem of understanding spatial information.
[0025] Optionally, the time information understanding problem includes any one or more of the following:
[0026] When are the two selected nodes in the dynamic graph linked?
[0027] When are the two selected nodes in the dynamic graph connected?
[0028] When do the three selected nodes in the dynamic graph first form a closed triangle?
[0029] Optionally, the spatial information understanding problem includes any one or more of the following:
[0030] At a selected point in time, which nodes are the neighbors of the selected node in the dynamic graph?
[0031] After the selected time point, which nodes are the neighbors of the selected node in the dynamic graph?
[0032] Does the selected three nodes in the dynamic graph form a closed triangle?
[0033] Optionally, the spatiotemporal decoupling prompt information includes:
[0034] Consider the nodes first, then the time.
[0035] Consider the time frame first, then the key points;
[0036] First select the node, then select the time;
[0037] First select the time, then select the node.
[0038] Optionally, the dynamic graph generator includes: a random graph model, a random block model, and a forest fire model.
[0039] A second aspect of this disclosure provides an evaluation apparatus for a large language model based on dynamic graphs, the apparatus comprising:
[0040] The first generation module is used to generate a dynamic graph, which includes a node set, an edge set, and timestamp information, wherein the timestamp information represents the generation time of each edge in the edge set.
[0041] The second generation module is used to generate question-answer pairs for the dynamic graph based on the dynamic graph, wherein the questions in the question-answer pairs include at least: spatiotemporal information understanding questions;
[0042] The input module is used to input the dynamic graph and the question into a large language model to be evaluated, and obtain the predicted answer output by the large language model.
[0043] The acquisition module is used to obtain the evaluation result of the large language model's understanding ability of the dynamic graph data based on the predicted answer and the answer in the question-answer pair of the dynamic graph.
[0044] Optionally, the input module is specifically used to perform:
[0045] Obtain spatiotemporal decoupling prompts;
[0046] The spatiotemporal decoupling prompt, the dynamic graph, and the question are input into the large language model to be evaluated to obtain the predicted answer output by the large language model.
[0047] Optionally, the first generation module is specifically used to execute:
[0048] Obtain the trained animation generator;
[0049] The generation parameters of the dynamic graph generator are set, including the number of nodes and the generation probability of edges between node pairs. The number of nodes is used to control the size of the generated dynamic graph, and the generation probability is used to control the density of the generated dynamic graph.
[0050] An initial dynamic graph is generated using a dynamic graph generator with the generated parameters set. The initial dynamic graph includes a node set and an edge set.
[0051] Each edge in the initial dynamic graph is assigned a timestamp to obtain the dynamic graph.
[0052] Optionally, assigning timestamps to each edge in the initial dynamic graph to obtain the dynamic graph includes:
[0053] Set a time range and obtain multiple timestamps from the time range;
[0054] Each timestamp obtained from the time range is assigned to a different edge in the initial dynamic graph to obtain the dynamic graph.
[0055] Optionally, the spatiotemporal information understanding problem includes any one or more of the following:
[0056] Do the three selected nodes in the dynamic graph form a path ordered by time?
[0057] What are the target paths in the dynamic graph? The target paths are: paths sorted by time, starting from the selected nodes in the dynamic graph.
[0058] What is the sorting result when the edges of the dynamic graph are sorted by time?
[0059] Optionally, the problem may also include any one or more of the following: the problem of understanding time information and the problem of understanding spatial information.
[0060] Optionally, the time information understanding problem includes any one or more of the following:
[0061] When are the two selected nodes in the dynamic graph linked?
[0062] When are the two selected nodes in the dynamic graph connected?
[0063] When do the three selected nodes in the dynamic graph first form a closed triangle?
[0064] Optionally, the spatial information understanding problem includes any one or more of the following:
[0065] At a selected point in time, which nodes are the neighbors of the selected node in the dynamic graph?
[0066] After the selected time point, which nodes are the neighbors of the selected node in the dynamic graph?
[0067] Does the selected three nodes in the dynamic graph form a closed triangle?
[0068] Optionally, the spatiotemporal decoupling prompt information includes:
[0069] Consider the nodes first, then the time.
[0070] Consider the time frame first, then the key points;
[0071] First select the node, then select the time;
[0072] First select the time, then select the node.
[0073] Optionally, the dynamic graph generator includes: a random graph model, a random block model, and a forest fire model.
[0074] A third aspect of this disclosure provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the evaluation method for a large language model based on dynamic graphs as described in the first aspect.
[0075] A fourth aspect of this disclosure provides a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform an evaluation method for a large language model based on dynamic graphs as described in the first aspect.
[0076] The embodiments disclosed herein have the following advantages:
[0077] In this embodiment, a dynamic graph is generated, comprising a node set, an edge set, and timestamp information, wherein the timestamp information represents the generation time of each edge in the edge set. Based on the dynamic graph, question-answer pairs are generated, wherein the questions in the question-answer pairs include at least a spatiotemporal information understanding question. The dynamic graph and the questions are input into a large-scale language model to be evaluated, and the predicted answers output by the large-scale language model are obtained. Based on the predicted answers and the answers in the question-answer pairs of the dynamic graph, an evaluation result of the large-scale language model's understanding ability of the dynamic graph data is obtained. Thus, the evaluation of the large-scale language model's understanding ability on dynamic graphs is achieved. Attached Figure Description
[0078] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0079] Figure 1 This is a flowchart illustrating the steps of an evaluation method for a large language model based on dynamic graphs, as described in this disclosure.
[0080] Figure 2 This is a dynamic diagram of an embodiment of this disclosure;
[0081] Figure 3 This is a schematic diagram illustrating the problem of understanding time information in an embodiment of this disclosure;
[0082] Figure 4 This is a schematic diagram illustrating the spatial information understanding problem in an embodiment of this disclosure;
[0083] Figure 5 This is a schematic diagram illustrating the problem of understanding spatiotemporal information in an embodiment of this disclosure. Detailed Implementation
[0084] To make the above-mentioned objectives, features and advantages of this disclosure more apparent and understandable, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0085] Research on large-scale language models can be mainly divided into three categories:
[0086] The first category involves applications of large language models (LLMs) on graph data. Recently, there has been a surge of work on using LLMs to solve graph data tasks. Some studies have enabled LLMs to not only perform zero-shot predictions but also generate coherent explanations for their decisions. These explanations can then be used to enhance the classification properties of nodes in text attribute graphs. Some studies have proposed using LLMs as enhancers and as predictors to solve graph-related tasks; the former uses LLMs to enhance Graph Neural Networks (GNNs), while the latter directly uses LLMs for prediction. Some studies have introduced Natural Language Graph, a custom benchmarking framework for evaluating the performance of LLMs on traditional graph-related tasks. Simultaneously, some studies have conducted comprehensive empirical research focusing on using LLMs to handle structural and semantic understanding tasks in graph-based environments. Contributions in this area include a method for fine-tuning LLMs inspired by LLaMA (a large language model developed by Meta), specifically designed for node classification tasks. Some research has enhanced the reasoning capabilities of LLMs in handling structured data sources (such as knowledge graphs and tables) by interfacing with external tools, opening up new avenues for exploration in this field. Other research has provided comprehensive roadmaps for the seamless integration of LLMs with knowledge graphs, offering valuable insights into the potential synergies between language models and structured knowledge. However, these works primarily focus on static graphs, neglecting the temporal nature of real-world network applications of graphs.
[0087] The second category involves applications of large language models (LLMs) in sequence and recommendation tasks. Related research focuses on LLMs in time series prediction and recommendation tasks. Some studies leverage the superior knowledge and reasoning capabilities of LLMs for interpretable financial time series prediction. Others utilize pre-trained LLMs to enhance time series prediction, demonstrating their excellent capabilities as powerful representation learners and effective few-shot learners. Some studies summarize two strategies for using LLMs to accomplish time series tasks: one strategy designs and trains a base large model for time series data, while the other enables pre-trained LLMs to process time series data. Some studies investigate various prompting strategies through input augmentation to enhance the personalized recommendation performance of large language models. However, these works do not consider the role of structure.
[0088] The third category is dynamic graph learning methods. Dynamic graphs are ubiquitous in many practical applications, covering fields such as event prediction and recommender systems. This ubiquity has spurred significant research interest in developing and improving dynamic graph neural networks. These networks are designed to simulate complex graph dynamics, including structures and properties that evolve over time. Various methods have been proposed to address the challenges posed by dynamic graphs. Some research efforts have focused on leveraging graph neural networks to aggregate neighborhood information from each individual snapshot of the graph. Subsequently, these methods use a sequence module to capture and simulate temporal information. Conversely, other studies have proposed using temporal encoding techniques. These methods encode temporal links into specific temporally aware embeddings and then utilize a GNN or memory module to process and manipulate the structural information embedded in the graph. However, these methods require large language models to be trained every time a new task is encountered, limiting their widespread use.
[0089] Overall, existing research has not considered the application of large language models in dynamic graphs, nor has it considered evaluating the spatial and temporal understanding capabilities of large language models, nor has it been able to improve the spatiotemporal understanding capabilities of large language models.
[0090] To address the aforementioned issues, this disclosure proposes, for the first time, a method for evaluating the spatial-temporal understanding capabilities of LLMs on dynamic graphs, thus filling this gap. Specifically, this disclosure proposes a dynamic graph framework and evaluation method based on a large-scale language model, including nine specially designed tasks to examine the applicability of LLMs in both temporal and spatial dimensions. Extensive experiments were then conducted to analyze the impact of different data generators, data statistics, prompting techniques, and LLMs on model performance. Finally, a Disentangled Spatial-Temporal Thoughts (DST2) approach for LLMs on dynamic graphs is proposed to enhance their spatial-temporal understanding capabilities.
[0091] Reference Figure 1 The diagram illustrates a flowchart of the steps involved in evaluating a large language model based on dynamic graphs, as described in an embodiment of this disclosure. Figure 1 As shown, the evaluation method for large language models based on dynamic graphs may specifically include steps S11 to S14.
[0092] Step S11: Generate a dynamic graph, which includes a node set, an edge set, and timestamp information, wherein the timestamp information represents the generation time of each edge in the edge set.
[0093] A graph is a data structure where nodes represent entities and edges represent relationships. An edge connects two nodes, and an edge represents the relationship between the two connected entities. A graph with invariable edges is called a static graph, while a graph with changing edges is called a dynamic graph. For example, a graph where there is no edge between nodes A and B at the first time step, but an edge exists between nodes A and B at the second time step, is a dynamic graph. Dynamic graphs can capture the evolution patterns of networks over time and are ubiquitous in real-world network data.
[0094] Dynamic graphs can be represented using node sets, edge sets, and timestamp information. Each node in the node set is a node in the dynamic graph, each edge in the edge set is an edge in the dynamic graph, and the timestamp information includes the timestamp of each edge, with the timestamp of an edge representing the time when the edge was generated. Figure 2 This is a dynamic diagram from an embodiment of this disclosure. The edge between node A and node B carries a timestamp of 0, indicating that nodes A and B are connected at time 0; the edge between node A and node C carries a timestamp of 1, indicating that nodes A and B are connected at time 1. This can be represented by (A,B,0)(A,C,1). Figure 2 Animated GIF.
[0095] Optionally, step S11 may include: obtaining a trained dynamic graph generator; setting generation parameters of the dynamic graph generator, the generation parameters including: the number of nodes and the generation probability of edges between node pairs, the number of nodes being used to control the size of the generated dynamic graph, and the generation probability being used to control the density of the generated dynamic graph; using the dynamic graph generator with the generation parameters set, generating an initial dynamic graph, the initial dynamic graph including a node set and an edge set; assigning timestamps to each edge in the initial dynamic graph to obtain the dynamic graph.
[0096] Dynamic graph generators can include: random graphs ( One of the following models: Rényi, Stochastic Block, and Forest Fire. The initial dynamic graph is an undirected graph.
[0097] Optionally, when assigning timestamps to each edge in the initial dynamic graph, a time range can be set, and each assigned timestamp can be a timestamp within that time range. In this way, the time span of the generated dynamic graph can be determined by controlling the size of the time range.
[0098] By employing the technical solutions of this disclosure, the ability of a large language model to understand dynamic graphs generated by different dynamic graph generators can be evaluated by selecting different dynamic graph generators; the ability of a large language model to understand dynamic graphs of different sizes and densities can be evaluated by setting the generation parameters of the dynamic graph generator; and the ability of a large language model to understand dynamic graphs with different time spans can be evaluated by setting the time range.
[0099] Step S12: Based on the dynamic graph, generate question-answer pairs for the dynamic graph, wherein the questions in the question-answer pairs include at least: spatiotemporal information understanding questions.
[0100] The question-answer pairs for dynamic graphs include: a question related to the dynamic graph, and the correct answer to the question. The correct answer to each question can be obtained from the parameters of the dynamic graph.
[0101] The problems can include spatiotemporal information understanding problems. Optionally, the problems can also include time information understanding problems and / or spatial information understanding problems. Time information understanding problems refer to problems that query time, spatial information understanding problems refer to problems that query nodes, and spatiotemporal information understanding problems include mixed space-time patterns that require simultaneous queries on both space and time. Several types of problems will be described in detail later.
[0102] Step S13: Input the dynamic graph and the question into the large language model to be evaluated to obtain the predicted answer output by the large language model.
[0103] By inputting the dynamic graph and the questions in the question-answer pairs into the large language model to be evaluated, we can obtain the predicted answers for each question output by the large language model based on the dynamic graph.
[0104] Alternatively, large-scale language models can be models such as GPT-3.5, Vicuna-7B, Llama-2-13B, etc.
[0105] Step S14: Based on the predicted answer and the answer in the question-answer pair of the dynamic graph, obtain the evaluation result of the large language model's ability to understand the dynamic graph data.
[0106] By comparing the predicted answers output by a large language model with the correct answers in the question-answer pairs, we can obtain the accuracy rate of the large language model. Based on this accuracy rate, we can evaluate the large language model's ability to understand dynamic graph data.
[0107] The technical solution of this disclosure embodiment enables the evaluation of the understanding ability of large language models on dynamic graphs.
[0108] The following section discusses issues related to understanding time information, such as... Figure 3 As shown, the problems related to understanding time information include:
[0109] Question 1: When are the two selected nodes in the dynamic graph linked?
[0110] Question 1 asks when two nodes are linked, where linking indicates the existence of an edge between them. When generating Question 1, an edge can be randomly selected from the edge set.
[0111] like Figure 3 As shown in part a, in a dynamic graph, if there is an edge between node 0 and node 1 in the edge set, the connection time of the edge between node 0 and node 1 can be determined based on the timestamp carried by the edge between node 0 and node 1.
[0112] Question 2: When are the two selected nodes in the dynamic graph connected?
[0113] Question 2 asks when two nodes are connected. In a dynamic graph, if there exists a path [(u,k1,t),(k1,k2,t),…,(k...]] from node u to node v in the edge set... i If , v, t), then node u and node v are connected in time t.
[0114] like Figure 3 As shown in part b, node 0 and node 2 are connected at time 0, and node 2 and node 3 are connected at time 2. Therefore, at time 2, node 0 and node 3 are connected for the first time.
[0115] Question 3: When do the three selected nodes in the dynamic graph first form a closed triangle?
[0116] Dynamic triangle closure is crucial for dynamic graph analysis. In a dynamic graph, if two nodes with a common neighbor begin linking at some point, forming a triplet, then these three nodes are said to have a triangle closure. When generating question 3, a closed triplet can be randomly selected as the query.
[0117] like Figure 3 As shown in section c, nodes 0, 1, and 2 first form a closed triangle at time 2.
[0118] It is worth noting that while temporal information understanding problems focus on queries about time, they also require large language models to understand the structure of dynamic graphs so that they can identify when structural patterns such as links, paths, and dynamic triples exist.
[0119] The following section introduces spatial information understanding problems, such as... Figure 4 As shown, spatial information understanding problems include:
[0120] Question 4: At a selected time point, which nodes are the neighbors of the selected node in the dynamic graph?
[0121] Question 4 queries which nodes are linked to a selected node at a given time. A node has edges with its neighbors. When generating Question 4, a time and a node that is not isolated in a time-related graph snapshot can be randomly selected to construct the query.
[0122] like Figure 4 As shown in part a, we can query which nodes are the neighbors of node 1 at the selected time 1. The answer is: node 0 and node 2.
[0123] Question 5: After the selected time point, which nodes are the neighbors of the selected node in the dynamic graph?
[0124] Question 5 asks which nodes are linked to a selected node at or after a selected time point, but not linked to it before the selected time point. When generating Question 5, a time point and nodes that were not orphaned before the selected time point can be randomly selected to construct the query. Question 5 can measure the ability of a large language model to understand structure over a time period, such as recent links.
[0125] like Figure 4 As shown in section b, we can query which nodes are the neighbors of node 2 at or after time 1. Answer: Node 1 and node 3.
[0126] Question 6: Do the three selected nodes in the animated diagram form a closed triangle?
[0127] Question 6 is a yes / no question, asking whether three selected nodes in a dynamic graph form a closed triangle. When generating Question 6, samples can be uniformly drawn from both the closed triplet set and the open triplet set to construct positive and negative samples respectively. Maintaining a balance between the number of positive and negative samples in the dataset ensures accurate evaluation.
[0128] like Figure 4 As shown in section c, we can ask whether nodes 0, 1, and 2 form a closed triangle. Answer: Yes.
[0129] Spatial information understanding problems require large language models to determine time points, from specific time points and time periods to the entire time range, and then identify the {t what} structural pattern, or {t whether} the selected structural pattern meets the query requirements.
[0130] The following section introduces the problem of understanding spatiotemporal information, such as... Figure 5 As shown, the problems of understanding spatiotemporal information include:
[0131] Question 7: Do the three selected nodes in the dynamic graph form a path ordered by time?
[0132] Question 7 is essentially a time path checking problem. Question 7 asks whether, in a dynamic graph, the selected three ordered nodes form a time-ordered path. In a dynamic graph, if the node sequence [v1, v2, ..., v...]... n The timestamp of the edge, along the path from source node v1 to target node v. n If the number of nodes does not decrease, then this sequence of nodes constitutes a path ordered by time. When generating problem 7, a balanced dataset can be constructed by randomly selecting positive and negative samples from the sets of paths ordered by time and paths not ordered by time.
[0133] like Figure 5 As shown in part a, we can query whether nodes 0, 1, and 2 form a path ordered by time. Answer: Yes.
[0134] Question 8: What are the target paths in the dynamic graph? The target paths are: paths sorted by time, with the selected nodes in the dynamic graph as the starting nodes.
[0135] Problem 8 is essentially a time-ordered path finding problem. It requires a large language model to find a time-ordered target path in a dynamic graph, starting from a selected node. A node is randomly selected, and this node becomes a starting node on any time-ordered path, to construct the query. Note that any valid time-ordered path starting from the selected node is a correct answer.
[0136] like Figure 5 As shown in section b, starting from node 1, searching for the path sorted by time yields a path consisting of nodes 1, 2, and 3.
[0137] Question 9: What is the sorting result when the edges in the dynamic graph are sorted by time?
[0138] Problem 9 is essentially a time-ordered edge problem. In this problem, the edges are shuffled, and a large language model is required to sort them from earliest to latest time. When some edges have the same timestamp, the internal order of these edges does not affect the correct answer.
[0139] like Figure 5 As shown in section c, there are a total of 4 edges, so the edge between node 1 and node 2 should be the earliest edge.
[0140] The problem of understanding spatiotemporal information requires large language models to understand the spatial-temporal information on dynamic graphs, whether at local or global scales, and the query target includes the spatiotemporal information on the dynamic graph.
[0141] The aforementioned problems of temporal information understanding, spatial information understanding, and spatiotemporal information understanding are designed to simultaneously consider both temporal and spatial dimensions, and use natural language to query large language models {when, what, or whether} spatial-temporal patterns occur. The range of spatial-temporal patterns extends from temporal links and time-ordered paths to dynamic closed triangles. Problems are categorized based on the query objective.
[0142] In this embodiment of the disclosure, the stem of the input question includes a description of the selected node and time point, etc.
[0143] This disclosure also proposes a spatiotemporal decoupling thought process, handling spatial and temporal information separately, for example, processing time first and then structure, or processing structure first and then time. This approach decomposes complex temporal and spatial information into two separate dimensions, thereby reducing the complexity. This disclosure guides large language models to consider nodes or time sequentially by inputting spatiotemporal decoupling prompts.
[0144] Based on the above technical solutions, the embodiments of this disclosure design a variety of spatiotemporal decoupling prompts. When the dynamic graph and the question are input into the large language model to be evaluated, the spatiotemporal decoupling prompts are also input into the large language model to be evaluated, thereby guiding the large language model and obtaining the predicted answer output by the large language model.
[0145] The spatiotemporal decoupling prompts include: consider nodes first, then time; consider time first, then nodes; select nodes first, then time; select time first, then nodes.
[0146] Table 1. Model performance for dynamic graph tasks using one-time hints (V0) and DST2 hints (V1 to V4)
[0147]
[0148] Among them, V0 represents the one-time prompting method in related technologies; V1 represents considering nodes first, then time; V2 represents considering time first, then nodes; V3 represents selecting nodes first, then time; and V4 represents selecting time first, then nodes. In Table 1, the best results for each problem are indicated in bold, and the second-best results are indicated in underline.
[0149] The technical solutions proposed in this disclosure provide the first evaluation of the spatiotemporal information understanding capabilities of large-scale language models on dynamic graphs. The model benchmarks proposed in this disclosure aim to comprehensively evaluate the performance of LLMs on dynamic graphs. The model benchmarks include nine natural language dynamic graph tasks considering both temporal and spatial dimensions, covering topics from temporal linking and time-ordered paths to dynamic triangular closure, and addressing {when, what, or whether} questions for LLMs. Extensive experiments were conducted, considering three data generators, three graph parameters, four general cues, and various different LLMs. Based on these experiments, fine-grained analysis and observations are provided for the evaluation of LLMs on dynamic graphs. This disclosure proposes a spatiotemporally decoupled thought chain cuing technique. Experimental results show that it can significantly improve the spatial-temporal reasoning capabilities of LLMs.
[0150] The main observations of this disclosure are: 1) LLMs have a preliminary spatial-temporal understanding of dynamic graphs; 2) As the size and density of the graph increase, the difficulty of dynamic graph tasks for LLMs also increases, while they are not sensitive to time span and data generation mechanism; 3) The DST2 hinting method proposed in this disclosure can help improve the spatial-temporal understanding of dynamic graphs for most tasks.
[0151] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this disclosure.
[0152] This disclosure provides an evaluation device for a large language model based on dynamic graphs. The device includes a first generation module, a second generation module, an input module, and an acquisition module, wherein:
[0153] The first generation module is used to generate a dynamic graph, which includes a node set, an edge set, and timestamp information, wherein the timestamp information represents the generation time of each edge in the edge set.
[0154] The second generation module is used to generate question-answer pairs for the dynamic graph based on the dynamic graph, wherein the questions in the question-answer pairs include at least: spatiotemporal information understanding questions;
[0155] The input module is used to input the dynamic graph and the question into a large language model to be evaluated, and obtain the predicted answer output by the large language model.
[0156] The acquisition module is used to obtain the evaluation result of the large language model's understanding ability of the dynamic graph data based on the predicted answer and the answer in the question-answer pair of the dynamic graph.
[0157] Optionally, the input module is specifically used to perform:
[0158] Obtain spatiotemporal decoupling prompts;
[0159] The spatiotemporal decoupling prompt, the dynamic graph, and the question are input into the large language model to be evaluated to obtain the predicted answer output by the large language model.
[0160] Optionally, the first generation module is specifically used to execute:
[0161] Obtain the trained animation generator;
[0162] The generation parameters of the dynamic graph generator are set, including the number of nodes and the generation probability of edges between node pairs. The number of nodes is used to control the size of the generated dynamic graph, and the generation probability is used to control the density of the generated dynamic graph.
[0163] An initial dynamic graph is generated using a dynamic graph generator with the generated parameters set. The initial dynamic graph includes a node set and an edge set.
[0164] Each edge in the initial dynamic graph is assigned a timestamp to obtain the dynamic graph.
[0165] Optionally, assigning timestamps to each edge in the initial dynamic graph to obtain the dynamic graph includes:
[0166] Set a time range and obtain multiple timestamps from the time range;
[0167] Each timestamp obtained from the time range is assigned to a different edge in the initial dynamic graph to obtain the dynamic graph.
[0168] Optionally, the spatiotemporal information understanding problem includes any one or more of the following:
[0169] Do the three selected nodes in the dynamic graph form a path ordered by time?
[0170] What are the target paths in the dynamic graph? The target paths are: paths sorted by time, starting from the selected nodes in the dynamic graph.
[0171] What is the sorting result when the edges of the dynamic graph are sorted by time?
[0172] Optionally, the problem may also include any one or more of the following: the problem of understanding time information and the problem of understanding spatial information.
[0173] Optionally, the time information understanding problem includes any one or more of the following:
[0174] When are the two selected nodes in the dynamic graph linked?
[0175] When are the two selected nodes in the dynamic graph connected?
[0176] When do the three selected nodes in the dynamic graph first form a closed triangle?
[0177] Optionally, the spatial information understanding problem includes any one or more of the following:
[0178] At a selected point in time, which nodes are the neighbors of the selected node in the dynamic graph?
[0179] After the selected time point, which nodes are the neighbors of the selected node in the dynamic graph?
[0180] Does the selected three nodes in the dynamic graph form a closed triangle?
[0181] Optionally, the spatiotemporal decoupling prompt information includes:
[0182] Consider the nodes first, then the time.
[0183] Consider the time frame first, then the key points;
[0184] First select the node, then select the time;
[0185] First select the time, then select the node.
[0186] Optionally, the dynamic graph generator includes: a random graph model, a random block model, and a forest fire model.
[0187] It should be noted that the device embodiments are similar to the method embodiments, so the description is relatively simple. For relevant details, please refer to the method embodiments.
[0188] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0189] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0190] This disclosure describes embodiments of methods, apparatus, electronic devices, and computer program products according to embodiments of this disclosure with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0193] While preferred embodiments of the present disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.
[0194] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.
[0195] The above provides a detailed description of an evaluation method for a large language model based on dynamic graphs. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this disclosure. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.
Claims
1. An evaluation method for large-scale language models based on dynamic graphs, characterized in that, The method includes: Generate a dynamic graph, which includes a node set, an edge set, and timestamp information, wherein the timestamp information represents the generation time of each edge in the edge set; Based on the dynamic graph, a question-answer pair for the dynamic graph is generated, wherein the question in the question-answer pair includes at least: a question about understanding spatiotemporal information; The process involves inputting the dynamic graph and the question into a large language model to be evaluated, and obtaining the predicted answer output by the large language model. This includes: obtaining spatiotemporal decoupling prompt information; inputting the spatiotemporal decoupling prompt information, the dynamic graph, and the question into the large language model to be evaluated, and obtaining the predicted answer output by the large language model. Based on the predicted answers and the answers in the question-answer pairs of the dynamic graph, the evaluation results of the large language model's ability to understand dynamic graph data are obtained; The spatiotemporal information understanding problem includes any one or more of the following: Do the three selected nodes in the dynamic graph form a path ordered by time? What are the target paths in the dynamic graph? The target paths are: paths sorted by time, starting from the selected nodes in the dynamic graph. What is the sorting result when the edges of the dynamic graph are sorted by time? Problems of understanding time information and problems of understanding spatial information; The time information understanding problem includes any one or more of the following: When are the two selected nodes in the dynamic graph linked? When are the two selected nodes in the dynamic graph connected? When do the three selected nodes in the dynamic graph first form a closed triangle? The spatial information understanding problem includes any one or more of the following: At a selected point in time, which nodes are the neighbors of the selected node in the dynamic graph? After the selected time point, which nodes are the neighbors of the selected node in the dynamic graph? Does the selected three nodes in the dynamic graph form a closed triangle? The spatiotemporal decoupling prompt information includes: Consider the nodes first, then the time. Consider the time frame first, then the key points; First select the node, then select the time; First select the time, then select the node.
2. The method according to claim 1, characterized in that, The generation of the dynamic graph includes: Obtain the trained animation generator; The generation parameters of the dynamic graph generator are set, including the number of nodes and the generation probability of edges between node pairs. The number of nodes is used to control the size of the generated dynamic graph, and the generation probability is used to control the density of the generated dynamic graph. An initial dynamic graph is generated using a dynamic graph generator with the generated parameters set. The initial dynamic graph includes a node set and an edge set. Each edge in the initial dynamic graph is assigned a timestamp to obtain the dynamic graph.
3. The method according to claim 2, characterized in that, The step of assigning timestamps to each edge in the initial dynamic graph to obtain the dynamic graph includes: Set a time range and obtain multiple timestamps from the time range; Each timestamp obtained from the time range is assigned to a different edge in the initial dynamic graph to obtain the dynamic graph.
4. The method according to claim 2, characterized in that, The dynamic graph generator includes: a random graph model, a random block model, and a forest fire model.