A Graph Visualization Automatic Question Answering Method Based on Knowledge Graph

Abstract

The present invention discloses a method for automatic question answering of graph visualization based on a knowledge graph. It is characterized by using a knowledge base extension module to perform semantic and graph attribute extension on the knowledge graph, extracting topic entities in the question by using shallow semantic analysis and named entity recognition, outputting the knowledge path from the topic entity to the answer entity based on a reinforcement learning and BERT-based automatic question answering model, and outputting text answers and visual answers according to different answer types. Compared with the prior art, the present invention solves the problem of automatic question answering for this complex visualization type of graph visualization, constructs a knowledge graph through the data characteristics of graph visualization, and uses a method based on reinforcement learning and BERT to improve the robustness and generalization ability of question answering. This method performs automatic question answering on network graph visualization charts in various practical application scenarios such as literary character relationship research, intelligent education, commodity analysis, and scientific research hotspot analysis, and has high practical value and good development prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of graph visualization knowledge graphs in natural language processing, and in particular to a graph visualization automatic question answering based on a knowledge graph. Background Art

[0002] When users face decision-making tasks, graph visualization is a common visualization type for analyzing topological data and answering questions. However, when users face many complex analysis questions about graph visualization, this is not an easy task. First, users need to understand the questions before observing the charts. When observing graph visualization, users also need to understand the legend and text information, perceive the graphic data, analyze communities, calculate the attributes of various graph data, etc. according to the different questions.

[0003] Currently, there are already some natural language systems for automatic chart question answering to help users analyze charts and answer questions faster. However, most of the existing work focuses on simple visualization charts, such as scatter plots, line charts, and bar charts. These charts can be easily converted into tabular data and are suitable for tabular question answering systems. Graph visualization is usually used to represent the topological relationship between nodes, such as the character relationship in a novel. People usually analyze data and answer questions through charts. In practical applications, the questions raised by users about graph visualization not only involve the topological relationship of nodes, but also involve the attributes unique to graph visualization, such as node degree, graph community, etc. Because graph visualization contains more attributes than simple visualization and graph data is not a two-dimensional structure, it is difficult to convert graph visualization into tabular data and apply the existing chart automatic question answering methods. In addition, most of the existing automatic question answering systems use text information to answer, which is very boring and deviates from the original intention of visualizing data with visualization. Visual answers combined with text answers will make the answers more useful, persuasive, and interpretable. Summary of the Invention

[0004] The object of the present invention is to provide a graph visualization automatic question answering method based on a knowledge graph in view of the deficiencies of the prior art. The method uses a knowledge graph to store the information data of graph visualization, expands the semantics and graph attributes of the knowledge graph through a knowledge base expansion module, extracts topic entities in the question by using shallow semantic analysis and named entity recognition, uses an automatic question answering model based on reinforcement learning and BERT to output the knowledge path from the topic entity to the answer entity, and finally outputs text answers and visual answers according to different answer types, effectively solving the automatic question answering problem of this complex visualization type of graph visualization. By constructing a knowledge graph based on the data characteristics of graph visualization and using a method based on reinforcement learning and BERT to improve the robustness and generalization ability of question answering, it can perform automatic question answering on network graph visualization charts in various practical application scenarios such as literary character relationship research, intelligent education, commodity goods analysis, and scientific research hotspot analysis. The method is simple, has a good use effect, has high practical value and good development prospects.

[0005] The specific technical solution for realizing the object of the present invention is: a graph visualization automatic question answering method based on a knowledge graph, which is characterized in that it uses a framework GVQA (Graph Visualization Question Answering) to answer questions about graph visualization and automatically generate visual answers. First, the graph visualization is converted into the standard GML format, then the graphic data and visual attributes are extracted from the GML. According to the characteristics of the graph, the data structure of the graph is designed and converted into a knowledge graph. The method also designs an expansion module for the knowledge base to expand semantic information and graph analysis data, uses shallow semantic analysis and named entity recognition to extract topic entities in the question, uses an automatic question answering model based on reinforcement learning and BERT to output the knowledge path from the topic entity to the answer entity, and finally outputs text answers and visual answers according to different answer types. The specific process of the present invention includes the following steps:

[0006] Step a: Input a graph visualization G and a question Q, and convert G into the GML standard format.

[0007] Step b: Convert the native data in the graph visualization into entities and native triples and store them in a knowledge graph where h represents the head entity, r represents the relationship, t represents the tail entity, E represents the entity set, and R represents the relationship set. The native data includes node information, edge information, weights, etc. originally contained in the graph visualization.

[0008] Step c: Expand Perform knowledge graph expansion, and store the derived triples obtained after expansion into Among them, the knowledge graph extension includes semantic information extension and graph attribute extension. The derived triples are calculated from the data of the native triples or other derived triples. The constructed extended knowledge graph includes graph topology information, graph attribute information, and graph semantic information for graph visualization.

[0009] The graph topology information takes each node and edge in the network graph as entities, takes the connection relationship of the edges and the neighbor relationship of the nodes as relationship predicates, constructs several triples, and at the same time takes the name of the entity as a metadata triple and adds it to the annotation graph. Introduce a graph entity to represent the entire graph, and add triples indicating that the edges and nodes belong to the graph; The graph attribute information is common analysis indicators in network graph analysis, including the degree of nodes (in-degree and out-degree), degree centrality of nodes, aggregation degree of nodes, weight of edges, community information, etc.; The community information is obtained by using the Louvain algorithm to detect the communities of the graph. An entity is constructed for each community, and attributes such as the average degree, average edge weight, and average central aggregation degree of each community are added. Information indicating that the community belongs to the graph and information indicating that the nodes and edges belong to the community are also added to the knowledge graph; The graph semantic information uses the predicate alias mechanism to replace r in the previously added information triples with r with semantic information s And add it to the knowledge graph again. A single relationship can be added repeatedly. Relationships that can have predicate aliases include: degree of nodes, degree centrality of nodes, aggregation degree of nodes, weight of edges, community membership, average weight of the community, average degree of the community, average aggregation degree of the community, connection relationship of edges, neighbor relationship of nodes, etc.

[0010] Step d: Extract the topic entities in the input question Q in step a through the topic entity extraction module, and output the candidate set of topic entities which contains a set {<t 1 , c 1 , <t 2 , c 2 >,...}, where t i is the entity name, c i is the credibility score. The value range of the credibility is [0, 1]. The higher the score, the more credible the prediction result of the entity.

[0011] The topic entity extraction module includes named entity recognition, multi-token entity recognition, edge entity matching, and graph entity addition, which are specifically as follows:

[0012] 1) Named entity recognition: Tokenize the input question, use the NER model in Flair to perform named entity recognition on each token in turn, use the SequenceTagger to obtain the POS information of the sentence, and match the tokens with the POS tag of noun among all entity names in ; If in the knowledge graph If there is a corresponding match, add it to the knowledge graph to obtain where is the score output by the NER model.

[0013] 2) Multi-token entity recognition: Use the N-Gram method to sequentially match entities whose names contain spaces, that is, entities corresponding to multiple tokens. Here, n ranges from 1 to 4 and is matched sequentially. If the match is successful, then add <t NGram , c = max(c n ) > (c n ∈ t) to the candidate set .

[0014] 3) Edge entity recognition: If multiple node entities are recognized, detect whether the tokens between these two entities are conjunctions. If so, detect whether there is a connection relationship between these two entities in the graph; if there is, then add the entity of this edge to the candidate set . Let the confidence scores c of the two entities be c a and c b respectively. Then add the entity <t edge , c = max(c a , c b ) >.

[0015] 4) Graph entity addition: Considering that all problems are related to the graph, if after the above steps then add the graph entity <t graph , c = 1> to the candidate set ; if at this time then add <t graph , c = 0.4> to the candidate set .

[0016] Step e: Construct a candidate query graph for each topic entity in the candidate set . The query graph contains a core relationship path starting from the topic entity and ending at the answer entity. The query graph is defined as QG = {N, E}, where N is the node set and there are four types of nodes: n i ∈ {n g , n ug , n ag , n an}, n g is the grounded node, indicating an entity existing in the knowledge graph; n ug is the ungrounded node, which can be used to represent multiple entities or an intermediate query result, n ag is the aggregation node, which can be used to perform aggregation operations, n anis the answer node, representing the query answer; E represents the edge set {e 1 , e 2 ,...}, where e i is the relationship r in the knowledge graph triple.

[0017] Step f: Iteratively expand the query graph. Let be the set of query graphs at the t-th iteration. When t = 0, In each iteration, for all attempt to attach a feasible relationship or a keyword-based aggregation operation to the tail of the query graph; the feasible relationship refers to a relationship existing in an , and the entity of this relationship is the tail node of the query graph; if there is only one topic entity in the current G, after attaching the feasible relationship, the tail is designated as n an ; if there is already one, change the original n ug to n an , and after attaching the relationship, use the new n

[0018] Step g: Calculate the feature vector of the query graph. Adopt a candidate query graph ranking model based on reinforcement learning. In each iteration, use beam search to retain the 3 query graphs with the highest scores and repeat step f until the query graph cannot be expanded. Finally, output the query graph with the highest score as the optimal query graph.

[0019] The candidate query graph ranking model based on reinforcement learning generates a 6-dimensional feature vector for each candidate query graph Specifically, it includes the following:

[0020] 1) BERT-based semantic matching: Use the standard BERT model to measure the semantic similarity between the tokenized problem sequence s q and the query graph sequence s g . s g is generated by sequentially connecting the grounded entity names and relationship names along the core relationship path. Provide the sequence [CLS]s q [SEP]s g to the BERT model to calculate their semantic similarity;

[0021] 2) Entity confidence: The cumulative confidence score c of all topic entities;

[0022] 3) Number of entities: The number of n g in the query graph;

[0023] 4) Number of entity types: The number of entity types;

[0024] 5) Number of answer entities: The number of n in the query graphan Quantity;

[0025] 6) Aggregation quantity: Query the quantity of n ag in the graph.

[0026] Feed the feature vector v of each candidate query graph q into the fully connected layer of the reinforcement model to obtain p(q|Q); the training objective is to learn the policy function p θ (q|Q), where θ represents the parameters in the model, and use the F1 score between the predicted answer and the correct answer label as the reward.

[0027] h step: Generate the SPARQL statement S from the head node to the tail node according to the optimal query graph, and execute this query statement on the knowledge graph to obtain the literal answer A t , A t can be divided into literal answers, entity answers, and statistical answers; according to different answer types, highlight the relevant elements in the graph visualization, and draw auxiliary information cards including statistical bar charts, relevant communities, relevant nodes, and relevant edges to generate the visual answer A v .

[0028] i step: Output the literal answer A t and the visual answer A v , completing the automatic question answering for graph visualization.

[0029] Compared with the prior art, the present invention solves the automatic question answering problem for graph visualization, a complex visualization type, constructs a knowledge graph through the data features of graph visualization, and uses a method based on reinforcement learning and BERT to improve the robustness and generalization ability of question answering. It better solves the automatic question answering problem for network graphs, a high-level visual encoding, especially stores the information data of graph visualization using a knowledge graph according to the data characteristics of graph visualization. This method can perform automatic question answering on network graph visualizations in various practical application scenarios such as literary character relationship research, intelligent education, commodity goods analysis, and scientific research hotspot analysis, and has high practical value and good development prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic flowchart of the present invention;

[0031] Figure 2 is a schematic diagram of graph visualization in Embodiment 1;

[0032] Figure 3 is a schematic diagram of the graph visualization conversion GML in Embodiment 1;

[0033] Figure 4 is a schematic diagram of the knowledge graph constructed in Embodiment 1;

[0034] Figure 5Schematic diagram for extracting candidate sets of topic entities in Embodiment 1. Detailed implementation manners

[0035] Refer to Figure 1 , the data extraction and automatic question answering of the network visualization graph of the present invention are carried out according to the following steps, specifically including the following steps:

[0036] Step 1: Input a graph visualization G and a question Q.

[0037] Step 2: Convert G into the GML standard format and construct a knowledge graph

[0038] Step 3: Generate derived triples to expand the knowledge graph

[0039] Step 4: Extract topic entities from Q to obtain a candidate set of topic entities

[0040] Step 5: Input and into the automatic question answering module based on BERT and reinforcement learning to output a text answer A t ;

[0041] Step 6: Generate a visual answer A s , and output it together with the text answer to complete the graph visualization automatic question answering.

[0042] The following further details the present invention by taking the question answering of a graph visualization of the relationship between characters in a novel as an example.

[0043] Embodiment 1

[0044] Refer to Figure 2 , Step 1: Input a graph visualization G of the characters in a novel and a set of questions Q.

[0045] Refer to Figure 3 , Step 2: Convert the graph visualization G into the GML standard format and construct a knowledge graph

[0046] Refer to Figure 4 , add the native triples containing graph topology data to the knowledge graph in (i.e., Figure 4 b in), and add the entity and subordinate triples of the overall graph (i.e., Figure 4 e in).

[0047] Step 3: Add the derived triples after semantic extension (i.e., Figure 4 a and d in) and graph attribute analysis extension (i.e., Figure 4 e in) to the knowledge graph in.

[0048] See Figure 5 , Step 4: Extract the topic entity candidate set for each question in the input Q For example, the topic entity candidate set for the question "What is the weight of the edge between Myriel and Napoleon?" is {<"m.node@myriel":1>, <"m.node@napoleo":1>, <"m.edge@myranap":1>, <"m.grap@2y96ai":0.4>}.

[0049] Step 5: Input the constructed knowledge graph and the topic entity candidate set into the automatic question answering module, and output the best query graph and the corresponding SPARQL statement SELECT?e1 WHERE { m.edge@myranap edge.property.weight?e1}, and obtain the literal answer "1" according to the query statement.

[0050] Step 6: The literal answer "1" is a literal answer, and the relevant graph visualization element is the edge connecting the nodes Myriel and Napolean. Therefore, the visual answer generation module will highlight this edge connecting the two nodes in the graph visualization and output an information card of their relevant properties as auxiliary information.

[0051] Step 7: Output the literal answer and the visual answer together to complete the graph visualization automatic question answering.

[0052] The above is only a further description of the present invention and is not intended to limit this patent. Equivalent implementations without departing from the spirit and scope of the present invention concept should be included within the scope of the claims of this patent.

Claims

1. A graph visualization automatic question answering method based on a knowledge graph, characterized in that, the graph visualization is converted into the standard GML format, and then the graph data and visual attributes are extracted from the GML. According to the characteristics of the graph, the data structure of the graph is designed and converted into a knowledge graph for graph visualization automatic question answering. The data extraction and automatic question answering of the network visualization graph specifically include the following steps: Step a: Input a graph visualization G and a question Q, and convert G into the GML standard format; Step b: Convert the native data in the graph visualization into entities and native triples and store them in a knowledge graph Where h represents the head entity, r represents the relationship, t represents the tail entity, E represents the entity set, R represents the relationship set, and the native data includes node information, edge information, and weights originally included in the graph visualization; Step c: Expand the knowledge graph and store the obtained derived triples into the knowledge graph . The expansion of the knowledge graph includes semantic information expansion and graph attribute expansion, and the derived triples are calculated from the data of native triples or other derived triples; Step d: Extract the topic entities in the input question Q in step a through the topic entity extraction module, and output the candidate set of topic entities contains a set {<t 1 , c 1 >, <t 2 , c 2 >,...}, where t i is the entity name, and c i is the credibility score, and its value range is [0, 1]. The higher the score, the more credible the entity prediction result is; Step e: For the candidate set Construct a candidate query graph for each topic entity in, the query graph contains a core relationship path starting from the topic entity and ending at the answer entity, and the query graph is defined as QG = {N, E}, where N is the set of nodes, and there are four types of nodes: n i ∈ {n g ,n ug ,n ag ,n an}, n g is the grounded node representing the entity existing in the knowledge graph; n ug is the ungrounded node representing multiple entities or an intermediate query result; n ag is the aggregation node for performing the aggregation operation; n an is the answer node representing the query answer; for the edge set {e, e 2 ,...}, where e i is the relationship in the knowledge graph triple; Step f: Iteratively expand the query graph, and let the parameter be the set of query graphs in the t-th iteration. When t = 0, In each iteration, for all try to attach a feasible relationship or a keyword-based aggregation operation to the tail of the query graph; the feasible relationship refers to a relationship existing in the knowledge graph and the entities of this relationship are the tail nodes of the query graph; if there is only one topic entity in the current visualization G, after attaching the feasible relationship, specify the tail as n an ; if there is already one, change the original n an to n ug , and after attaching the relationship, use the new n an as the new tail node; Step g: Calculate the feature vector of the query graph, and use a candidate query graph ranking model based on reinforcement learning to rank the query graph. In each round of iteration, beam search is used to retain the top 3 query graphs with the highest scores and repeat Step f until the query graph cannot be expanded. Finally, the query graph with the highest score is output as the optimal query graph; Step h: Generate the SPARQL statement S from the head node to the tail node according to the optimal query graph, and execute the query statement on the knowledge graph to obtain the literal answer A t , the literal answer A t is a literal answer, an entity answer, and a statistical answer. According to different answer types, highlight the relevant elements in the visualization of the graph, and draw auxiliary information cards including statistical bar charts, relevant communities, relevant nodes, and relevant edges to generate the visual answer A v ; Step i: Output the text answer A t and the visual answer A v , completing the visual automatic question and answer for the figure.

2. The graph visualization automatic question answering method based on a knowledge graph according to claim 1, characterized in that, the graph visualization includes: diverse network graphs of vector graphs or bitmap graphs drawn by open-source visualization frameworks such as D3, ECharts, Matplotlib, and Scipy.

3. The graph visualization automatic question answering method based on a knowledge graph according to claim 1, characterized in that, the knowledge graph expansion includes the graph topology information, graph attribute information, and graph semantic information of the graph visualization; the graph topology information takes each node and edge in the network graph as entities, takes the connection relationship of the edges and the neighbor relationship of the nodes as relation predicates, constructs a number of triples, and at the same time adds the name of the entity as a metadata triple into the annotation graph. Introduce a graph entity to represent the whole graph, and add triples indicating that the edges and nodes belong to the graph; The graph attribute information includes: the degree of a node, the degree centrality of a node, the aggregation degree of a node, the weight of an edge, and community information; after using the Louvain algorithm to detect the communities of the graph for the community information, an entity is constructed for each community, and the average degree, average edge weight, and average central aggregation degree attributes of each community are added; for the graph semantic information, the r in the triple with added information is replaced with an r with semantic information using the predicate alias mechanism. s It is added into the knowledge graph again, and a single relationship is added repeatedly multiple times; the relationships of the predicate aliases include: the degree of a node, the degree centrality of a node, the aggregation degree of a node, the weight of an edge, community membership, community average weight, community average degree, community average aggregation degree, edge connection relationship, and neighbor relationship of a node.

4. The graph visualization automatic question answering method based on a knowledge graph according to claim 1, characterized in that, The topic entity extraction module includes named entity recognition, multi-token entity recognition, edge entity matching, and graph entity addition. The named entity recognition tokenizes the input question, uses the NER model in Flair to perform named entity recognition on each token in sequence, obtains the POS information of the statement using the SequenceTagger, and matches the tokens with the POS tag of noun among all entity names in the knowledge graph ; if there is a corresponding match in the knowledge graph , add it to the candidate set to obtain where t NER is the entity output by the NER model is the score output by the NER model; The multi-token entity recognition uses the N-Gram method to sequentially match entities whose names contain spaces, that is, entities corresponding to multiple tokens. Take n = 1 to 4 and perform sequential matching. If the match is successful, then <t NGram , c = max(c n ) > (c n ∈ t) is added to the candidate set , where t NGram is the entity recognized by N-Gram; c n is the confidence score; The edge entity recognition is to identify multiple node entities, and then detect whether the token between these two entities is a conjunction. If so, it is further detected whether there is a connection relationship between these two entities in the graph. If there is, the edge entity is added to the candidate set ; Let the confidence scores c of the two entities be c a and c b , then add the graph entity <t edge , c = max(c a , c b )>; The addition of the graph entity is after the graph entity goes through named entity recognition and edge entity recognition. For example, then add the graph entity <t graph , c = 1> to the candidate set ; For example, then add <t graph , c = 0.4> to the candidate set where is a parameter representing an empty set.

5. The graph visualization automatic question answering method based on a knowledge graph according to claim 1, characterized in that, The candidate query graph ranking model based on reinforcement learning generates a 6-dimensional feature vector for each candidate query graph q Specifically, it includes: based on BERT-based semantic matching, entity confidence, entity type count, answer entity count, and aggregation count, feeding the feature vector ν of each candidate query graph q into the fully connected layer of the reinforcement model to obtain p(q|Q), and the training objective is to learn the policy function p q (q∣Q), where θ represents the parameters in the model; p is the policy function; using the F1 score between the predicted answer and the correct answer label as the reward, and the BERT-based semantic matching uses the standard BERT model to measure the semantic similarity between the tokenized question sequence s q and the query graph sequence s g , where s g is generated by sequentially connecting the grounded entity names and relation names along the core relation path, and feeding the sequence [CLS]s q [SEP]s g to the BERT model to calculate their semantic similarity; the entity count is the number of n g in the query graph, where CLS is the sentence start flag; SEP is the flag separating two sentences; the entity type count is the number of entity types; the answer entity count is the number of n an in the query graph; the aggregation count is the number of n ag in the query graph.

Images