Argumentation knowledge graph, text graph encoding model, graph-based pre-training method

Abstract

The application aims to provide an argumentation knowledge graph and an automatic construction method thereof, a model for simultaneously encoding text and the graph and a pre-training method thereof, and a method for secondarily pre-training the model by using graph information. The method comprises the following steps: organizing argument and evidence sentences into intra-sentence semantic graphs and inter-sentence logical graphs, and automatically constructing the graphs from original argumentation corpus; combining a pre-trained language model and a graph neural network to encode text and graph information, and realizing the interaction of the two types of information in the model, and pre-training the corresponding model on argumentation corpus through multiple self-supervised tasks; and expanding the pre-training corpus by using the graph information, adding corresponding pre-training tasks, and applying the tasks to the secondary pre-training of the model or other pre-trained language models. The application can effectively integrate information of various argumentation corpus, and provide a language model with clear and accurate argumentation semantics and logical information, which is helpful for the model to complete argumentation scene tasks requiring complex reasoning and argumentation.

Description

Technical Field

[0001] This invention relates to the field of computer science, and more particularly to an argumentation knowledge graph, a text graph encoding model, and a pre-training method based on the graph. Background Technology

[0002] Autonomous debate is one of the key capabilities of strong artificial intelligence. It requires AI to comment on a topic based on its knowledge of the world and its reasoning conclusions, and to respond to comments or replies from others on the same topic or to their comments. This involves tasks such as understanding, reasoning, and generating debate texts.

[0003] Autonomous debate can provide multi-faceted analysis in decision-making processes (including industrial decisions, public policy decisions, and business decisions). For example, when making unilateral decisions, autonomous debate techniques can be used to provide opposing opinions or ideas, thereby interactively refining the decision. Alternatively, multiple autonomous debate entities can be used to execute the above interactive process, helping decision-makers examine different perspectives and make a comprehensive and optimal choice. For instance, it can be applied to the advertising industry, especially in developing related products such as copywriting generation systems and simulated advertising systems.

[0004] In addition, autonomous debate can be used to simulate community discussions on key issues, helping to explore the possible development of community public opinion when different perspectives are introduced (related comments are generated by a controlled autonomous debate model), thereby assisting in various public relations and evaluation work, and facilitating subsequent evaluation and public relations for scenarios including but not limited to industrial construction.

[0005] Furthermore, autonomous debate can also utilize human dialogue to create AI-powered chatbots that can assist humans in industrial settings. Current research focuses on modeling knowledge and reasoning methods using large-scale parameters of pre-trained language models, but these models do not demonstrate sufficiently good reasoning abilities in complex reasoning situations, such as debate scenarios. Therefore, it is necessary to organize debate knowledge, especially subjective viewpoints and their relationships, explicitly as an auxiliary information source for current language models. Summary of the Invention

[0006] To address the aforementioned shortcomings, this invention provides an argumentation knowledge graph and its automated construction method, a model that simultaneously encodes text and graph and its pre-training method, and a method for secondary pre-training of the model using graph information.

[0007] To achieve the above objectives, this invention proposes a knowledge graph for organizing argument and supporting text, which consists of a lower-level intra-sentence semantic graph and an upper-level inter-sentence logical graph. The key feature is that each argument or supporting text corresponds to a sub-graph of the intra-sentence semantic graph, represented by one of its top nodes. These top nodes representing each argument or supporting text then form the inter-sentence logical graph.

[0008] The sentence-level semantic graph consists of semantic unit nodes and semantic relation edges, satisfying directed acyclicity. The subgraph corresponding to any argument or evidence forms a connected directed acyclic graph, and includes a top node that represents the argument or evidence, satisfying that no other edge in the subgraph points to the top node. The sentence-level logical graph consists of the top node, combination nodes and combination edges, and logical relation edges. Combination nodes connect the corresponding top nodes of multiple parallel arguments or evidences involved in the logical relation through combination edges. Logical relation edges connect the corresponding top nodes or combination nodes according to different logical relations between arguments or evidences.

[0009] This invention also provides an automated method for constructing the argumentation knowledge graph, characterized in that: arguments, evidence, and their relationships are extracted from the original argumentation corpus using an argumentation mining model; then, each argument and evidence sentence is parsed into an AMR graph using an Abstract Semantic Representation (AMR) parsing model; finally, all AMR graphs are merged into an intra-sentence semantic graph by merging isomorphic nodes; and the inter-sentence logical graph is completed based on the argument and evidence relationships obtained by the argumentation mining algorithm.

[0010] When merging isomorphic nodes, the following steps are performed iteratively until the encoding of all nodes is different from each other: First, all nodes are encoded according to node attributes, outgoing edge attributes and the nodes they point to. Then, each group of nodes with the same encoding is replaced with a single new node whose outgoing edge is the same as any node in the group, and whose incoming edge is the union of the incoming edges of all nodes in the group.

[0011] The present invention also provides a model for simultaneously encoding text and corresponding graphs. The model is characterized by taking argumentative text and its corresponding subgraph in the graph as input, firstly segmenting and initially encoding the text, then initially encoding the graph node attributes and edge attributes, then simultaneously processing the text encoding and graph node encoding through multi-layer interaction, and finally using a cross-modal attention layer to output the final text encoding representation and graph node encoding representation.

[0012] Specifically, during text segmentation and preliminary encoding, an existing pre-trained language model and corresponding word segmenter are used. The word segmenter segments the text, and the text is encoded through the embedding layer and the first few layers of the pre-trained language model. When initially encoding graph node attributes and edge attributes, the embedding layer of the pre-trained language model is expanded, and its embedding encoding of graph node attributes and edge attributes is calculated. When jointly processing text encoding and graph encoding through multi-layer interaction, the last few layers of the pre-trained language model are used to process text encoding, and a multi-layer graph neural network with the same number of layers is introduced to process graph node encoding. A fully connected layer is used to process the interaction between text and graph node encoding before each layer outputs to the next layer. When using a cross-modal attention layer to output the final text encoding representation and graph node encoding representation, the text encoding output by the pre-trained language model and the graph node encoding output by the multi-layer graph neural network are input into a unified cross-modal attention layer, and the final text encoding representation and graph node encoding representation are output.

[0013] Furthermore, when processing the graph, the model transforms the unidirectional edges in the graph into bidirectional edges, where the reverse edge attributes are the inverse of the original edge attributes; when processing multiple sentences with contextual relationships, it adds a transition node corresponding to each sentence to the graph and adds a root node to connect all transition nodes; at the same time, for multiple sentences with the same subgraph, only one subgraph is retained.

[0014] The present invention also provides a method for pre-training the model, characterized in that it employs multiple self-supervised tasks, including masking tasks and graph topology tasks, and uses argumentative corpora to pre-train the model.

[0015] The masking tasks include:

[0016] Text masking prediction: Randomly mask or replace words in the text, and predict the masked or replaced words using the model;

[0017] Node mask prediction: Apply the same method as text mask prediction to the graph node attributes;

[0018] Edge mask prediction: Apply the same method as text mask prediction to the graph edge attributes.

[0019] Graph topology tasks include:

[0020] Graph Comparison Learning: Randomly shuffle the attributes of graph nodes and predict whether the attributes of the graph nodes have been shuffled using the model;

[0021] Transition node order prediction: Predict the relative order of the text corresponding to the transition nodes added as described above;

[0022] Edge direction prediction: After transforming a unidirectional edge in the graph into a bidirectional edge as described above, predict the original direction of the edge.

[0023] Finally, the present invention provides a method for secondary pre-training of a model using graph information, characterized in that, for each document participating in pre-training, several related argumentative sentences are sampled from the graph, placed after the document, and the updated document is used to secondary pre-train the language model or the text graph encoding model provided by the present invention.

[0024] When sampling a number of related argumentative sentences from the graph, for each sentence in the document, according to the tightness of the subgraph connection, a sentence related to it is found in the graph, and the tightness is used as a weight for sampling to obtain a batch of related argumentative sentences; when using the updated document to pre-train the language model or the model described in this invention, on the basis of the original pre-training task of the model, the task of judging the angular relationship between the related argumentative sentences and the corresponding document is added.

[0025] Compared with existing technologies, the argumentation knowledge graph provided by this invention organizes the semantic and logical information contained in various argumentation texts in a clear and efficient manner, improving the utilization efficiency and extent of argumentation knowledge. It allows users to start from a single argument or evidence and search for other semantically and logically related arguments and evidence within the graph, or compare the relationships between multiple arguments and evidence, thus facilitating the improvement of accuracy and logical consistency in tasks such as autonomous argumentation comprehension and generation. This argumentation knowledge graph can also be automatically constructed from original argumentation corpora, making it easier to adapt to rapidly expanding corpus sizes. Furthermore, the text graph encoding model provided by this invention can incorporate relevant graph information during the text encoding modeling process, enabling the model to utilize this information to improve performance in tasks such as argumentation comprehension and generation. It can also provide graph node encoding based on text content, facilitating the subsequent application of graph-based algorithms. The graph-based pre-training method provided by this invention can utilize existing language models, allowing the model to master the structured information in the graph without altering the existing architecture, thereby improving model performance in subsequent downstream applications. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of an example of an argumentation knowledge graph provided by the present invention;

[0028] Figure 2 This is a schematic diagram of the automated construction process of an intra-sentence semantic graph for an argumentation knowledge graph provided by the present invention;

[0029] Figure 3 This is a schematic diagram of the overall architecture of a text graph coding model provided by the present invention. Detailed Implementation

[0030] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0031] This invention proposes a knowledge graph for organizing argument and supporting text, consisting of a lower-level intra-sentence semantic graph and an upper-level inter-sentence logical graph. Each argument or supporting text corresponds to a sub-graph of the intra-sentence semantic graph and is represented by one of its top nodes. These top nodes representing each argument or supporting text then form the inter-sentence logical graph.

[0032] Please see Figure 1 , Figure 2 The semantic graph within the sentence consists of semantic unit nodes (such as gun, ban, kill, person, etc.) and semantic relation edges (such as...). Figure 2 The output results of Phase 3 (":ARG0", ":ARG1") constitute the semantic unit nodes and semantic relationship edges, which characterize semantic information through attribute values ​​and satisfy the directed acyclic property (DAG), meaning there are no loops in the graph. Furthermore, the subgraph corresponding to any argument or premise forms a connected directed acyclic graph, such as... Figure 2 The phrase "We should ban guns." corresponds to the sub-graphs formed by "recommend-01", "ban-01", "we", and "gun". These sub-graphs are in... Figure 1 The subgraph is abstracted into G0 (including the connected TOP0), G1 (including TOP1), etc.; at the same time, each subgraph includes a vertex node that can represent the argument or evidence, satisfying that no other edge in the subgraph points to the vertex node, i.e. Figure 1 The nodes such as TOP0 and TOP1 in the data.

[0033] Please see Figure 1 The logical graph between sentences is composed of top nodes, combination nodes, combination edges, and logical relation edges. Combination nodes connect the corresponding top nodes of multiple parallel arguments or evidence involved in a logical relation through combination edges, for example, a certain connection... Figure 1The top 0 and top 1 nodes (not shown in the diagram) are connected by logical relationship edges, which connect the corresponding top nodes or combined nodes based on the different logical relationships between arguments or evidence, such as... Figure 1 The edges are represented by dashed lines such as Support and Attack. The attribute values ​​of these logical relationship edges record the logical information between the nodes forming the logical relationship edge, while the node attributes are left blank. In other words, when two top nodes form a logical relationship edge, the attribute values ​​of this logical relationship edge record the logical information of the two top nodes; when a top node and a combined node form a logical relationship edge, the attribute values ​​of this logical relationship edge record the logical information of the top node and the combined node; when two combined nodes form a logical relationship edge, the attribute values ​​of this logical relationship edge record the logical information between the two combined nodes.

[0034] This invention also provides an automated method for constructing the aforementioned argumentation knowledge graph. Please refer to [link / reference needed]. Figure 2 This method first extracts arguments, evidence, and their relationships from the original argumentative corpus using an argument mining model. Figure 2 In stage 1), the argument mining model can be a simple sentence segmentation, filtering, or a more complex argument extraction model. Then, the Abstract Semantic Representation (AMR) parsing model is used to parse each argument and supporting sentence into an AMR graph (i.e., Figure 2 In stage 2), automated processing can be achieved using publicly available models on the internet. Finally, all AMR graphs are merged into the intra-sentence semantic graph (i.e., by merging isomorphic nodes). Figure 2 Stage 3), and complete the logical graph between sentences based on the relationship between arguments and evidence obtained by the argument mining algorithm. Figure 2 Not shown in the drawing, please refer to the reference. Figure 1 (The dashed edge in the text). Among them, the argument extraction model and the argument mining algorithm can adopt a variety of argument extraction / argument mining algorithms and models, such as Niculae et al. (2017), Eger et al. (2017), Schulz et al. (2018) and Cheng et al. (2020). The above algorithms and models have published open source code, which can be applied to this scenario after simple modification.

[0035] The merging of isomorphic nodes involves iteratively executing the following steps until all nodes are paired differently: First, all nodes are encoded according to their node attributes, outgoing edge attributes, and the nodes they point to. Then, for each group of nodes with the same encoding, a single new node is used, whose outgoing edges are the same as any node in the group, and whose incoming edges are the union of the incoming edges of all nodes in the group. The specific algorithm implementation can be readily obtained from the above description and will not be elaborated further.

[0036] This invention also provides a model for simultaneously encoding text and its corresponding graph. The input is argumentative text and its corresponding sub-graph in the graph. The model first segments and initially encodes the text, then initially encodes the graph node attributes and edge attributes, and then simultaneously processes the text encoding and graph node encoding through multi-layer interactive joint processing. Finally, a cross-modal attention layer is used to output the final text-encoded representation and graph node-encoded representation. The corresponding sub-graph in the argumentative knowledge graph is at least one of an intra-sentence semantic graph and / or an inter-sentence logical graph. That is, it is possible to obtain a portion of the intra-sentence semantic graph, or a portion of the inter-sentence logical graph, or both. Graph node attributes and edge attributes can be characterized by attribute values ​​of the semantic unit nodes and semantic relationship edges in the intra-sentence semantic graph, or they can be attribute values ​​of the logical relationship edges in the inter-sentence logical graph, with node attributes left blank.

[0037] Please see Figure 3During the text segmentation and initial encoding, an existing pre-trained language model (e.g., RoBERTa) and corresponding tokenizer are used. The tokenizer (“Tokenizer” in the figure) segments the text and encodes it through the embedding layer (“Token Embed.Layer” in the figure) and the first few layers (“LMBlock” in the figure) of the pre-trained language model, resulting in the second “Token Embeddings” from the left in the figure. When initially encoding the graph node attributes and edge attributes, the embedding layer of the pre-trained language model is expanded (the expanded part is the “Node & Edge Embed.Layer” in the figure), and it is used to calculate the embedding encoding of the graph node attributes and edge attributes, i.e., “Node & Edge Embed.” in the figure. When the text encoding and graph encoding are processed jointly through multi-layer interaction, as shown in the “Joint Block” module in the figure, the last few layers of the pre-trained language model are used to process the text encoding, and a multi-layer graph neural network (e.g., Graph) with the same number of layers is introduced. The Transformer process handles the graph node encoding, and before each layer outputs to the next layer, a fully connected layer (“Mix Layer” in the diagram) is used to process the interaction parts in the text and graph node encoding. The text interaction part is selected as the first word of the text (the dark squares in each “Token Embeddings” in the diagram), and the graph interaction part is selected as the added root node (i.e., the R node in “Graph Input” in the diagram). When using the cross-modal attention layer to output the final text encoding representation and graph node encoding representation, the text encoding output by the pre-trained language model and the graph node encoding output by the multi-layer graph neural network are input into a unified cross-modal attention layer (“Cross-Modal Attn.Layer” in the diagram), and the final text encoding representation and graph node encoding representation are output, i.e., “TextRepresentation” and “Graph Representation” on the right side of the diagram.

[0038] In addition, please see Figure 3When processing the graph, the model transforms unidirectional edges into bidirectional edges, where the reverse edge attributes are the inverse of the original edge attributes. For example, in the figure, "Graph Input" only has unidirectional edges, but after transformation and encoding, "Node & Edge Embed" has bidirectional edges. When processing multiple sentences with contextual relationships, a transition node corresponding to each sentence is added to the graph (nodes S1 and S2 in "Graph Input" in the figure), and a root node is added to connect all transition nodes (node ​​R in "Graph Input" in the figure). At the same time, for multiple sentences with the same subgraph, only one subgraph is retained.

[0039] The present invention also provides a method for pre-training the model, employing various self-supervised tasks, including masking tasks and graph topology tasks, and using argumentative corpora to pre-train the model.

[0040] The masking tasks include:

[0041] Text masking prediction: Randomly mask or replace words in the text (80% probability of masking with special symbols, 10% probability of replacing with random words, and the remaining 10% probability of remaining unchanged). Predict the masked or replaced words through the model. Based on the results of node masking and edge masking as described below, firstly mask or replace the parts of the text corresponding to the masked or replaced nodes and edges. Then, if the proportion of masked or replaced text is less than 30%, additionally randomly mask or replace a portion of the previously unaffected text.

[0042] Node mask prediction: Apply the same method as the text mask prediction to the graph node attributes, but only process 15% of the nodes;

[0043] Edge mask prediction: Apply the same method as the text mask prediction to the graph edge attributes, but only process 15% of the edges.

[0044] Graph topology tasks include:

[0045] Graph Comparison Learning: Randomly shuffle the attributes of graph nodes and predict whether the attributes of the graph nodes have been shuffled using the model. The model takes one text and two graphs (one normal graph and one graph with shuffled node attributes) as input and outputs whether the nodes in the two graphs belong to the normal graph or the shuffled graph.

[0046] Transition node order prediction: Predict the relative order of the text corresponding to the transition nodes added as described above, that is, whether the sentence corresponding to one transition node appears before the sentence corresponding to another transition node, where only the transition nodes corresponding to adjacent sentences are considered.

[0047] Edge direction prediction: After transforming the unidirectional edges in the graph into bidirectional edges as described above, predict the original direction of the edges, that is, predict which direction of the bidirectional edges comes from the original graph and which is a subsequent addition.

[0048] Among these features, the prediction of transition node order and edge direction are specifically designed based on the proposed argument graph structure and application method. Although multi-task pre-training is a framework already practiced by predecessors, the unique aspect of this invention lies in combining the various self-supervised tasks for pre-training, particularly masking and graph topology tasks. Finally, this invention provides a method for secondary pre-training of a model using graph information. For each document participating in pre-training, several related argument sentences are sampled from the graph, placed after the document, and the updated document is used to secondary pre-train the language model or the text graph encoding model provided by this invention.

[0049] When sampling a number of related argumentative sentences from the graph, for each sentence in the document, sentences associated with it are found in the graph according to the tightness of the subgraph connections. The tightness of two sentences is defined as follows: an undirected bipartite graph is constructed between the transition node and the original nodes in the graph, where the transition node connects all nodes under the corresponding sentence's subgraph. The tightness of two sentences is the probability of reaching the transition node of another sentence after starting from the transition node of one sentence and moving randomly along the edges of the bipartite graph twice. Sampling is performed according to the tightness as weights to obtain a batch of related argumentative sentences. When using the updated document for secondary pre-training of the language model or the model described in this invention, in addition to the original pre-training tasks of the model (such as text mask prediction of RoBERTa and the various self-supervised tasks of the model described in this invention), a task is added to determine the angular relationship between the related argumentative sentences and the corresponding documents, that is, to consider whether the stance of the argumentative sentence on its topic is the same as or opposite to the stance of the original document on the same topic.

[0050] The devices or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware, or a module that implements the same function can be implemented by a combination of multiple sub-modules, etc. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0051] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0052] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0053] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0054] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0055] Although this application has been described by way of examples, those skilled in the art will know that this application has many variations and modifications without departing from the spirit of this application, and it is intended that the appended claims cover such variations and modifications without departing from this application.

Claims

1. An automated method for constructing a knowledge graph of argumentation, characterized in that, The argumentation knowledge graph includes a lower-level intra-sentence semantic graph and an upper-level inter-sentence logic graph. The intra-sentence semantic graph includes multiple sub-graphs that correspond one-to-one with each argument or evidence, and each sub-graph has a top node. The inter-sentence logic graph is composed of top nodes that represent each argument or evidence. The semantic graph within the sentence is composed of semantic unit nodes and semantic relation edges. The semantic unit nodes and semantic relation edges describe semantic information through attribute values ​​and satisfy the directed acyclic property. The subgraph corresponding to any argument or evidence constitutes a connected directed acyclic graph and includes a top node that can represent the argument or evidence. The subgraph corresponding to the argument or evidence has no other edges pointing to the top node. The sentence-to-sentence logic graph includes multiple top nodes, at least two top nodes form a combined node through combined edges, and the two top nodes, the top node and the combined node, and the two combined nodes are connected by logical relationship edges. The attribute values ​​of the logical relationship edges record the logical information between the nodes that constitute the logical relationship edges, while the node attributes are left blank. Argument mining model extracts arguments, evidence, and their relationships from the original argument corpus. Then, abstract semantic representation parsing model is used to parse each argument and evidence sentence into an AMR graph. Finally, all the AMR graphs are merged into the semantic graph within the sentence by merging isomorphic nodes. The logical graph between sentences is then completed based on the relationship between arguments and evidence obtained by the argument mining algorithm.

2. The method as described in claim 1, characterized in that: The step "merging all the AMR maps into the intra-sentence semantic map by merging isomorphic nodes" includes: Perform the following steps iteratively until all nodes are encoded differently: First, encode all nodes according to their node attributes, outgoing edge attributes, and the nodes they point to. Then, replace each pair of nodes with the same encoding with a single new node whose outgoing edges are the same as any node in the group, and whose incoming edges are the union of the incoming edges of all nodes in the group.

3. An encoding method that simultaneously encodes text and its corresponding argumentation knowledge graph, characterized in that, The argumentation knowledge graph includes a lower-level intra-sentence semantic graph and an upper-level inter-sentence logic graph. The intra-sentence semantic graph includes multiple sub-graphs that correspond one-to-one with each argument or evidence, and each sub-graph has a top node. The inter-sentence logic graph is composed of top nodes that represent each argument or evidence. The semantic graph within the sentence is composed of semantic unit nodes and semantic relation edges. The semantic unit nodes and semantic relation edges describe semantic information through attribute values ​​and satisfy the directed acyclic property. The subgraph corresponding to any argument or evidence constitutes a connected directed acyclic graph and includes a top node that can represent the argument or evidence. The subgraph corresponding to the argument or evidence has no other edges pointing to the top node. The sentence-to-sentence logic graph includes multiple top nodes, at least two top nodes form a combined node through combined edges, and the two top nodes, the top node and the combined node, and the two combined nodes are connected by logical relationship edges. The attribute values ​​of the logical relationship edges record the logical information between the nodes that constitute the logical relationship edges, while the node attributes are left blank. The process involves obtaining the argument text and its corresponding subgraph in the argument knowledge graph, segmenting the text into words and performing preliminary encoding, then performing preliminary encoding on the node attributes and edge attributes of the argument knowledge graph, and simultaneously processing the pre-encoded text and graph node encodings through multi-layer interactive joint processing. Finally, a cross-modal attention layer is used to output the final text encoding representation and graph node encoding representation. The corresponding subgraph in the argument knowledge graph is at least one of an intra-sentence semantic graph and / or an inter-sentence logical graph.

4. The encoding method as described in claim 3, characterized in that: The step "text segmentation and preliminary encoding" includes: segmenting the text using a word segmenter and encoding it through the embedding layer and the first few layers of a pre-trained language model; The step "to preliminarily encode the node attributes and edge attributes of the argumentation knowledge graph" includes: expanding the embedding layer of the pre-trained language model and using it to calculate the embedding encoding of the graph node attributes and edge attributes; The step "jointly process text encoding and graph encoding through multi-layer interaction" includes: using the last few layers of the pre-trained language model to process text encoding, introducing a multi-layer graph neural network with the same number of layers to process graph node encoding, and using a fully connected layer to process the interaction part in text and graph node encoding before each layer outputs to the next layer. The step "outputting the final text encoding representation and graph node encoding representation using the cross-modal attention layer" includes: inputting the text encoding output by the pre-trained language model and the graph node encoding output by the multi-layer graph neural network into a unified cross-modal attention layer, and outputting the final text encoding representation and graph node encoding representation.

5. The encoding method as described in claim 3, characterized in that, The step "Multilayer Graph Neural Network Processing Graph Node Encoding" includes: transforming unidirectional edges in the graph into bidirectional edges, where the reverse edge attributes are the inverse of the original edge attributes; when processing multiple sentences with contextual relationships, adding a transition node corresponding to each sentence to the graph, and adding a root node to connect all transition nodes; and for multiple sentences with the same subgraph, only one subgraph is retained.

6. A pre-training method for a language model formed by the encoding method as described in claim 3, characterized in that, We employ various self-supervised tasks, including masking and graph topology tasks, and use argumentation corpora for pre-training.

7. The method as described in claim 6, characterized in that, The masking tasks include: Text masking prediction: Randomly mask or replace words in the text, and predict the masked or replaced words using the model; Node mask prediction: Apply the same method as text mask prediction to the graph node attributes; Edge mask prediction: Apply the same method as text mask prediction to the graph edge attributes; The graph topology tasks include: Graph Comparison Learning: Randomly shuffle the attributes of graph nodes and predict whether the attributes of the graph nodes have been shuffled using the model; Transition node order prediction: Predicts the relative order of the text corresponding to the added transition nodes; Edge direction prediction: After converting a unidirectional edge in the graph into a bidirectional edge, predict the original direction of the edge.

8. A method for pre-training a model using graph information from an argumentation knowledge graph, characterized in that, The argumentation knowledge graph includes a lower-level intra-sentence semantic graph and an upper-level inter-sentence logic graph. The intra-sentence semantic graph includes multiple sub-graphs that correspond one-to-one with each argument or evidence, and each sub-graph has a top node. The inter-sentence logic graph is composed of top nodes that represent each argument or evidence. The semantic graph within the sentence is composed of semantic unit nodes and semantic relation edges. The semantic unit nodes and semantic relation edges describe semantic information through attribute values ​​and satisfy the directed acyclic property. The subgraph corresponding to any argument or evidence constitutes a connected directed acyclic graph and includes a top node that can represent the argument or evidence. The subgraph corresponding to the argument or evidence has no other edges pointing to the top node. The sentence-to-sentence logic graph includes multiple top nodes, at least two top nodes form a combined node through combined edges, and the two top nodes, the top node and the combined node, and the two combined nodes are connected by logical relationship edges. The attribute values ​​of the logical relationship edges record the logical information between the nodes that constitute the logical relationship edges, while the node attributes are left blank. For each document participating in pre-training, several related argumentative sentences are sampled from the graph and placed after the document. The updated document is then used to pre-train the language model again, or the language model is formed after encoding. When sampling several related argumentative sentences from the graph, for each sentence in the document, sentences related to it are found in the graph according to the tightness of the sub-graph connections, and the tightness is used as the weight for sampling to obtain a batch of related argumentative sentences. When using the updated document to pre-train the language model again, or the language model is formed after encoding, the task of judging the angular relationship between the related argumentative sentences and the corresponding document is added to the original pre-training task of the model.

Images