Text processing model training method and device and text processing method and device

Abstract

The invention provides a training method of a text processing model and a text processing method and device.The training method of the text processing model comprises the steps that an initial concept atlas of a training text is determined based on a knowledge atlas, the initial concept atlas is input into a relation awareness attention network RGAT model for training, and a target RGAT model is obtained, in the training process, the first concept atlas in the (i + 1) th iteration is determined according to the relevancy between nodes in the second concept atlas in the ith iteration and the training text and the weight of edges in the second concept atlas, the first concept atlas is a sub-graph of the initial concept atlas, and the second concept atlas is a sub-graph of the initial concept atlas. According to the scheme, the training effect of the RGAT model can be improved, more accurate knowledge-level codes can be learned, and therefore the accuracy of downstream text processing tasks is improved.

Description

Text processing model training method, text processing method and device Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a text processing model training method, a text processing method and a device. Background Art

[0002] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0003] Natural language processing (NLP) is a key research area in artificial intelligence. Natural language processing tasks typically rely on processing natural language text itself. However, natural language text itself contains relatively limited features, and the text processing results may not meet expectations. Some solutions utilize knowledge graphs as auxiliary information for text processing, but these solutions may introduce knowledge graph information completely irrelevant to the text content, thus affecting the effectiveness of text processing.

[0004] Summary of the Invention

[0005] The present application provides a text processing model training method, a text processing method and a device, which can improve the effect of text processing.

[0006] In a first aspect, a training method for a text processing model is provided, comprising: obtaining training text; obtaining a knowledge graph; determining an initial concept graph of the training text based on the knowledge graph, wherein the nodes in the initial concept graph include topic nodes, the topic nodes include candidate entities in the knowledge graph corresponding to target noun phrases in the training text, and the edges between the nodes in the initial concept graph are used to represent the entity relationships between the nodes in the initial concept graph; the initial concept graph is input into a relationship-aware attention network RGAT model for training to obtain a target RGAT model, wherein during the training process, the first concept graph in the i+1th iteration is determined based on the relevance of the nodes in the second concept graph in the i-th iteration to the training text and the weights of the edges in the second concept graph, where i is a positive integer, the first concept graph is a subgraph of the initial concept graph, and the second concept graph is a subgraph of the initial concept graph.

[0007] According to the solution of the embodiment of the present application, during the iterative training process of the RGAT model, the concept map can be optimized based on the relevance of the nodes in the concept map to the training text, as well as the importance of the edges in the concept map, and the optimized concept map is used as the concept map for the next iteration. This helps reduce the RGAT model's focus on knowledge with less text relevance and strengthens the model's focus on knowledge with greater text relevance, thereby improving the training effect of the RGAT model, learning more accurate knowledge-level encoding, and thus improving the accuracy of downstream text processing tasks.

[0008] Exemplarily, the knowledge graph may be a knowledge graph of the professional field to which the training text belongs.

[0009] A target noun phrase is a noun phrase in the text that corresponds to at least one candidate entity in the knowledge graph. In other words, if a noun phrase in the training text corresponds to at least one candidate entity in the knowledge graph, then that noun phrase can be used as a target noun phrase in the training text.

[0010] The concept graphs in the iterative process are all subgraphs of the initial concept graph. The relevance between a node in the iterative concept graph and the training text is the same as the relevance between the same node in the initial concept graph and the training text.

[0011] For example, the relevance between the nodes in the initial concept graph and the training text may be determined based on the importance of the nodes, which may be determined by the eigenvector centrality of the nodes.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the topic node includes all candidate entities corresponding to the target noun phrase in the knowledge graph.

[0013] According to the solution of the embodiment of this application, the subject node can include all candidate entities corresponding to each target noun phrase in the knowledge graph. The resulting concept graph covers all candidate entities related to the text data and the corresponding entity relationships. This can provide comprehensive and complete knowledge related to the text for subsequent processing. Using the solution of the embodiment of this application to learn the knowledge-level expression of the text further ensures the accuracy of downstream tasks and avoids incorrect reasoning paths caused by missing some knowledge.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the first concept graph in the (i+1)th iteration is determined based on the relevance of the nodes in the second concept graph in the (i)th iteration to the training text and the weight of the edges in the second concept graph, including: selecting edges in the second concept graph as edges in the first concept graph in descending order of the ratio of the benefit to the first consumption of the edges in the second concept graph until the sum of the first consumptions of the selected edges is greater than a threshold, the benefit of the edges in the second concept graph is positively correlated with the weight of the edges in the (i)th iteration, and the first consumption of the edges in the second concept graph is negatively correlated with the relevance of the two nodes connected by the edges to the training text.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the first concept graph in the (i+1)th iteration is determined based on the relevance of the nodes in the second concept graph in the (i)th iteration to the training text and the weights of the edges in the second concept graph, including: selecting the connected subgraphs in the second concept graph as the connected subgraphs of the first concept graph in order of the second consumption of the connected subgraphs in the second concept graph from small to large, until all the selected connected subgraphs include at least one candidate entity corresponding to the target noun phrase.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the relevance of the topic node to the training text is determined based on the eigenvector centrality of the topic node in the topic correlation graph, the nodes in the topic correlation graph include the topic node, and the weight of the edge in the topic correlation graph is determined based on the number of entity relationships between the entities corresponding to the two nodes connected by the edge in the knowledge graph.

[0017] The eigenvector centrality of a topic node is determined based on the initial importance of the topic node and the weight of the edge in the topic-related graph. The initial importance of a node in the topic-related graph is set based on the probability of the node appearing in the facts recorded in the knowledge graph.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the initial concept graph further includes neighbor nodes, and the neighbor nodes include neighbor entities of the candidate entities corresponding to the target noun phrase in the knowledge graph.

[0019] The initial concept graph of the embodiment of the present application also includes neighbor entities connected to the candidate entity (i.e., subject node) in the knowledge graph, i.e., neighbor nodes, which can further provide more comprehensive and complete knowledge and help improve the accuracy of knowledge-level encoding.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the relevance of the neighbor node to the training text is determined based on the score of the strongly connected branch where the neighbor node is located on the information propagation graph. The nodes in the information propagation graph include nodes in the initial concept graph. When the first node in the initial concept graph is a one-hop neighbor of the second node, there is a directed edge between the second node and the first node in the information propagation graph, pointing from the second node to the first node.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the score of the strongly connected branch on the information propagation graph is obtained by propagating the initial score of the strongly connected branch where the subject node is located to the downstream strongly connected branch according to topological sorting, and the initial score of the strongly connected branch where the subject node is located is determined based on the maximum importance of the nodes in the strongly connected branch where the subject node is located.

[0022] In a second aspect, a method for text processing is provided, which includes: obtaining a text to be processed; obtaining a knowledge graph; determining a text encoding of the text to be processed; determining a concept graph of the text to be processed based on the knowledge graph; processing the concept graph of the text to be processed by a target RGAT to obtain a knowledge encoding of the text to be processed, wherein the target RGAT is obtained by inputting an initial concept graph of a training text into the RGAT for training, and during the training process, the first concept graph in the i+1th iteration is determined based on the relevance of the nodes in the second concept graph in the i-th iteration to the training text and the weight of the edges in the second concept graph, i is a positive integer, the first concept graph is a subgraph of the initial concept graph, the second concept graph is a subgraph of the initial concept graph, the nodes in the initial concept graph include topic nodes, the topic nodes include candidate entities in the knowledge graph corresponding to the target noun phrases in the training text, and the edges between the nodes in the initial concept graph are used to represent the entity relationship between the nodes in the initial concept graph; the processing result of the text to be processed is determined based on the text encoding of the text to be processed and the knowledge encoding of the text to be processed.

[0023] According to the solution of the embodiment of the present application, during the iterative training process of the RGAT model, the concept map can be optimized based on the relevance of the nodes in the concept map to the training text, as well as the importance of the edges in the concept map, and the optimized concept map is used as the concept map for the next iteration. This helps reduce the RGAT model's focus on knowledge with less text relevance and strengthens the model's focus on knowledge with greater text relevance, thereby improving the training effect of the RGAT model, learning more accurate knowledge-level encoding, and thus improving the accuracy of the processing results.

[0024] In combination with the second aspect, in certain implementations of the second aspect, the topic node includes all candidate entities corresponding to the target noun phrase in the knowledge graph.

[0025] The topic nodes in the concept graph of the text to be processed may include all candidate entities in the knowledge graph corresponding to the target noun phrase in the text to be processed.

[0026] According to the solution of the embodiment of this application, the subject node can include all candidate entities corresponding to each target noun phrase in the knowledge graph. The resulting concept graph covers all candidate entities related to the text data and the corresponding entity relationships. This can provide comprehensive and complete knowledge related to the text for subsequent processing, avoiding incorrect reasoning paths caused by missing some knowledge. Moreover, the target RGAT focuses on knowledge with high relevance to the text during training, further ensuring the accuracy of downstream tasks.

[0027] In combination with the second aspect, in certain implementations of the second aspect, the first concept graph in the (i+1)th iteration is determined based on the relevance of the nodes in the second concept graph in the (i)th iteration to the training text and the weight of the edges in the second concept graph, including: selecting edges in the second concept graph as edges in the first concept graph in descending order of the ratio of the benefit to the first consumption of the edges in the second concept graph until the sum of the first consumptions of the selected edges is greater than a threshold, the benefit of the edges in the second concept graph is positively correlated with the weight of the edges in the (i)th iteration, and the first consumption of the edges in the second concept graph is negatively correlated with the relevance of the two nodes connected by the edges to the training text.

[0028] In combination with the second aspect, in certain implementations of the second aspect, the first concept graph in the i+1th iteration is determined based on the relevance of the nodes in the second concept graph in the i-th iteration to the training text and the weights of the edges in the second concept graph, including: selecting the connected subgraphs in the second concept graph as the connected subgraphs of the first concept graph in order of the second consumption of the connected subgraphs in the second concept graph from small to large, until all the selected connected subgraphs include at least one candidate entity corresponding to the target noun phrase.

[0029] In combination with the second aspect, in certain implementations of the second aspect, the relevance of the topic node to the training text is determined based on the eigenvector centrality of the topic node in the topic correlation graph, the nodes in the topic correlation graph include the topic node, and the weight of the edge in the topic correlation graph is determined based on the number of entity relationships between the entities corresponding to the two nodes connected by the edge in the knowledge graph.

[0030] The eigenvector centrality of a topic node is determined based on the initial importance of the topic node and the weight of the edge in the topic-related graph. The initial importance of a node in the topic-related graph is set based on the probability of the node appearing in the facts recorded in the knowledge graph.

[0031] In combination with the second aspect, in certain implementations of the second aspect, the nodes in the initial concept graph also include neighbor nodes, and the neighbor nodes include neighbor entities of the candidate entities corresponding to the target noun phrase in the knowledge graph.

[0032] The neighbor nodes in the concept graph of the text to be processed may include neighbor entities in the knowledge graph of the candidate entity corresponding to the target noun phrase in the text to be processed.

[0033] In combination with the second aspect, in certain implementations of the second aspect, the relevance of the neighbor node to the training text is determined based on the score of the strongly connected branch where the neighbor node is located on the information propagation graph. The nodes in the information propagation graph include nodes in the initial concept graph. When the first node in the initial concept graph is a one-hop neighbor of the second node, there is a directed edge between the second node and the first node in the information propagation graph, pointing from the second node to the first node.

[0034] In combination with the second aspect, in certain implementations of the second aspect, the score of the strongly connected branch on the information propagation graph is obtained by propagating the initial score of the strongly connected branch where the subject node is located to the downstream strongly connected branch according to topological sorting, and the initial score of the strongly connected branch where the subject node is located is determined based on the maximum importance of the nodes in the strongly connected branch where the subject node is located.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the method further includes: outputting a knowledge path in the concept map of the text to be processed based on the text encoding of the text to be processed and the knowledge encoding of the text to be processed, and the knowledge path is used to indicate the basis for judging the processing result.

[0036] A knowledge path is a path between two nodes in the concept map. For example, node e q and node e q+k The k-hop knowledge path between q ,r q ,e q+1 ,r q+1 ,…,r q+k-1 ,e q+k ),(e q ,r q ,e q+1 ) is a triple, r qRepresents the entity relationship between two nodes, and so on. q is a positive integer.

[0037] The knowledge path can improve the interpretability of the model and provide users with a basis for judging the processing results, that is, the complete logic for deriving the processing results, which is conducive to improving user trust.

[0038] In a third aspect, a training device for a text processing model is provided, which includes a unit for executing the method of any one of the implementation modes of the first aspect.

[0039] In a fourth aspect, a text processing apparatus is provided, which includes a unit for executing the method of any one of the implementation modes of the second aspect.

[0040] In a fifth aspect, a training device for a text processing model is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in any one of the implementations of the first aspect.

[0041] The processor in the fifth aspect can be either a central processing unit (CPU) or a combination of a CPU and a neural network processor. The neural network processor here can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU). The TPU is a custom AI accelerator ASIC developed by Google for machine learning.

[0042] In a sixth aspect, a text processing device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute a method unit of any one of the implementation methods of the above-mentioned second aspect.

[0043] The processor in the sixth aspect mentioned above can be either a CPU or a combination of a CPU and a neural network operation processor. The neural network operation processor here can include a GPU, NPU or TPU, etc.

[0044] In a seventh aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementations of any one of the first to second aspects.

[0045] In an eighth aspect, a computer program product comprising instructions is provided, which, when run on a computer, enables the computer to execute a method in any one of the implementations of any one of the first to second aspects above.

[0046] In a ninth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes a method in any one of the implementations of any one of the first to second aspects.

[0047] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the method in any implementation method of any aspect from the first aspect to the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG1 is a schematic block diagram of a natural language processing system provided in an embodiment of the present application;

[0049] FIG2 is a schematic block diagram of a system architecture provided in an embodiment of the present application;

[0050] FIG3 is a schematic block diagram of a text processing system provided in an embodiment of the present application;

[0051] FIG4 is a schematic block diagram of a text processing model provided in an embodiment of the present application;

[0052] FIG5 is a schematic flow chart of a method for training a text processing model provided in an embodiment of the present application;

[0053] FIG6 is a schematic diagram of a knowledge extraction method provided in an embodiment of the present application;

[0054] FIG7 is a schematic diagram of a process for constructing a concept map provided in an embodiment of the present application;

[0055] FIG8 is a schematic diagram of a topic-related graph provided in an embodiment of the present application;

[0056] FIG9 is a schematic diagram of an information dissemination diagram provided in an embodiment of the present application;

[0057] FIG10 is a schematic diagram of another text processing model training method provided in an embodiment of the present application;

[0058] FIG11 is a schematic flow chart of a text processing method provided in an embodiment of the present application;

[0059] FIG12 is a schematic flow chart of another text processing method provided in an embodiment of the present application;

[0060] FIG13 is a schematic diagram of a text classification result provided in an embodiment of the present application;

[0061] FIG14 is a schematic block diagram of a training device provided in an embodiment of the present application;

[0062] FIG15 is a schematic block diagram of a text processing apparatus provided in an embodiment of the present application;

[0063] FIG16 is a schematic block diagram of another training device provided in an embodiment of the present application;

[0064] FIG17 is a schematic block diagram of another text processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solution in this application will be described below with reference to the accompanying drawings.

[0066] Natural language processing (NLP) is a key research area in artificial intelligence, enabling human-machine interaction through natural language. Natural language processing tasks typically execute processing based on natural language text itself. Natural language text itself contains relatively limited features, and the text processing results may not meet expectations. To improve text processing, some solutions have introduced knowledge graphs as auxiliary information for text processing. However, the introduction of knowledge graphs can introduce other problems. For example, the ambiguity of entities in the knowledge graph and the noise in the knowledge graph can lead to the introduction of knowledge graph information completely irrelevant to the text content during text processing, making the accuracy of the text processing results impossible to guarantee.

[0067] The embodiment of the present application provides a method for text processing, which is conducive to improving the effect of text processing.

[0068] In order to facilitate understanding of the embodiments of the present application, the relevant concepts of the relevant terms involved in the embodiments of the present application are first introduced below.

[0069] (1) Natural Language Processing

[0070] Natural language refers to human language, and natural language processing (NLP) is the processing of human language. Natural language processing is the process of systematically analyzing, understanding, and extracting information from text data in an intelligent and efficient manner. NLP and its components can manage very large amounts of text data or perform a large number of automated tasks, solving a wide variety of problems such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering, and topic segmentation.

[0071] (2) Knowledge Graph (KG)

[0072] A knowledge graph is a knowledge base that integrates real-world facts through a graph-structured data model. Knowledge graphs are typically used to store interconnected entities. For example, a fact representing a relationship between two entities can be represented as a triple data structure of the form (entity, entity relationship, entity).

[0073] Entities, represented as nodes in the knowledge graph, represent conceptual entities in the real world. For example, "Peking University (organization)", "Vitamin B12 (medical element)", and "Hemoglobin (medical element)". Entity relationships, represented as edges between nodes corresponding to two entities in the knowledge graph, represent the relationship between two entities in the real world. For example, the entity relationship between "Vitamin B12" and "Hemoglobin" is "Increase". The fact indicated by (Vitamin B12, Increase, Hemoglobin) is that vitamin B12 increases hemoglobin.

[0074] A knowledge graph in a professional field refers to a knowledge graph that contains the entities, relationships, and facts in that field. For example, a knowledge graph in the financial field is used to represent the entities, relationships, and facts in the financial field. A knowledge graph in the medical field is used to represent the entities, relationships, and facts in the medical field.

[0075] The triple data structure that can express facts extracted from natural language text can be called the knowledge triple of the text, in the form of (noun phrase, relation phrase, noun phrase). Among them, the noun phrase can include one word or multiple words. The relation phrase can include one word or multiple words.

[0076] The noun phrase can correspond to one or more candidate entities in the knowledge graph. For example, the Chinese noun phrase "苹果" can correspond to candidate entities such as "苹果(水果)" or "苹果(公司)". Another example is that the English noun phrase "anemia" can correspond to candidate entities such as "anemia(disease)", "anemia(symptom)", or "anemia(plant)".

[0077] (3) Knowledge graph embedding

[0078] Knowledge graph embedding refers to mapping the entities and entity relationships in the knowledge graph to a low-dimensional vector space to obtain the embedding representation of the knowledge graph, and realizing the semantic information representation of entities and entity relationships. The embedding representation of the knowledge graph can be used for various tasks related to the knowledge graph. Exemplarily, the embedding representation of the knowledge graph can include at least one of the following: the embedding representation of entities or the embedding representation of relationships, etc.

[0079] The embedding representation of the knowledge graph can be obtained through a knowledge graph embedding model. The knowledge graph embedding model can be implemented based on a graph neural network (GNN).

[0080] (4) The k-hop neighbours of a node

[0081] In graph theory, the k-hop neighbours of a node in a graph refer to the set of all nodes whose shortest path to this node is k hops starting from this node. k is a positive integer. <00....>[The original text seems to be incomplete here. Please check and provide the full text if possible.]

[0082] (5) Directed acyclic graph (DAG)

[0083] In graph theory, if a directed graph cannot return to this point after passing through several edges starting from any node, then this graph is a directed acyclic graph.

[0084] (6) Topological sorting

[0085] To perform a topological sort on a directed acyclic graph G, all nodes in G are arranged into a linear sequence so that for any pair of nodes u and v in the graph, the edge<u,v> Represents the path from node u to node v, and the edge set E(G) represents the set of edges between each node in G.<u,v> ∈E(G), then u appears before v in the linear sequence. Usually, such a linear sequence is called a sequence that satisfies topological order, or simply a topological sequence.

[0086] In other words, the topological sequence needs to meet two conditions:

[0087] 1) Each node in the directed acyclic graph G appears only once in the topological sequence.

[0088] 2) If there is a path from node u to node v in the directed acyclic graph G, then node u appears before node v in the topological sequence.

[0089] A directed acyclic graph can have one or more topological sorting sequences.

[0090] (7) Strongly connected branches

[0091] A directed graph G is said to be strongly connected if, for any two nodes v and u, there exists a directed path from v to u and from u to v. In a directed graph G, if two nodes u and v have a directed path in both directions, then u and v are said to be strongly connected.

[0092] In other words, if two nodes v and u in a directed graph G can reach each other, then the two nodes are strongly connected. If every two nodes in a directed graph G are strongly connected, then the directed graph G is called a strongly connected graph.

[0093] For a strongly connected subgraph S of a directed graph G that is not a strongly connected graph, if adding any node to S causes S to lose its strongly connected property, then the strongly connected subgraph S is a maximal strongly connected subgraph of G, or S can also be called a strongly connected branch of G.

[0094] Exemplarily, the Tarjan algorithm can be used to solve the strongly connected branches of a directed graph. Specifically, the algorithm can be used to calculate the size of each strongly connected branch in the directed graph, the nodes of each strongly connected branch, and the total number of strongly connected branches.

[0095] (8) Connected subgraph

[0096] In graph theory, any two nodes in a connected subgraph of an undirected graph are connected to each other by a path, and in a hypergraph, the nodes are unconnected.

[0097] (9) Eigenvector centrality

[0098] In graph theory, eigenvector centrality is a way to measure the influence of a node on the network.

[0099] The importance of a node usually depends on the number of its neighbor nodes (i.e., the degree of the node) and the importance of its neighbor nodes. The more important the neighbor nodes connected to it are, the more important the node is.

[0100] For example, a node's importance can be represented by a score, where a higher score indicates a more important node. For nodes with the same number of connections, a node with higher scores for adjacent nodes will have a higher score than a node with lower scores. This principle allows all nodes to be assigned a corresponding score. A high eigenvector score indicates that the node is connected to many nodes that themselves have high scores.

[0101] (10) Neural Network

[0102] A neural network can be composed of neural units, which can be represented by x s The output of the operation unit with the intercept 1 as input can be shown as follows:

[0103]

[0104] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight of , b is the bias of the neural unit.

[0105] f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to transform the input signal of the neural unit into an output signal. The output signal of this activation function can be used as the input of the next layer. For example, the activation function can be ReLU, tanh, or sigmoid function.

[0106] A neural network is formed by connecting many of these single neural units. That is, the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features of the local receptive field, which can be an area composed of several neural units.

[0107] (11) Graph Neural Networks

[0108] Graph neural network refers to a general term for algorithms that use neural networks to learn graph-structured data, extract and discover features and patterns in graph-structured data, and meet the needs of graph learning tasks such as clustering, classification, prediction, segmentation, and generation.

[0109] For graph-structured data, since each node is closely associated with its neighboring nodes, graph neural networks can aggregate the neighborhood information of each node to obtain an embedded representation of each node.

[0110] The attention mechanism allows a neural network to focus only on the information needed for task learning. Introducing the attention mechanism into GNNs forms a graph attention network (GAT). GAT focuses on nodes and edges that are more relevant to the task, which can improve processing performance.

[0111] (12) Relation-aware graph attention network (RGAT)

[0112] RGAT is a graph neural network that can model various relationships in a graph structure through a graph attention mechanism to obtain vector representations of the nodes and relationships in the graph structure in a low-dimensional space. For example, RGAT can be used to process the input knowledge graph to obtain low-dimensional space vectors for each entity and entity relationship in the knowledge graph.

[0113] As mentioned above, the solution of the embodiment of the present application can be applied to natural language processing tasks. Figure 1 (a) shows an application scenario of a natural language processing system. In this scenario, the natural language processing system includes a user device and a data processing device. The user device includes a user and an intelligent terminal such as a mobile phone, a personal computer, or an information processing center. The user device is the initiator of natural language data processing. As the initiator of a request such as a language question and answer or query, the user usually initiates the request through the user device.

[0114] Data processing devices can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. Data processing devices receive queries, voice, or text from smart terminals via interactive interfaces. They then use memory and processors to perform language data processing, including machine learning, deep learning, search, reasoning, and decision-making. Memory is a general term encompassing both local storage and databases storing historical data. Databases can be located on the data processing device or on other network servers.

[0115] Figure 1(b) illustrates another application scenario for a natural language processing system. In this scenario, the user device directly functions as a data processing device, receiving user input and processing it directly through the user device's hardware. The specific process is similar to that of Figure 1(a) and can be found in the description above. This description is omitted here.

[0116] Figure 1(c) shows a schematic diagram of the relevant devices of the natural language processing system provided by an embodiment of the present application. The natural language processing system may include a local device 101, a local device 102, an execution device 110, and a data storage system 150, wherein the local device 101 and the local device 102 are connected to the execution device 110 via a communication network.

[0117] Execution device 110 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, and load balancers. Execution device 110 can be deployed at a single physical site or distributed across multiple physical sites. Execution device 110 can use data in data storage system 150 or invoke program code in data storage system 150 to implement the text processing model training method of the present embodiment.

[0118] It should be noted that the execution device 110 may also be referred to as a cloud device, in which case the execution device 110 may be deployed in the cloud. Alternatively, the execution device 110 may also be a terminal device, in which case the execution device 110 may be deployed on the user terminal side, which is not limited in this embodiment of the present application.

[0119] Users can operate their respective user devices (e.g., local device 101 and local device 102) to interact with execution device 110. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0120] Each user's local device can interact with the execution device 110 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0121] The data storage system 150 may be integrated into the execution device 110 , the local device 101 or the local device 102 , or may be set up on a cloud or other network server.

[0122] In one implementation, the local device 101 or the local device 102 may obtain relevant parameters of the text processing model from the execution device 110 , and the local device 101 or the local device 102 may use the text processing model to obtain the execution result of the text processing task.

[0123] In another implementation, the text processing model can be directly deployed on the execution device 110. The execution device 110 obtains the text to be processed from the local device 101 and the local device 102, and obtains the execution result of the text processing task through the text processing model.

[0124] The user device in Figure 1 (a) and Figure 1 (b) can be the local device 101 or 102 in Figure 1 (c), and the data processing device in Figure 1 (a) and Figure 1 (b) can be the execution device 110 in Figure 1 (c).

[0125] FIG2 illustrates a system architecture 200 provided by an embodiment of the present application. A data acquisition device 260 is used to collect training data and store it in a database 230. A training device 220 generates a target model / rule 201 based on the training data maintained in the database 230, such as a text processing model in an embodiment of the present application. The model in an embodiment of the present application can be a neural network model, or other models. The training data can include training text and target processing results for the training text, such as labels for the training text.

[0126] It should be noted that, in actual applications, the training data maintained in the database 230 may not all be collected by the data acquisition device 260, but may also be received from other devices. It should also be noted that the training device 220 may not train the target model / rule 201 entirely based on the training data maintained by the database 230, but may also obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation on the embodiments of the present application.

[0127] FIG2 shows a functional module diagram of the data processing process. For example, the client device 240 in FIG2 may be the user device in FIG1 . When the data processing capability of the user device in FIG1 is relatively strong, the execution device 210 and the data storage system 250 in FIG2 may be integrated into the user device in FIG1 . In some embodiments, the execution device 210 and the data storage system 250 in FIG2 may also be integrated into the data processing device in FIG1 . The database 230, the training device 220, and the data acquisition device 260 in FIG2 may be integrated into the data processing device in FIG1 , and may be set up on the cloud or other servers on the network.

[0128] Exemplarily, the data acquisition device 260 may be a terminal device, or an input / output interface of a server or cloud, an interaction layer (interface) for obtaining user input and returning processing results.

[0129] The target model / rules obtained by the training device 220 can be applied to different systems or devices. For example, when applied to the execution device 210 shown in FIG2 , the execution device 210 can be a terminal such as a mobile phone terminal, tablet computer, laptop computer, AR / VR, vehicle terminal, etc., or a server or cloud terminal. In FIG2 , the execution device 210 is configured with an I / O interface 212 for data exchange with external devices. A "user" can input data into the I / O interface 212 through a client device 240.

[0130] When the execution device 210 preprocesses the input data, or when the calculation module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call data, code, etc. in the data storage system 250, and can also store data, instructions, etc. in the data storage system 250.

[0131] Finally, the I / O interface 212 returns the processing result to the client device 240 and provides it to the user.

[0132] It is worth noting that the training device 220 can generate corresponding target models / rules 201 based on different data for different goals to provide users with better results.

[0133] In the scenario shown in Figure 2 , the user can manually specify data input into execution device 210, for example, by operating within the interface provided by I / O interface 212. Alternatively, client device 240 can automatically input data into I / O interface 212 and obtain results. If automatic data input by client device 240 requires user authorization, the user can set the corresponding permissions within client device 240. The user can view the results output by execution device 210 on client device 240, which can be presented through a display, sound, action, or other specific means. Client device 240 can also serve as a data acquisition terminal, storing the collected data in database 230.

[0134] It is worth noting that Figure 2 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0135] Figure 3 shows a schematic block diagram of a text processing system provided in an embodiment of the present application. The server in Figure 3 may be deployed with the execution device 210 in Figure 2, the data processing devices in Figures 1(a) and 1(b), or the execution device 110 in Figure 1(c).

[0136] The text processing model in the embodiment of the present application can be implemented by a program code deployed on the hardware of the server. In other words, the text processing model in the embodiment of the present application can be modified and implemented on the basis of an existing software platform. Specifically, the program code runs on the host storage (such as the host memory or disk in Figure 3) and the acceleration hardware (such as GPU, FPGA or dedicated chip) memory of the server. Exemplarily, the dedicated chip can be a neural network operation processor that can be used to perform operations on the neural network model.

[0137] Figure 4 shows a schematic structural diagram of a text processing model provided by an embodiment of the present application. As shown in Figure 4 , the text processing model 400 includes a knowledge extraction module 410 , a text encoding module 420 , a knowledge encoding module 430 , and a task processing module 440 .

[0138] The knowledge extraction module 410 is used to extract knowledge from input data. The input data can be text. During the training process, the input data can be training text, and during the inference process, the input data can be text to be processed.

[0139] The text encoding module 420 is used to encode the text to obtain a character-level encoding of the text.

[0140] The knowledge encoding module 430 is used to process the knowledge extracted by the knowledge extraction module 410 through RGAT to generate a knowledge-level encoding of the text.

[0141] In a possible implementation, the text-level encoding of the text can be used as an input of the knowledge encoding module 430 and participate in the process of the knowledge encoding module 430 generating the knowledge-level encoding.

[0142] During the training process, the output of the knowledge encoding module 430 can be understood as the predictive encoding of the knowledge level of the text.

[0143] During the reasoning process, the output of the knowledge encoding module 430 is the knowledge-level encoding of the text.

[0144] The task processing module 440 is used to output the processing results of the text based on the encoding at the text level and the encoding at the knowledge level.

[0145] For example, the text processing model shown in Figure 4 may be a text classification model for text classification. In this case, the task processing module 440 may be used to output the text classification result, that is, to predict the text category based on the character-level encoding and the knowledge-level encoding.

[0146] The specific process of text processing by the text processing model 400 can refer to the method described below.

[0147] Figure 5 shows a schematic flow chart of a method for training a text processing model provided in an embodiment of the present application. The method 500 shown in Figure 5 can be executed by a device or equipment that can execute the text processing model training process. For example, the device can be a cloud service device or a terminal device, such as a computer, server, vehicle, mobile phone, etc., which has sufficient computing power to execute the text processing model training method, or a system composed of a cloud service device and a terminal device. For example, the method 500 can be executed by any of the execution device 110, the local device, or the training device 220 in Figure 2 in Figure 1.

[0148] For example, the method 500 may be specifically executed by the training device 220 shown in FIG. 2 , and the training data in the method 500 may be the training data maintained in the database 230 shown in FIG. 2 .

[0149] For example, the text processing model in method 500 may be the text processing model shown in Figure 4. The knowledge encoding module in the text processing model may be implemented by the RGAT model, and the training method of the text processing model may also be understood as the training method of the RGAT.

[0150] The method 500 includes steps 510 to 540. Steps 510 to 540 are described below.

[0151] 510, obtain training text.

[0152] 520, obtain the knowledge graph.

[0153] 530. Determine an initial concept graph of the training text based on the knowledge graph, wherein the nodes in the initial concept graph include topic nodes, wherein the topic nodes include candidate entities in the knowledge graph corresponding to target noun phrases in the training text, and the edges between the nodes in the initial concept graph are used to represent entity relationships between the nodes in the initial concept graph.

[0154] At 540, the initial concept graph is input into the RGAT model for training to obtain a target RGAT model. During the training process, the first concept graph in the (i+1)th iteration is determined based on the relevance between the nodes in the second concept graph in the (i)th iteration and the training text, as well as the weights of the edges in the second concept graph, where i is a positive integer. The first concept graph is a subgraph of the initial concept graph, and the second concept graph is a subgraph of the initial concept graph.

[0155] According to the solution of the embodiment of the present application, during the iterative training process of the RGAT model, the concept map can be optimized based on the relevance of the nodes in the concept map to the training text, as well as the importance of the edges in the concept map, and the optimized concept map is used as the concept map used in the next iteration. In this way, it is beneficial to reduce the RGAT model's attention to knowledge with less text relevance, strengthen the model's attention to knowledge with greater text relevance, and thus improve the training effect of the RGAT model, learn more accurate knowledge-level encoding, and thus improve the accuracy of downstream text processing tasks. Knowledge with less text relevance can also be understood as redundant knowledge or ambiguous knowledge. Knowledge with greater text relevance can also be understood as key knowledge.

[0156] Exemplarily, a knowledge graph can be represented as graph-structured data, used to represent entities and their relationships in the real world. Entities can be represented as nodes in a knowledge graph, and entity relationships can be represented as edges in a knowledge graph. In embodiments of the present application, nodes in a knowledge graph may also be referred to as entity nodes in a knowledge graph or entities in a knowledge graph. Edges in a knowledge graph may also be referred to as relationship edges in a knowledge graph or entity relationships in a knowledge graph.

[0157] The knowledge graph can be an existing knowledge graph, or it can be a pre-built knowledge graph, which is not limited in the embodiments of the present application.

[0158] The knowledge graph in step 520 may be a knowledge graph of the professional field to which the training text belongs.

[0159] For example, if the text is data in the medical field, the knowledge graph can be a knowledge graph in the medical field.

[0160] For another example, if the text is data in the financial field, the knowledge graph can be a knowledge graph in the financial field.

[0161] Knowledge graphs can be constructed based on corpora in specialized fields. For example, corpora can include website articles or books. Based on corpora in different specialized fields, knowledge graphs in different specialized fields can be constructed.

[0162] For example, a fact representing the existence of an entity relationship between two entities in a knowledge graph can be represented as a triple data structure.

[0163] It should be understood that this is only an example, and facts in the knowledge graph can also be represented in other forms besides triples, which is not limited in the embodiments of the present application.

[0164] Exemplarily, the process of constructing the initial concept map may be performed by the knowledge encoding module 430 shown in FIG. 4 .

[0165] The initial concept graph can be represented as graph-structured data, representing the knowledge in the text. The initial concept graph can be understood as a subgraph of the knowledge graph. Nodes in the concept graph can also be referred to as entities in the concept graph or entity nodes in the concept graph.

[0166] The topic nodes in the initial concept graph include candidate entities in the knowledge graph that correspond to the target noun phrases in the training text. Alternatively, the topic nodes in the initial concept graph correspond to candidate entities in the knowledge graph that correspond to the target noun phrases in the training text. There can be a one-to-one correspondence between topic nodes and candidate entities.

[0167] Edges in the initial concept graph represent entity relationships between nodes in the initial concept graph. Alternatively, edges in the initial concept graph represent entity relationships between entities in the knowledge graph corresponding to nodes in the initial concept graph. Edges between nodes in the initial concept graph are determined based on entity relationships in the knowledge graph. Edges between nodes in the initial concept graph are formed by connecting nodes in the initial concept graph based on entity relationships in the knowledge graph.

[0168] For example, an edge between nodes A and B in the initial concept graph represents the entity relationship between them. Node A in the initial concept graph corresponds to entity A in the knowledge graph, and node B in the initial concept graph corresponds to entity B in the knowledge graph. The entity relationship between nodes A and B is the entity relationship between entities A and B in the knowledge graph. Entity A in the knowledge graph can also be referred to as node A in the knowledge graph, and node B in the knowledge graph can also be referred to as node B in the knowledge graph. The entity relationship between nodes A and B is the entity relationship between nodes A and B in the knowledge graph.

[0169] A target noun phrase is a noun phrase in the text that corresponds to at least one candidate entity in the knowledge graph. In other words, if a noun phrase in the training text corresponds to at least one candidate entity in the knowledge graph, then that noun phrase can be used as a target noun phrase in the training text.

[0170] The target noun phrases in the text can be obtained based on the knowledge graph.

[0171] Exemplarily, knowledge triples, i.e., knowledge triples of the text, are extracted based on the knowledge graph. The knowledge triples of the text can be in the form of (noun phrase, relation phrase, noun phrase). The noun phrases in the knowledge triples are phrases in the text data. The relation phrases in the knowledge triples are phrases in the text data. The noun phrases in the knowledge triples correspond to at least one candidate entity in the knowledge graph. The relation phrases in the knowledge triples can correspond to at least one entity relationship in the knowledge graph. A noun phrase can include one word or multiple words. A relation phrase can include one word or multiple words. The noun phrases in the knowledge triples are the target noun phrases. In other words, among the noun phrases in the text data, the noun phrases that have corresponding candidate entities in the knowledge graph can be used as the target noun phrases. The relation phrases in the knowledge triples can be used as the target relation phrases. In other words, among the relation phrases in the text data, the relation phrases that have corresponding entity relationships in the knowledge graph can be used as the target relation phrases. Extracting knowledge triples from text data can also be understood as identifying noun phrases and relation phrases that constitute knowledge triples in text data.

[0172] Extracting knowledge triples from text based on knowledge graphs can also be called extracting knowledge from text based on knowledge graphs.

[0173] Exemplarily, the process of extracting knowledge may be performed by the knowledge extraction module 410 in FIG. 4 .

[0174] Figure 6 shows a schematic diagram of knowledge extraction. For example, as shown in Figure 6, knowledge triples are identified in text data based on the knowledge graph in the medical field. For example, given the text "Vitamin B12 creates risk for anemia," the knowledge triple extracted from the text is (Vitamin B12, creates risk for (+), anemia), where the noun phrases are "Vitamin B12" and "anemia," and the relation phrase is "creates risk for (+)."

[0175] It should be understood that FIG6 only takes the extraction of one knowledge triple from the text as an example for illustration. In actual applications, there may be multiple knowledge triples identified from the text, and this embodiment of the present application does not limit this.

[0176] It should be noted that the entity relationship corresponding to the relational phrase in the knowledge triple in the knowledge graph is not necessarily the entity relationship between the candidate entities corresponding to the noun phrase in the knowledge triple in the knowledge graph. Taking Figure 6 as an example, the knowledge triple extracted from the text is (Vitamin B12, creates risk for (+), anemia), where the noun phrases "Vitamin B12" and "anemia" both have corresponding candidate entities in the knowledge graph, and the relational phrase "creates risk for (+)" has a corresponding entity relationship in the knowledge graph. However, in the knowledge graph, the entity relationship between the candidate entities corresponding to "Vitamin B12" and "anemia" in the knowledge graph is not necessarily "creates risk for (+)".

[0177] The topic node may include candidate entities corresponding to all target noun phrases in the text data in the knowledge graph. In other words, the topic node may include candidate entities corresponding to noun phrases in all knowledge triples extracted from the text data.

[0178] Optionally, the topic node includes all candidate entities corresponding to the target noun phrase in the knowledge graph.

[0179] A target noun phrase may correspond to one or more candidate entities in the knowledge graph.

[0180] For example, the target noun phrase "anemia" can correspond to multiple candidate entities such as "anemia (disease)", "anemia (symptom)" or "anemia (plant)".

[0181] In this approach, the topic node can include all candidate entities corresponding to each target noun phrase in the knowledge graph. The resulting concept graph encompasses all candidate entities and their corresponding entity relationships related to the text data. This provides comprehensive and complete knowledge related to the text for subsequent processing, avoiding incorrect reasoning paths caused by missing knowledge.

[0182] Optionally, the nodes in the initial concept graph also include neighbor nodes, and the neighbor nodes include neighbor entities of the candidate entities corresponding to the target noun phrase in the knowledge graph.

[0183] In other words, the neighbor nodes in the initial concept graph correspond to the neighbor entities of the candidate entity corresponding to the target noun phrase in the knowledge graph. There can be a one-to-one correspondence between neighbor nodes.

[0184] The neighbor nodes may include k-hop neighbors of the candidate entity corresponding to the target noun phrase in the knowledge graph. k is a positive integer. For example, k may be a positive integer less than or equal to 3.

[0185] Exemplarily, the neighbor nodes may include k-hop neighbors of all candidate entities corresponding to the target noun phrase in the knowledge graph.

[0186] Neighbor nodes play an important role in knowledge reasoning. The initial concept graph of the embodiment of the present application also includes neighbor entities connected to the candidate entity (i.e., the subject node) in the knowledge graph, i.e., neighbor nodes, which can further provide more comprehensive and complete knowledge and help improve the accuracy of knowledge-level encoding.

[0187] Furthermore, the initial concept graph may also include the entity relationships corresponding to the target relationship phrases in the knowledge graph.

[0188] Figure 7 shows a schematic diagram of an initial concept graph. As shown in Figure 7, a concept graph corresponding to the text data is constructed based on the medical field knowledge graph and knowledge triples extracted from the text data. The initial concept graph shown in Figure 7 is an initial concept graph corresponding to the text shown in Figure 6. The topic nodes "anemia (disease)", "anemia (symptom)", and "anemia (plant)" in Figure 7 are all candidate entities in the knowledge graph corresponding to the target noun phrase "anemia" in the text shown in Figure 6. The topic node "Vitamin B 12 (chemical)" in Figure 7 is all candidate entities in the knowledge graph corresponding to the target noun phrase "Vitamin B 12" in the text shown in Figure 6. The neighbor nodes in Figure 7 are the one-hop neighbor entities of the above candidate entities in the knowledge graph. "Vitamin B12 (chemical)", "anemia (disease)", "anemia (symptom)", and "anemia (plant)" are the topic nodes in the concept graph. "Hemoglobin (biological substance)", "GI bleeding (biologic function)" and "plant (type)" are neighbor nodes in the concept graph. The edges between the nodes in Figure 7 are used to represent the entity relationships between the corresponding entities in the knowledge graph.

[0189] Exemplarily, the relevance between a node in the initial concept graph and the training text may be determined based on the node's importance. For example, the relevance between a node in the initial concept graph and the training text may be the node's importance. Alternatively, the relevance between a node in the initial concept graph and the training text may be positively correlated with the node's importance, i.e., the greater the node's importance, the higher the relevance between the node and the training text.

[0190] For example, the importance of a node can be determined by a centrality measurement method, such as the PageRank algorithm, degree centrality, etc. However, these methods have certain limitations and cannot accurately represent the importance of a node.

[0191] Exemplarily, the importance of a node may be determined by the eigenvector centrality of the node. For example, the importance of a node may be the eigenvector centrality of the node. Alternatively, the importance of a node may be positively correlated with the eigenvector centrality of the node.

[0192] Optionally, the relevance between each topic node in the initial concept map and the training text may be the feature vector centrality of each topic node.

[0193] Furthermore, the relevance of each topic node in the initial concept graph to the training text can be the eigenvector centrality of each topic node on the topic correlation graph. The nodes in the topic correlation graph are all topic nodes. The eigenvector centrality of each topic node is determined based on the initial importance of each topic node and the weight of the edge in the topic correlation graph. The initial importance of the node in the topic correlation graph is set based on the probability of the node appearing in the facts recorded in the knowledge graph. The edges in the topic correlation graph are obtained by connecting the entity relationships between the candidate entities in the knowledge graph. The weight of the edge in the topic correlation graph is determined based on the number of entity relationships between the candidate entities corresponding to the two topic nodes connected by the edge in the knowledge graph, that is, the number of edges between the candidate entities.

[0194] For example, nodes C and D in the topic-related graph are candidate entities C and D in the knowledge graph, respectively. If there are n edges between candidate entities C and D in the knowledge graph, then an edge with a weight of n is constructed between nodes C and D in the topic-related graph. n is a positive integer.

[0195] FIG8 shows a topic-related graph provided in an embodiment of the present application, in which the nodes are the topic nodes in FIG7 .

[0196] In an embodiment of the present application, eigenvector centrality is used to calculate the relevance of the topic node and the training text, and the importance of a node is determined based on the importance of its neighboring nodes. The initial importance of the node is set based on the probability of the node appearing in the facts recorded in the knowledge graph, which can more accurately reflect the importance of the node.

[0197] Furthermore, the relevance of each neighbor node in the initial concept graph to the training text can be the score of the strongly connected branch where each neighbor node is located on the information diffusion graph corresponding to the initial concept graph. The nodes in the information diffusion graph corresponding to the initial concept graph are the nodes in the initial concept graph. When a first node in the initial concept graph is a 1-hop neighbor of a second node, there is a directed edge from the second node to the first node in the information diffusion graph.

[0198] Figure 9 shows an information propagation diagram provided in an embodiment of the present application. The information propagation diagram shown in Figure 9 is the information propagation diagram corresponding to the initial concept map shown in Figure 7, or the information propagation diagram corresponding to the training text shown in Figure 6.

[0199] Each concept graph corresponds to an information diffusion graph. An information diffusion graph is a directed graph. Edges in a directed graph are directed edges, meaning they have directionality. The nodes in the information diffusion graph corresponding to a concept graph are the nodes in the concept graph. A directed edge from node v to node u in the information diffusion graph is constructed if and only if node u in the concept graph is a one-hop neighbor of node v. Node u can be the first node, and node v can be the second node.

[0200] For example, the scores of each strongly connected branch on the information propagation graph can be obtained by propagating the initial score of the strongly connected branch where the subject node is located to the downstream strongly connected branches according to the topological sorting. The initial scores of each strongly connected branch on the information propagation graph can be determined based on the maximum importance of the nodes in each strongly connected branch. For example, the initial scores of each strongly connected branch on the information propagation graph can be the maximum importance of the nodes in each strongly connected branch.

[0201] Topological sorting can be achieved by performing a depth-first search on strongly connected branches. The topological sorting result can be understood as a depth-first search of strongly connected branches. "Propagation" can be understood as transferring the scores of the previous strongly connected branches to the subsequent strongly connected scores based on the topological sorting, or in other words, updating the scores of the downstream strongly connected branches to the scores of the upstream strongly connected branches.

[0202] For example, the topological sorting can be {C1, C2, C3}, where C1, C2, and C3 represent three strongly connected branches. "Propagation" can be understood as updating the score of C2 to the score of C1 and updating the score of C3 to the score of C2.

[0203] In step 540 , the RGAT is trained to learn the knowledge encoding expression corresponding to the training text.

[0204] Exemplarily, the training process of RGAT may be performed according to the following process.

[0205] 1) Input the concept map into RGAT for processing to obtain the predicted knowledge encoding of the training text output by RGAT.

[0206] 2) Determine the prediction result of the training text based on the text encoding and prediction knowledge encoding of the training text.

[0207] 3) Adjust the parameters of RGAT based on the prediction results of the training text.

[0208] 4) Using the adjusted RGAT as the RGAT in step 1), repeat steps 1) to 3) for iterative training. In other words, the RGAT in the next iteration is the adjusted RGAT in the current iteration.

[0209] Steps 1) to 3) can be considered as an iterative process.

[0210] Exemplarily, step 1) may be performed by the knowledge encoding module 430 in FIG. 4 .

[0211] The knowledge encoding of the training text can also be called the knowledge-level encoding of the training text. The knowledge encoding can be expressed as a knowledge embedding vector or a knowledge feature vector.

[0212] Exemplarily, the knowledge encoding of the training text may include encoding of nodes in the concept graph and encoding of edges in the concept graph.

[0213] For example, the knowledge encoding of the training text may include the embedding vectors of the nodes and the embedding vectors of the edges in the concept graph.

[0214] In step 2), the text encoding of the training text can also be called the character-level encoding of the training text. The character-level encoding refers to the low-dimensional space vector used to express the text content and character sequence in the text data.

[0215] Text encoding can be represented as a text embedding vector or a text feature vector.

[0216] Exemplarily, the text encoding of the training text may include the text encoding of the training text sequence and the text encoding of the target phrase in the training text. The target phrase may include a target noun phrase and a target relational phrase. The text encoding of the training text sequence may also be referred to as the text encoding of the training text itself.

[0217] Specifically, the training text may be processed by a pre-trained language model to obtain a text encoding of the training text.

[0218] For example, the training text can be processed by methods such as a transformer-based bidirectional encoder representation from transformers (BERT) model, a bidirectional gating recurrent unit (BiGRU), or a bidirectional long short-term memory (BiLSTM) model to obtain a text encoding of the training text.

[0219] Exemplarily, the process of obtaining the text encoding of the training text may be performed by the text encoding module 420 in FIG. 4 .

[0220] Exemplarily, step 2) may be performed by the task processing module 440 in FIG. 4 .

[0221] The type of prediction results for the training text is related to the type of text processing task, that is, the type of downstream task.

[0222] Exemplarily, method 500 is used for a text classification task. In this case, the prediction result of the training text can be the predicted category of the training text. For example, the text encoding and the predictive knowledge encoding of the training text are input into a classifier to obtain the predicted category of the training text. In another example, the text encoding and the predictive knowledge encoding of the training text are fused, and the fused result is input into a classifier to obtain the predicted category of the training text. The fusion method can be to concatenate the text encoding and the predictive knowledge encoding of the training text into vectors to obtain a text fusion encoding. The classifier can be a softmax function.

[0223] In step 3), the parameters of RGAT can be adjusted with the goal of reducing the gap between the target processing result of the training text and the prediction result.

[0224] Taking the text classification task as an example, in step 3), the RGAT parameters can be adjusted to reduce the gap between the training text's label and its predicted category. The training text's label is the target processing result. The training text's label indicates the true value of the category corresponding to the training text, that is, the actual category of the training text.

[0225] It should be understood that the above is only an example, and method 500 can also be used for other natural language processing tasks, such as multi-hop reasoning question answering tasks. This embodiment of the present application does not limit this.

[0226] In the first iteration process, the concept map in step 1) can be the initial concept map.

[0227] The first concept graph in the (i+1)th iteration can be understood as the concept graph input to the RGAT during the (i+1)th iteration. In other words, during the (i+1)th iteration, the RGAT performs forward propagation and backward propagation based on the first concept graph.

[0228] The second concept graph in the i-th iteration can be understood as the concept graph input to RGAT during the i-th iteration. During the i-th iteration, RGAT performs forward propagation and backward propagation based on the second concept graph.

[0229] The i-th iteration can be any iteration in the RGAT training process.

[0230] It should be noted that the “first” in the “first concept map” and the “second” in the second concept map are only used to distinguish the input data of RGAT in the two iterative processes and have no other limiting effect.

[0231] In the embodiment of the present application, the concept map can be continuously optimized during the RGAT iterative training process, and the concept map in each iteration process can be different. Taking the above training process as an example, the concept map in step 1) can be different in different iteration processes.

[0232] For example, in the process of RGAT learning the expression of concept graph, that is, the training process of RGAT, the concept graph in each round of iteration can be determined based on the relevance of the nodes in the concept graph in the previous round of iteration to the training text and the weight of the edges in the concept graph in the previous round of iteration.

[0233] The concept graphs in the iterative process are all subgraphs of the initial concept graph. The relevance between a node in the iterative concept graph and the training text is the same as the relevance between the same node in the initial concept graph and the training text.

[0234] During the iterative training process of RGAT, the concept graph is optimized. The optimization direction can be understood as pruning nodes with less relevance to the training text and / or edges with smaller weights, and retaining nodes with greater relevance to the training text and / or edges with greater weights.

[0235] The weight of the edge in the concept graph can also be called the weight of the fact in the concept graph, that is, the attention weight.

[0236] In one possible implementation, the first concept graph belongs to a set of first subgraphs of the initial concept graph, the first cost of the first subgraph is less than or equal to a threshold, the benefit of the first concept graph is greater than or equal to the benefits of other first subgraphs in the set of first subgraphs, the first cost of the first subgraph is determined based on the first cost of the edge within the first subgraph, the benefit of the first concept graph is determined based on the benefit of the edge within the first concept graph, the benefit of the first subgraph is determined based on the benefit of the edge within the first subgraph, the benefit of the edge is positively correlated with the weight of the edge in the i-th iteration, and the first cost of the edge is negatively correlated with the relevance of the two nodes connected by the edge to the training text.

[0237] Among the subgraphs of the initial concept graph, any subgraph whose first cost is less than or equal to the threshold can be referred to as a first subgraph. The set of first subgraphs is the set of subgraphs whose first cost is less than or equal to the threshold. A first concept graph is an element in this set. In other words, a first concept graph is a first subgraph.

[0238] In other words, in the set of subgraphs whose first consumption of the initial concept graph is less than or equal to the threshold, the first subgraph with the largest benefit is selected as the first concept graph. In other words, the first concept graph can be obtained by selecting a subgraph of the initial concept graph and maximizing the benefit of the subgraph when the first consumption of the subgraph is less than or equal to the threshold.

[0239] The higher the relevance between the two nodes connected by an edge and the training text, the smaller the first cost of the edge. The lower the relevance between the two nodes connected by an edge and the training text, the larger the first cost of the edge. For example, the higher the average relevance between the two nodes connected by an edge and the training text, the smaller the first cost of the edge. For another example, the lower the average relevance between the two nodes connected by an edge and the training text, the larger the first cost of the edge.

[0240] For example, the first cost of a subgraph may be determined based on the first costs of all edges within the subgraph. For example, the first cost of a subgraph may be the sum of the first costs of all edges within the subgraph. For another example, the first cost of a subgraph may be the average of the first costs of all edges within the subgraph.

[0241] The greater the weight of an edge in the i-th iteration, the greater the benefit of the edge. The smaller the weight of an edge in the i-th iteration, the smaller the benefit of the edge.

[0242] For example, the revenue of a subgraph can be determined based on the revenue of all edges within the subgraph. For example, the revenue of a subgraph can be the sum of the revenues of all edges within the subgraph. For another example, the revenue of a subgraph can be the average of the revenues of all edges within the subgraph.

[0243] For example, the first concept graph in the (i+1)th iteration may be obtained by optimizing the second concept graph in the (i)th iteration, and the optimized concept graph may be a subgraph of the concept graph in the (i)th iteration. In other words, the first concept graph may be a subgraph of the second concept graph.

[0244] Specifically, the second concept graph is optimized according to the relevance of the nodes in the second concept graph to the training text and the weights of the edges in the second concept graph in the i-th iteration to obtain the first concept graph in the i+1-th iteration.

[0245] Optionally, the ratio of the benefit of the first edge to the first cost of the first edge is less than or equal to the ratio of the benefit of the second edge to the first cost of the second edge. The first edge belongs to the second concept graph, and the first edge does not belong to the first concept graph, and the second edge belongs to the first concept graph. The benefit of the edge is positively correlated with the weight of the edge in the i-th iteration. The first cost of the edge is negatively correlated with the relevance of the two nodes connected by the edge to the training text. The first cost of the first concept graph is less than or equal to a threshold. The sum of the first cost of the first concept graph and the first cost of the first edge is greater than a threshold.

[0246] The second side belongs to the second concept map, and the second side also belongs to the first concept map.

[0247] The second edge is any edge in the first concept graph. The first edge is any edge in the second concept graph that is not part of the first concept graph. In other words, the ratio between the benefit and the first cost of any edge in the second concept graph that is not part of the first concept graph is less than or equal to the ratio between the benefit and the first cost of any edge in the first concept graph.

[0248] The greater the edge weight in the i-th iteration, the greater the benefit of the edge. The higher the relevance between the two nodes connected by the edge and the training text, the smaller the first cost of the edge. For example, the higher the average relevance between the two nodes connected by the edge and the training text, the smaller the first cost of the edge.

[0249] Exemplarily, the first consumption of the first concept graph is less than or equal to a threshold value, which may be that the sum of the first consumptions of all edges in the first concept graph is less than or equal to the threshold value.

[0250] Alternatively, the first consumption of the first concept graph is less than or equal to the threshold, which may be that the average value of the first consumption of all edges in the first concept graph is less than or equal to the threshold.

[0251] Illustratively, before the start of the (i+1)th iteration, the first concept map in the (i+1)th iteration may be determined by the following steps.

[0252] S11, obtaining the first consumption and benefit of the edge in the second concept graph in the i-th iteration.

[0253] S12: Select edges in the second concept graph as edges in the first concept graph in descending order of the ratio of the benefit to the first cost of the edges, until the sum of the first costs of the selected edges exceeds a threshold. The first cost of the first concept graph is less than or equal to the threshold.

[0254] In other words, the ratio of the benefit to the first cost of any edge of the first concept graph is greater than or equal to the ratio of the benefit to the first cost of any edge in the first concept graph that does not belong to the second concept graph.

[0255] Exemplarily, the threshold may be a tolerance for the number of uncertain edges. The threshold W may be determined based on a tolerance ratio θ and the number N of uncertain edges in the initial concept graph. For example, the tolerance threshold may be W = θN. 0 < θ < 1. An uncertain edge refers to an edge in the initial concept graph with a weight less than 1. N is less than or equal to the number of edges in the initial concept graph. N is a positive integer.

[0256] For example, if the number N of uncertain edges in the initial concept graph is 60 and the tolerance ratio θ is 0.5, then the threshold W is 30. In this case, step 2) can be understood as selecting edges from the second concept graph as edges in the first concept graph in descending order of the ratio between the benefit and the first cost of the edge, until the total first cost of the selected edges reaches 30. The first cost of an edge can be determined based on the average relevance between the two nodes connected by the edge and the training text.

[0257] It should be understood that the above is only an example, and the concept map can also be optimized in other ways to retain the edges with larger weights and nodes with greater relevance to the training text in the concept map, so as to reduce the model's attention to knowledge with little relevance to the text and strengthen the model's attention to knowledge with high relevance to the text.

[0258] In one possible implementation, the first concept graph is determined based on a first subset of a set of connected subgraphs of the initial concept graph, and the second cost of the first subset is less than or equal to the second cost of the second subset. The second cost of the first subset is determined based on the second cost of edges within the connected subgraph of the first subset. The second cost of the second subset is determined based on the second cost of edges within the connected subgraph of the second subset. The second cost of an edge is negatively correlated with the relevance of the two nodes connected by the edge to the training text, and the second cost of an edge is negatively correlated with the weight of the edge during the i-th iteration.

[0259] Exemplarily, the first subset includes at least one candidate entity corresponding to the target noun phrase, and the second subset includes at least one candidate entity corresponding to the target noun phrase.

[0260] A subset of the set of connected subgraphs includes at least one candidate entity corresponding to a target noun phrase. This means that the at least one candidate entity corresponding to the target noun phrase exists on at least one connected subgraph within the subset. Candidate entities corresponding to different target noun phrases may exist on different connected subgraphs within the subset, or may exist on the same connected subgraph within the subset.

[0261] In other words, among the subsets of the set of connected subgraphs of the initial concept graph, the subset that contains at least one candidate entity corresponding to all target noun phrases and has the lowest second cost is used as the first subset. In other words, a subset of the set of connected subgraphs of the initial concept graph is selected such that all target noun phrases have at least one corresponding candidate entity in the subset, and the second cost of the subset is the lowest among the subsets containing at least one candidate entity corresponding to all target noun phrases, thus obtaining the first subset, or in other words, the first concept graph.

[0262] Exemplarily, the first subset includes at least one entity relationship corresponding to the target relationship phrase, and the second subset includes at least one entity relationship corresponding to the target relationship phrase.

[0263] A subset of the set of connected subgraphs includes at least one entity relationship corresponding to a target relationship phrase. This means that the at least one entity relationship corresponding to the target relationship phrase exists on at least one connected subgraph within the subset. Entity relationships corresponding to different target relationship phrases may exist on different connected subgraphs within the subset, or may exist on the same connected subgraph within the subset.

[0264] In other words, among the subsets of the connected subgraphs of the initial concept graph, the subset that contains at least one entity relationship corresponding to all target relationship phrases and has the lowest second cost is used as the first subset. Alternatively, a subset of the connected subgraphs of the initial concept graph is selected such that all target relationship phrases have at least one corresponding entity relationship in the subset, and the second cost of the subset is the lowest among the subsets containing at least one entity relationship corresponding to all target relationship phrases. This yields the first subset, or in other words, the first concept graph.

[0265] Exemplarily, the first subset includes at least one candidate entity corresponding to the target noun phrase and at least one entity relationship corresponding to the target relationship phrase. The second subset includes at least one candidate entity corresponding to the target noun phrase and at least one entity relationship corresponding to the target relationship phrase.

[0266] In other words, among the subsets of the connected subgraphs of the initial concept graph, the subset that contains at least one candidate entity corresponding to all target noun phrases and at least one entity relationship corresponding to all target relationship phrases, and has the smallest second cost, is used as the first subset. In other words, a subset of the connected subgraphs of the initial concept graph is selected, such that all target relationship phrases have at least one corresponding entity relationship in the subset, all target noun phrases have at least one corresponding candidate entity in the subset, and the second cost of the subset is the smallest among the subsets that contain at least one entity relationship corresponding to all target relationship phrases and at least one candidate entity corresponding to all target noun phrases, thereby obtaining the first subset, or in other words, the first concept graph.

[0267] The higher the relevance between the two nodes connected by an edge and the training text, the greater the weight of the edge during the i-th iteration, and the smaller the second cost of the edge. The lower the relevance between the two nodes connected by an edge and the training text, the smaller the weight of the edge during the i-th iteration, and the greater the second cost of the edge. For example, the higher the average relevance between the two nodes connected by an edge and the training text, the smaller the second cost of the edge. For another example, the lower the average relevance between the two nodes connected by an edge and the training text, the greater the second cost of the edge.

[0268] For example, the second consumption of the first subset may be the sum of the second consumptions of all edges in all connected subgraphs in the first subset. The second consumption of the second subset may be the sum of the second consumptions of all edges in all connected subgraphs in the second subset.

[0269] As mentioned above, the first concept graph can be a subgraph of the second concept graph.

[0270] Optionally, the second cost of the first connected subgraph is greater than or equal to the second cost of the second connected subgraph, the first connected subgraph belongs to the second concept graph in the i-th iteration, and the first connected subgraph does not belong to the first concept graph in the i+1-th iteration, and the second connected subgraph is the connected subgraph with the largest second cost within the first concept graph. The second cost of the first connected subgraph is determined based on the second cost of the edges within the first connected subgraph, and the second cost of the second connected subgraph is determined based on the second cost of the edges within the second connected subgraph. The second cost of the edges is negatively correlated with the weight of the edges in the i-th iteration, and the second cost of the edges is negatively correlated with the relevance of the two nodes connected by the edges to the training text. The nodes of the first concept graph include at least one candidate entity corresponding to the target noun phrase, the nodes of the second connected subgraph include at least one candidate entity corresponding to the first noun phrase in the target noun phrase, and the nodes of other connected subgraphs in the first concept graph do not include the at least one candidate entity corresponding to the first noun phrase.

[0271] The second connected subgraph belongs to the first concept graph, and correspondingly, the second connected subgraph belongs to the second concept graph.

[0272] The first connected subgraph is any connected subgraph in the second concept graph that does not belong to the first concept graph. In other words, the second cost of any connected subgraph in the second concept graph that does not belong to the second concept graph is greater than or equal to the second cost of any connected subgraph in the second concept graph.

[0273] Exemplarily, the second consumption of a connected subgraph may be the sum of the second consumptions of all edges in the connected subgraph.

[0274] Alternatively, the second cost of a connected subgraph is an average of the second costs of all edges in the connected subgraph.

[0275] Illustratively, before the start of the (i+1)th iteration, the concept map in the (i+1)th iteration may be determined by the following steps.

[0276] S21, obtaining the second consumption of the connected subgraph in the second concept graph in the i-th iteration.

[0277] S22, selecting connected subgraphs in the second concept graph as connected subgraphs of the first concept graph in ascending order of the second consumption of the connected subgraphs, until all selected connected subgraphs include at least one candidate entity corresponding to the target noun phrase of the training text.

[0278] For example, the initial subset is set to an empty set, and connected subgraphs in the second concept graph are selected in ascending order of the second consumption of the connected subgraphs and added to the subset until the current subset includes at least one candidate entity corresponding to the target noun phrase in the training text. The first concept graph can be determined based on the current subset. The second connected subgraph is the last connected subgraph added to the subset. The first connected subgraph can be any connected subgraph of the second concept graph that has not been added to the subset.

[0279] It should be understood that the above is only an example. For example, the "at least one candidate entity corresponding to the target noun phrase" in step S22 can be replaced with "at least one entity relationship corresponding to the target relationship phrase".

[0280] The target RGAT is the trained RGAT. It can be used to derive feature vectors for the graph data fed into the target RGAT. The concept graph of the text to be processed is fed into the target RGAT, and the output data serves as an embedded representation of the concept graph of the text to be processed, i.e., an encoding of the knowledge level of the text to be processed.

[0281] According to the solution of the embodiment of the present application, in the process of RGAT learning expression through iterative messages between entity nodes, that is, in the training process, the concept map is optimized according to the relevance of the entity node to the text and the weight of the edge. For example, nodes with less relevance to the training text and / or edges with smaller weights are pruned, which is beneficial to reducing the model's attention to knowledge with less relevance to the text and enhancing its attention to knowledge with greater relevance to the text, thereby improving the expressive ability of RGAT, and further helping to improve the accuracy of downstream tasks.

[0282] In addition, in the solution of the embodiment of the present application, the nodes in the initial concept graph can include all candidate entities corresponding to the target noun phrase, which is conducive to ensuring that no text-related knowledge is missed in the concept graph, ensuring the integrity of the knowledge in the concept graph, and using the solution of the embodiment of the present application to learn the knowledge-level expression of the text, further ensuring the accuracy of downstream tasks and avoiding incorrect reasoning paths due to missing some knowledge.

[0283] In addition, in the solution of the embodiment of the present application, the nodes in the initial concept graph can also include k-hop neighbor entities of each candidate entity, which can further improve the completeness of the knowledge in the concept graph.

[0284] FIG10 illustrates a method 800 for training a text processing model provided in an embodiment of the present application. The text processing model may be the text processing model shown in FIG4 . Method 800 may be considered a specific implementation of method 500. For the sake of brevity, some descriptions of method 800 are omitted.

[0285] Method 800 includes steps 810 to 850 .

[0286] 810, extracting knowledge from the training text.

[0287] Exemplarily, step 810 may be performed by the knowledge extraction module 410 in FIG. 4 .

[0288] For example, the knowledge graph is a knowledge graph of a business field related to text data. For ease of understanding and description, the following text uses the business field of medicine as an example to illustrate method 800, without limiting the solution of the embodiment of the present application.

[0289] As shown in FIG8 , step 810 may include: identifying knowledge triples T in the training text d based on the knowledge graph d , to get the noun phrase M in the knowledge triple d and the relative phrase P d , namely the target noun phrases and target relation phrases in the training text, as shown in Figure 6.

[0290] 820, generating text-level encoding of training text through a natural language pre-training model.

[0291] Exemplarily, step 820 may be performed by the text encoding module 420 in FIG. 4 .

[0292] The text-level encoding of the training text may include each noun phrase M in the training text d d The text-level encoding of each relational phrase P d The text-level encoding of the training text sequence is performed.

[0293] 830, the extracted knowledge is processed through RGAT to generate knowledge-level predictive coding of the training text.

[0294] Exemplarily, step 830 may be performed by the knowledge encoding module 430 in FIG. 4 .

[0295] In one possible implementation, the text-level encoding of the training text can be used as an input to RGAT to participate in the process of generating knowledge-level predictive encoding.

[0296] 840, a prediction result is obtained based on the encoding at the text level and the predictive encoding at the knowledge level.

[0297] Exemplarily, step 840 may be performed by the task processing module 440 in FIG. 4 .

[0298] For example, if the downstream task is a text classification task, step 840 may include concatenating the text-level encoding and the knowledge-level predictive encoding into vectors to obtain a predictive text fusion encoding. The predictive text fusion encoding is then input into a classifier to obtain a predictive classification result for the training text.

[0299] 850, iteratively train RGAT based on the prediction result.

[0300] Before iterative training, that is, before step 830 , an initial concept graph of the training text may be constructed, and the relevance between the nodes in the initial concept graph and the training text may be calculated.

[0301] For example, each noun phrase M in the training text is located from the knowledge graph d The corresponding candidate entities are identified, and the k-hop neighboring entities of each candidate entity are located as nodes of the initial concept graph. The nodes in the initial concept graph are connected according to the entity relationships between each pair of entities recorded in the knowledge graph to obtain the initial concept graph. This initial concept graph contains the complete knowledge.

[0302] Specifically, the following steps can be used to construct an initial concept graph and calculate the relevance between the nodes in the initial concept graph and the training text.

[0303] (1) Locating the topic node

[0304] For each noun phrase M obtained in step 810 d , locate all candidate entities corresponding to the noun phrase in the knowledge graph as topic nodes.

[0305] The topic nodes are used as nodes in the topic-related graph. Based on the entity relationships between candidate entities recorded in the knowledge graph, the corresponding topic nodes in the topic-related graph are connected to obtain edges in the topic-related graph. Specifically, for a pair of nodes in the topic-related graph, if there are n edges between the candidate entities corresponding to the pair of nodes in the knowledge graph, then the weight of the edge between the pair of topic nodes in the topic-related graph is n.

[0306] (2) Calculate the relevance between the topic node and the training text

[0307] The initial relevance of each topic node is set based on the probability of the topic node appearing in the facts recorded in the knowledge graph.

[0308] The eigenvector centrality of each topic node is calculated on the topic correlation graph to obtain the importance of each topic node. The importance of each topic node is used as the relevance between the topic node and the training text.

[0309] (3) Locating neighbor nodes

[0310] For each topic node, locate the k-hop neighbor entities of the topic node in the knowledge graph as neighbor nodes.

[0311] Construct an information propagation graph based on the progressive relationship of the hop layers.

[0312] (4) Locating strongly connected branches

[0313] Locate all strongly connected branches on the information diffusion graph and calculate the initial score of each strongly connected branch based on the maximum importance of the nodes in each strongly connected branch. The time complexity is O(V,E). Where V represents the set of nodes on the information diffusion graph and E represents the set of edges on the information diffusion graph.

[0314] (5) Calculate the correlation between neighbor nodes and training text

[0315] According to the topological sorting result, the initial score of the strongly connected branch where the subject node is located is propagated to the downstream strongly connected branches along the topological sorting result, thereby updating the score of each strongly connected branch.

[0316] The score of the updated strongly connected branch of each neighbor node is used as the relevance between the neighbor node and the training text.

[0317] (6) Constructing a concept map

[0318] The topic node and neighbor nodes are used as nodes in the initial concept graph. The corresponding nodes are connected according to the entity relationships recorded in the knowledge graph to obtain the initial concept graph, as shown in Figure 7.

[0319] During the iterative training process, that is, in step 850 , the concept graph is optimized.

[0320] For example, in the process of learning the expression of nodes through iterative message passing between nodes based on RGAT, before each round of iteration, RGAT's attention to knowledge with little relevance to the text can be reduced, and RGAT's attention to knowledge with high relevance to the text can be strengthened based on the relevance of the node to the training text and the weight of the edge in the forward propagation of RGAT.

[0321] The following example 1 or example 2 is used to illustrate the optimization method of the concept map during the iteration process.

[0322] Example 1

[0323] Define the optimization problem to be solved: select a subgraph of the initial concept graph such that the sum of the first costs of all edges in the subgraph is less than or equal to a threshold, and the sum of the benefits of all edges in the subgraph is maximized. The solution to this optimization problem is the optimized concept graph.

[0324] (1) Before each iteration, obtain the first cost and benefit of each edge in the current concept graph.

[0325] For example, the lower the average value of the relevance between two nodes connected to an edge and the training text, the greater the first cost of the edge.

[0326] For example, in the last round of iterations, the higher the weight of an edge is, the greater the benefit of the edge is.

[0327] (2) The optimized concept map is obtained by using the approximate algorithm for solving the above optimization problem.

[0328] Specifically, the edges with the highest ratio of benefit to first cost in the current concept graph are prioritized, and the optimization process continues until the sum of the first costs of the selected edges reaches a threshold. This results in an optimized solution, an optimized concept graph, in linear time complexity (|E| * log|E|). This optimized concept graph serves as the concept graph for the next iteration.

[0329] Example 2

[0330] Define the optimization problem to be solved: select a subset of the set of connected subgraphs of the initial concept graph so that each noun phrase M in the training text d or relative phrase P d , there is at least one corresponding node or edge included in the subset, and the first cost of the connected subgraph in the subset is the smallest among all subset selection methods. The solution to this optimization problem is the optimized concept graph.

[0331] (1) Before each iteration, obtain the second cost of each edge in the current concept graph.

[0332] The second cost of each connected subgraph in the concept graph is calculated. Exemplarily, the second cost of the connected subgraph is the average of the second costs of all edges in the connected subgraph.

[0333] Exemplarily, the lower the average relevance between the two nodes connected to an edge and the training text, and the lower the weight of the edge in the previous iteration, the greater the second consumption of the edge.

[0334] (2) The optimized concept map is obtained by using the approximate algorithm for solving the above optimization problem.

[0335] Specifically, the initial state of the subset is the empty set state, and the connected subgraph with the second smallest consumption in the current concept graph is added to the subset first, until each noun phrase M in the training text is d or relative phrase P d, there is at least one corresponding node or edge included in the subset. This allows us to obtain an approximate solution to the optimization problem, i.e., an optimized concept graph, in O(|V| + |E|) linear time complexity. This optimized concept graph serves as the concept graph for the next iteration.

[0336] FIG11 is a schematic flow chart of a text processing method 900 provided in an embodiment of the present application. This method can be executed by a device or apparatus capable of text processing. For example, the device can be a cloud service device or a terminal device, such as a computer or server with sufficient computing power to execute the text processing method, or a system composed of a cloud service device and a terminal device. For example, method 900 can be executed by execution device 210 in FIG2 , execution device 110 in FIG1 , or a local device.

[0337] For example, the method 900 may be specifically executed by the execution device 210 shown in FIG. 2 , and the text to be processed in the method 900 may be input data provided by the client device 240 shown in FIG. 2 .

[0338] The model used in the text processing method 900 in FIG11 can be constructed by the method described in FIG5 or FIG10. The related description can refer to the aforementioned method 500 or method 800. To avoid unnecessary repetition, the repeated description will be omitted when introducing the method 900.

[0339] Method 900 includes steps 910 to 960 , which are described below.

[0340] 910, obtain the text to be processed.

[0341] 920, obtain the knowledge graph.

[0342] 930, determine the text encoding of the text to be processed.

[0343] 940, determining a concept graph of the text to be processed based on the knowledge graph.

[0344] 950, input the concept map into the target RGAT for processing to obtain the knowledge encoding of the text to be processed.

[0345] 960 , determining a processing result of the text to be processed based on the text encoding of the text to be processed and the knowledge encoding of the text to be processed.

[0346] The target RGAT is obtained by inputting the initial concept graph of the training text into the RGAT for training. During the training process, the first concept graph in the i+1th iteration is determined based on the relevance of the nodes in the second concept graph in the i-th iteration to the training text and the weight of the edges in the second concept graph, where i is a positive integer. The first concept graph is a subgraph of the initial concept graph, and the second concept graph is a subgraph of the initial concept graph. The nodes in the initial concept graph include topic nodes, wherein the topic nodes include candidate entities in the knowledge graph corresponding to the target noun phrases in the training text. The edges between the nodes in the initial concept graph are used to represent the entity relationships between the nodes in the initial concept graph.

[0347] It should be understood that the step numbers in the embodiments of the present application are only for convenience of description and do not limit the order in which the steps are executed.

[0348] In step 940, the topic nodes in the concept graph of the text to be processed may include candidate entities in the knowledge graph corresponding to the target noun phrases in the text to be processed, and the edges between the nodes in the concept graph of the text to be processed are used to represent the entity relationships between the nodes in the concept graph.

[0349] According to the solution of the embodiment of the present application, in the process of RGAT learning expression through iterative messages between entity nodes, that is, in the training process, the concept map is optimized according to the relevance of the entity node to the text and the weight of the edge. For example, nodes with less relevance to the training text and / or edges with smaller weights are pruned, which is beneficial to reducing the model's attention to knowledge with less relevance to the text and enhancing its attention to knowledge with greater relevance to the text, thereby improving the expressive ability of RGAT, and further helping to improve the accuracy of downstream tasks.

[0350] Optionally, the initial concept graph further includes neighbor nodes, and the neighbor nodes include neighbor entities in the knowledge graph of candidate entities corresponding to the target noun phrase in the training text.

[0351] Accordingly, the neighbor nodes in the concept graph of the text to be processed may include the neighbor entities in the knowledge graph of the candidate entities corresponding to the target noun phrase in the text to be processed.

[0352] Optionally, the topic nodes in the initial concept graph include all candidate entities in the knowledge graph that correspond to the target noun phrases in the training text.

[0353] The topic nodes in the concept graph of the text to be processed may include all candidate entities in the knowledge graph corresponding to the target noun phrase in the text to be processed.

[0354] Optionally, the first concept graph in the (i+1)th iteration is determined based on the relevance of the nodes in the second concept graph in the (i)th iteration to the training text and the weight of the edges in the second concept graph, including: selecting edges in the second concept graph as edges in the first concept graph in order of the ratio of the benefit to the first consumption of the edges in the second concept graph from large to small, until the sum of the first consumption of the selected edges is greater than a threshold, the benefit of the edges in the second concept graph is positively correlated with the weight of the edges in the (i)th iteration, and the first consumption of the edges in the second concept graph is negatively correlated with the relevance of the two nodes connected by the edges to the training text.

[0355] Optionally, the first concept graph in the (i+1)th iteration is determined based on the relevance of the nodes in the second concept graph in the (i)th iteration to the training text and the weights of the edges in the second concept graph, including: selecting the connected subgraphs in the second concept graph as the connected subgraphs of the first concept graph in order of the second consumption of the connected subgraphs in the second concept graph from small to large, until all the selected connected subgraphs include at least one candidate entity corresponding to the target noun phrase.

[0356] Optionally, the relevance of the topic node to the training text is determined based on the eigenvector centrality of the topic node on the topic relevance graph, the nodes in the topic relevance graph include the topic node, and the weight of the edge in the topic relevance graph is determined based on the number of entity relationships between the entities corresponding to the two nodes connected by the edge in the knowledge graph.

[0357] Optionally, the relevance of the neighbor node to the training text is determined based on the score of the strongly connected branch where the neighbor node is located on the information propagation graph, where the nodes in the information propagation graph include the nodes in the initial concept graph. When the first node in the initial concept graph is a one-hop neighbor of the second node, there is a directed edge between the second node and the first node in the information propagation graph, from the second node to the first node.

[0358] Optionally, the score of the strongly connected branch on the information propagation graph is obtained by propagating the initial score of the strongly connected branch where the subject node is located to the downstream strongly connected branches according to topological sorting, and the initial score of the strongly connected branch where the subject node is located is determined based on the maximum importance of the nodes in the strongly connected branch where the subject node is located.

[0359] Optionally, method 900 further includes: outputting a knowledge path in the concept map of the text to be processed based on the text encoding of the text to be processed and the knowledge encoding of the text to be processed, where the knowledge path is used to indicate a basis for determining a processing result.

[0360] A knowledge path is a path between two nodes in the concept map. For example, node eq and node e q+k The k-hop knowledge path between q ,r q ,e q+1 ,r q+1 ,…,r q+k-1 ,e q+k ),(e q ,r q ,e q+1 ) is a triple, r q Represents the entity relationship between two nodes, and so on. q is a positive integer.

[0361] The knowledge path can improve the interpretability of the model, provide users with a basis for judging the processing results, and help improve user trust. Moreover, the concept map in the solution of the embodiment of the present application has comprehensive and accurate knowledge, which is conducive to ensuring the integrity and accuracy of the knowledge path.

[0362] Optionally, the weight of the knowledge path is determined according to the attention weights of the triples in the knowledge path.

[0363] The attention weight of a triple is used to indicate the importance of the triple in the reasoning process of RGAT. The attention weight of the knowledge path is used to indicate the importance of the knowledge path.

[0364] Exemplarily, the weight of the knowledge path may be the average of the attention weights of the triples in the knowledge path.

[0365] For example, during the reasoning process of RGAT, the expression of node i at layer l+1 is The following formula can be satisfied:

[0366]

[0367] Where ψ represents the set of all triples in the knowledge graph. σ represents an activation function, which can be, for example, a binary step function, a liner activation function, a sigmoid function, a rectified linear unit (ReLU), or a leaky ReLU. In the embodiments of the present application, a LeakyReLU can be used as the activation function. Represents the expression of node i’s neighbor node j at layer l. Represents the importance of node j to node i in relation r. The attention weight of the triple (j, r, i) is R' represents a set of relations. l is an integer greater than or equal to 0.

[0368] Figure 12 shows a schematic flow chart of a text classification method according to an embodiment of the present application. The method shown in Figure 12 can be regarded as a specific implementation of the method shown in Figure 11. For the sake of brevity, some descriptions are omitted when describing method 1000.

[0369] Method 1000 includes steps 1010 to 1040 .

[0370] 1010, performing knowledge extraction on the text to be processed.

[0371] Exemplarily, step 1010 may be performed by the knowledge extraction module 410 in FIG. 4 .

[0372] For example, the knowledge graph is a knowledge graph of a business field related to text data. For ease of understanding and description, the following text uses the business field of medicine as an example to illustrate method 1010, without limiting the solution of the embodiment of the present application.

[0373] As shown in FIG10 , step 1010 may include: identifying the knowledge triples T in the text d to be processed based on the knowledge graph; d , to get the noun phrase M in the knowledge triple d and the relative phrase P d , that is, the target noun phrase and target relation phrase in the text to be processed.

[0374] 1020, generate a text-level encoding of the text to be processed.

[0375] Exemplarily, step 1020 may be performed by the text encoding module 420 in FIG. 4 .

[0376] The text-level encoding of the text to be processed may include each noun phrase M in the text to be processed d d The text-level encoding of each relational phrase P d The text-level encoding of the text sequence to be processed and the text-level encoding of the text sequence to be processed.

[0377] 1030, generating a knowledge-level encoding of the text to be processed through the target RGAT.

[0378] In a possible implementation, the text-level encoding of the text to be processed may participate in the process of the target RGAT generating the knowledge-level encoding of the text to be processed.

[0379] The target RGAT used in FIG10 may be obtained by training through the method 800 shown in FIG10 . For a specific training method, reference may be made to the description of the method 800 , which will not be described in detail here.

[0380] Exemplarily, step 1030 may be performed by the knowledge encoding module 430 in FIG. 4 .

[0381] For example, each noun phrase M in the text to be processed is located in the knowledge graph d All corresponding candidate entities are identified, and the k-hop neighboring entities of each candidate entity are located as nodes in the concept graph of the text to be processed. The nodes in the concept graph of the text to be processed are connected according to the entity relationships between each pair of entities recorded in the knowledge graph to obtain the concept graph of the text to be processed. This concept graph of the text to be processed contains the complete knowledge.

[0382] The specific method of constructing the concept map of the text to be processed can refer to the method of constructing the initial concept map of the training text in the previous article. Just replace the training text in the relevant description with the text to be processed. I will not go into details here.

[0383] 1040, a predicted classification result is obtained based on the encoding at the text level and the encoding at the knowledge level.

[0384] Exemplarily, step 1040 may be performed by the task processing module 440 in FIG. 4 .

[0385] Step 1040 may include performing vector concatenation on the code at the text level and the code at the knowledge level to obtain a text fusion code, and inputting the text fusion code into a classifier to obtain a classification result of the text to be processed.

[0386] Figure 13 shows a schematic diagram of a text classification result of an embodiment of the present application. As shown in Figure 13, the text is roughly "... Diabetes has become an epidemic, and the number of people with type 2 diabetes is increasing at an alarming rate. We know that controlling diet and a Western lifestyle can lead to type 2 diabetes and cardiovascular disease...". The solution of the embodiment of the present application determines that this paragraph of text contains false information. The primary basis for judgment is the knowledge path ('diet', 'reducesRiskFor', 'atherosclerosis', 'causes', 'cardiovascular diseases') obtained from the concept map, that is, (control diet, reduce risk (-), arteriosclerosis, lead to (+), cardiovascular disease), with a weight of 0.99998. The secondary basis for judgment is the knowledge path ('diet', 'alleviates', 'diabetes') obtained from the concept map, that is, (control diet, alleviate (-), diabetes), with a weight of 0.57651. These two knowledge paths are inconsistent with the text semantics "control diet can lead to type 2 diabetes and cardiovascular disease", and the text is judged to contain false information. The solution of the embodiment of the present application can improve the accuracy of the classification task, and at the same time generate a knowledge path with weights as an explainable classification basis.

[0387] Table 1 shows the comparison results of the performance indicators of text classification on two datasets, diabetes and cancer, using the solution of the embodiment of the present application and the existing knowledge guided graph attention network for detecting healthcare misinformation (DETERRENT). Table 1 shows four performance indicators, namely: accuracy, precision, recall and F1 score (F1Score). The optimization scheme of the concept map used in the training process of the RGAT model in the solution of the embodiment of the present application in Table 1 is the scheme in Example 1.

[0388] Table 1

[0389]

[0390] As can be seen from Table 1, the solution of the embodiment of the present application improves the above indicators by 1-5 percentage points compared with the existing solution. The solution of the embodiment of the present application can effectively improve the accuracy of the classification results.

[0391] Table 2 shows the comparison of the performance indicators of text classification using the solution of the embodiment of the present application and the existing DETERRENT on the two datasets of diabetes and cancer. The optimization scheme of the concept map used in the training process of the RGAT model used in the solution of the embodiment of the present application in Table 2 is the scheme in Example 2.

[0392] Table 2

[0393]

[0394] As can be seen from Table 2, the solution of the embodiment of the present application improves the above indicators by 1-58 percentage points compared with the existing solution. The solution of the embodiment of the present application can effectively improve the accuracy of the classification results.

[0395] The following describes the device of the embodiment of the present application in detail with reference to the accompanying drawings. It should be understood that the device described below can perform the method of the embodiment of the present application. To avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing the device of the embodiment of the present application.

[0396] FIG14 is a schematic block diagram of a training device according to an embodiment of the present application. The training device 3000 shown in FIG14 includes an acquisition unit 3010 and a processing unit 3020 .

[0397] The training device can be used to execute method 500 or method 800 of the embodiment of the present application.

[0398] In one possible implementation, the acquisition unit 3010 may execute the above steps 510 and 520. The processing unit 3020 may execute the above steps 530 to 540. It should be noted that the acquisition unit for executing step 510 and the acquisition unit for executing step 520 may be the same or different.

[0399] FIG15 is a schematic block diagram of a text processing apparatus according to an embodiment of the present application. The apparatus 4000 shown in FIG15 includes an acquisition unit 4010 and a processing unit 4020.

[0400] The apparatus 4000 may be used to execute the method 900 of the embodiment of the present application.

[0401] In one possible implementation, the acquisition unit 4010 may execute the above steps 910 and 920. The processing unit 4020 may execute the above steps 930 to 960. It should be noted that the acquisition unit for executing step 910 and the acquisition unit for executing step 920 may be the same or different.

[0402] It should be noted that the training apparatus 3000 and the apparatus 4000 are implemented in the form of functional units. The term "unit" herein can be implemented in the form of software and / or hardware, and is not specifically limited thereto.

[0403] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0404] Therefore, the units of each example described in the embodiments of this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0405] Figure 16 is a schematic diagram of the hardware structure of a training device provided in an embodiment of the present application. The training device 5000 shown in Figure 16 (which may specifically be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via bus 5004.

[0406] Memory 5001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 5001 may store a program. When the program stored in memory 5001 is executed by processor 5002, processor 5002 is configured to execute the various steps of the training method of the embodiment of the present application. For example, processor 5002 may execute method 500 or method 800 described above.

[0407] The processor 5002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the training method of the embodiment of the method of the present application.

[0408] The processor 5002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the present application may be completed by hardware integrated logic circuits in the processor 5002 or software instructions.

[0409] The processor 5002 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application may be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-established in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or register. The storage medium is located in the memory 5001. The processor 5002 reads the information in the memory 5001 and, in conjunction with its hardware, performs the functions required to be performed by the units included in the apparatus shown in FIG. 14 , or executes method 500 or method 800 of the method embodiments of this application.

[0410] The communication interface 5003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the device 5000 and other devices or a communication network. For example, training text and knowledge graphs can be obtained through the communication interface 5003.

[0411] The bus 5004 may include a path for transmitting information between various components of the device 5000 (eg, the memory 5001 , the processor 5002 , and the communication interface 5003 ).

[0412] Figure 17 is a schematic diagram of the hardware structure of a text processing device provided in an embodiment of the present application. The device 6000 shown in Figure 17 (which may specifically be a computer device) includes a memory 6001, a processor 6002, a communication interface 6003, and a bus 6004. The memory 6001, processor 6002, and communication interface 6003 are interconnected via bus 6004.

[0413] Memory 6001 may be a ROM, static storage device, dynamic storage device, or RAM. Memory 6001 may store programs. When the programs stored in memory 6001 are executed by processor 6002, processor 6002 is configured to perform the various steps of the text processing method according to an embodiment of the present application. For example, processor 6002 may execute method 900 described above.

[0414] The processor 6002 can adopt a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the text processing method of the method embodiment of the present application.

[0415] The processor 6002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the text processing method of the present application may be completed by hardware integrated logic circuits in the processor 6002 or software instructions.

[0416] The processor 6002 may also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor, or it may be any conventional processor. The steps of the methods disclosed in the embodiments of this application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software module may be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory 6001. The processor 6002 reads the information in memory 6001 and, in conjunction with its hardware, performs the functions required by the units included in the apparatus shown in FIG. 15 , or executes method 900 of the method embodiment of this application.

[0417] The communication interface 6003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the device 6000 and other devices or communication networks. For example, the text and knowledge graph to be processed can be obtained through the communication interface 6003.

[0418] The bus 6004 may include a path for transmitting information between various components of the device 6000 (eg, the memory 6001 , the processor 6002 , and the communication interface 6003 ).

[0419] An embodiment of the present application also provides a computer-readable medium that stores program code for execution by a device, wherein the program code includes any one of the training methods or text processing methods for executing the text processing model in the embodiment of the present application.

[0420] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the text processing model training methods or text processing methods in the embodiments of the present application.

[0421] An embodiment of the present application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface and executes any one of the text processing model training methods or text processing methods in the embodiment of the present application.

[0422] Optionally, as an implementation method, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute any one of the training methods of the text processing model or the text processing methods in the embodiments of the present application.

[0423] It should also be understood that in the embodiments of the present application, the memory may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the processor may also include non-volatile random access memory. For example, the processor may also store device type information.

[0424] It should be understood that the term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0425] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0426] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0427] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0428] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0429] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0430] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0431] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a universal serial bus flash disk (USB flash disk, UFD), which can also be referred to as a USB flash drive or USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0432] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A training method for a text processing model, characterized in that: include: Get training text; Obtain knowledge graph; Determining an initial concept graph of the training text based on the knowledge graph, wherein the nodes in the initial concept graph include topic nodes, the topic nodes include candidate entities in the knowledge graph corresponding to target noun phrases in the training text, and edges between nodes in the initial concept graph are used to represent entity relationships between nodes in the initial concept graph; The initial concept graph is input into the relationship-aware attention network RGAT model for training to obtain a target RGAT model. During the training process, the first concept graph in the i+1th iteration is determined based on the relevance of the nodes in the second concept graph in the i-th iteration to the training text and the weight of the edges in the second concept graph, where i is a positive integer, the first concept graph is a subgraph of the initial concept graph, and the second concept graph is a subgraph of the initial concept graph.

2. The training method according to claim 1, characterized in that The subject node includes all candidate entities in the knowledge graph corresponding to the target noun phrase.

3. The training method according to claim 1 or 2, characterized in that: The first concept graph in the (i+1)th iteration is determined based on the relevance between the nodes in the second concept graph in the (i)th iteration and the training text and the weights of the edges in the second concept graph, including: In descending order of the ratio of the benefit to the first consumption of the edges in the second concept graph, edges in the second concept graph are selected as edges in the first concept graph until the sum of the first consumption of the selected edges is greater than a threshold value, the benefit of the edges in the second concept graph is positively correlated with the weight of the edges in the i-th iteration, and the first consumption of the edges in the second concept graph is negatively correlated with the relevance between the two nodes connected by the edges and the training text.

4. The training method according to claim 1 or 2, characterized in that: The first concept graph in the (i+1)th iteration is determined based on the relevance between the nodes in the second concept graph in the (i)th iteration and the training text and the weights of the edges in the second concept graph, including: According to the order of the second consumption of the connected subgraph in the second concept graph from small to large, the connected subgraph in the second concept graph is selected as the connected subgraph of the first concept graph until all the selected connected subgraphs include at least one candidate entity corresponding to the target noun phrase.

5. The training method according to any one of claims 1 to 4, characterized in that: The relevance of the topic node to the training text is determined based on the eigenvector centrality of the topic node on the topic relevance graph, the nodes in the topic relevance graph include the topic node, and the weight of the edge in the topic relevance graph is determined based on the number of entity relationships between the entities corresponding to the two nodes connected by the edge in the knowledge graph.

6. The training method according to any one of claims 1 to 5, characterized in that: The initial concept graph also includes neighbor nodes, and the neighbor nodes include neighbor entities of the candidate entity corresponding to the target noun phrase in the knowledge graph.

7. The training method according to claim 6, characterized in that The relevance of the neighbor node to the training text is determined based on the score of the strongly connected branch where the neighbor node is located on the information propagation graph. The nodes in the information propagation graph include the nodes in the initial concept graph. When the first node in the initial concept graph is a one-hop neighbor of the second node, there is a directed edge between the second node and the first node in the information propagation graph, from the second node to the first node.

8. The training method according to claim 7, characterized in that: The score of the strongly connected branch on the information propagation graph is obtained by propagating the initial score of the strongly connected branch where the subject node is located to the downstream strongly connected branch according to topological sorting. The initial score of the strongly connected branch where the subject node is located is determined based on the maximum importance of the nodes in the strongly connected branch where the subject node is located.

9. A method for text processing, characterized in that: include: Get the text to be processed; Obtain knowledge graph; Determining the text encoding of the text to be processed; Determine a concept graph of the text to be processed based on the knowledge graph; The concept graph of the text to be processed is processed by a target RGAT to obtain a knowledge encoding of the text to be processed, wherein the target RGAT is obtained by inputting the initial concept graph of the training text into the RGAT for training. During the training process, the first concept graph in the i+1th iteration is determined based on the relevance of the nodes in the second concept graph in the i-th iteration to the training text and the weight of the edges in the second concept graph, i is a positive integer, the first concept graph is a subgraph of the initial concept graph, the second concept graph is a subgraph of the initial concept graph, the nodes in the initial concept graph include topic nodes, the topic nodes include candidate entities in the knowledge graph corresponding to the target noun phrases in the training text, and the edges between the nodes in the initial concept graph are used to represent the entity relationships between the nodes in the initial concept graph; A processing result of the text to be processed is determined based on the text encoding of the text to be processed and the knowledge encoding of the text to be processed.

10. The method according to claim 9, characterized in that The subject node includes all candidate entities in the knowledge graph corresponding to the target noun phrase.

11. The method according to claim 9 or 10, characterized in that The first concept graph in the (i+1)th iteration is determined based on the relevance between the nodes in the second concept graph in the (i)th iteration and the training text and the weights of the edges in the second concept graph, including: In descending order of the ratio of the benefit to the first consumption of the edges in the second concept graph, edges in the second concept graph are selected as edges in the first concept graph until the sum of the first consumption of the selected edges is greater than a threshold value, the benefit of the edges in the second concept graph is positively correlated with the weight of the edges in the i-th iteration, and the first consumption of the edges in the second concept graph is negatively correlated with the relevance between the two nodes connected by the edges and the training text.

12. The method according to claim 9 or 10, characterized in that The first concept graph in the (i+1)th iteration is determined based on the relevance between the nodes in the second concept graph in the (i)th iteration and the training text and the weights of the edges in the second concept graph, including: According to the order of the second consumption of the connected subgraph in the second concept graph from small to large, the connected subgraph in the second concept graph is selected as the connected subgraph of the first concept graph until all the selected connected subgraphs include at least one candidate entity corresponding to the target noun phrase.

13. The method according to any one of claims 9 to 12, characterized in that The initial concept graph also includes neighbor nodes, and the neighbor nodes include neighbor entities of the candidate entity corresponding to the target noun phrase in the knowledge graph.

14. A training device for a text processing model, characterized in that: include: Get unit for: Get training text; Obtain knowledge graph; A processing unit for: Determining an initial concept graph of the training text based on the knowledge graph, wherein the nodes in the initial concept graph include topic nodes, the topic nodes include candidate entities in the knowledge graph corresponding to target noun phrases in the training text, and edges between nodes in the initial concept graph are used to represent entity relationships between nodes in the initial concept graph; The initial concept graph is input into the relationship-aware attention network RGAT model for training to obtain a target RGAT model. During the training process, the first concept graph in the i+1th iteration is determined based on the relevance of the nodes in the second concept graph in the i-th iteration to the training text and the weight of the edges in the second concept graph, where i is a positive integer, the first concept graph is a subgraph of the initial concept graph, and the second concept graph is a subgraph of the initial concept graph.

15. The training device according to claim 14, characterized in that The subject node includes all candidate entities in the knowledge graph corresponding to the target noun phrase.

16. The training device according to claim 14 or 15, characterized in that The processing unit is specifically configured to: In descending order of the ratio of the benefit to the first consumption of the edges in the second concept graph, edges in the second concept graph are selected as edges in the first concept graph until the sum of the first consumption of the selected edges is greater than a threshold value, the benefit of the edges in the second concept graph is positively correlated with the weight of the edges in the i-th iteration, and the first consumption of the edges in the second concept graph is negatively correlated with the relevance between the two nodes connected by the edges and the training text.

17. The training device according to claim 14 or 15, characterized in that The processing unit is specifically configured to: According to the order of the second consumption of the connected subgraph in the second concept graph from small to large, the connected subgraph in the second concept graph is selected as the connected subgraph of the first concept graph until all the selected connected subgraphs include at least one candidate entity corresponding to the target noun phrase.

18. The training device according to any one of claims 14 to 17, characterized in that The relevance of the topic node to the training text is determined based on the eigenvector centrality of the topic node on the topic relevance graph, the nodes in the topic relevance graph include the topic node, and the weight of the edge in the topic relevance graph is determined based on the number of entity relationships between the entities corresponding to the two nodes connected by the edge in the knowledge graph.

19. The training device according to any one of claims 14 to 18, characterized in that The initial concept graph also includes neighbor nodes, and the neighbor nodes include neighbor entities of the candidate entity corresponding to the target noun phrase in the knowledge graph.

20. The training device according to claim 19, characterized in that The relevance of the neighbor node to the training text is determined based on the score of the strongly connected branch where the neighbor node is located on the information propagation graph. The nodes in the information propagation graph include the nodes in the initial concept graph. When the first node in the initial concept graph is a one-hop neighbor of the second node, there is a directed edge between the second node and the first node in the information propagation graph, from the second node to the first node.

21. The training device according to claim 20, characterized in that The score of the strongly connected branch on the information propagation graph is obtained by propagating the initial score of the strongly connected branch where the subject node is located to the downstream strongly connected branch according to topological sorting. The initial score of the strongly connected branch where the subject node is located is determined based on the maximum importance of the nodes in the strongly connected branch where the subject node is located.

22. A text processing device, characterized in that: include: Get unit for: Get the text to be processed; Obtain knowledge graph; A processing unit for: Determining the text encoding of the text to be processed; Determine a concept graph of the text to be processed based on the knowledge graph; The concept graph of the text to be processed is processed by a target RGAT to obtain a knowledge encoding of the text to be processed, wherein the target RGAT is obtained by inputting the initial concept graph of the training text into the RGAT for training. During the training process, the first concept graph in the i+1th iteration is determined based on the relevance of the nodes in the second concept graph in the i-th iteration to the training text and the weight of the edges in the second concept graph, i is a positive integer, the first concept graph is a subgraph of the initial concept graph, the second concept graph is a subgraph of the initial concept graph, the nodes in the initial concept graph include topic nodes, the topic nodes include candidate entities in the knowledge graph corresponding to the target noun phrases in the training text, and the edges between the nodes in the initial concept graph are used to represent the entity relationships between the nodes in the initial concept graph; A processing result of the text to be processed is determined based on the text encoding of the text to be processed and the knowledge encoding of the text to be processed.

23. The device according to claim 22, characterized in that The subject node includes all candidate entities in the knowledge graph corresponding to the target noun phrase.

24. The device according to claim 22 or 23, characterized in that The first concept graph in the (i+1)th iteration is determined based on the relevance between the nodes in the second concept graph in the (i)th iteration and the training text and the weights of the edges in the second concept graph, including: In descending order of the ratio of the benefit to the first consumption of the edges in the second concept graph, edges in the second concept graph are selected as edges in the first concept graph until the sum of the first consumption of the selected edges is greater than a threshold value, the benefit of the edges in the second concept graph is positively correlated with the weight of the edges in the i-th iteration, and the first consumption of the edges in the second concept graph is negatively correlated with the relevance between the two nodes connected by the edges and the training text.

25. The device according to claim 22 or 23, characterized in that The first concept graph in the (i+1)th iteration is determined based on the relevance between the nodes in the second concept graph in the (i)th iteration and the training text and the weights of the edges in the second concept graph, including: According to the order of the second consumption of the connected subgraph in the second concept graph from small to large, the connected subgraph in the second concept graph is selected as the connected subgraph of the first concept graph until all the selected connected subgraphs include at least one candidate entity corresponding to the target noun phrase.

26. The device according to any one of claims 22 to 25, characterized in that The initial concept graph also includes neighbor nodes, and the neighbor nodes include neighbor entities of the candidate entity corresponding to the target noun phrase in the knowledge graph.

27. A text processing device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 8 or claims 9 to 13.

28. A computer-readable storage medium, characterized in that The computer-readable medium stores program codes for execution by a device, the program codes including codes for executing the method according to any one of claims 1 to 8 or claims 9 to 13.

29. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 8 or claims 9 to 13.

Images