An entity semantic association reasoning method and system based on reinforcement learning

Abstract

The application discloses an entity semantic correlation reasoning method and system based on reinforcement learning, wherein the method comprises the following steps: acquiring standard information documents and entities to be reasoned; performing natural language processing on the standard information documents to determine entities, correlation information and a measurable quantity information four-tuple in the standard information documents; performing entity correlation processing on the entities and the correlation information to determine entity contact data sets; constructing a polygon structure network according to the measurable quantity information four-tuple; taking the polygon structure network as a reinforcement learning environment, combining a reward function set based on the reinforcement learning environment, and determining a reinforcement learning model; and inputting the entities to be reasoned into the reinforcement learning model according to the entity contact data sets to perform correlation reasoning processing and determine a target entity set. The embodiment of the application can better process semantic information of entities and corresponding measurable quantity information, reduce the learning cost of a reinforcement learning intelligent agent, and can be widely applied to the field of artificial intelligence technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an entity semantic correlation reasoning method and system based on reinforcement learning. BACKGROUND

[0002] In related technologies, knowledge graph technology is usually combined with reinforcement learning to process entities in text, so as to perform entity correlation reasoning. However, for standard information documents, not only entity information in the documents needs to be processed, but also measurable quantity information corresponding to an entity needs to be processed, and how to fuse the semantics of the entity and the semantics of the quantity information corresponding to the entity for processing. However, the knowledge graph has no way to well fuse the semantic information of the entity and the measurable quantity information for processing, which to some extent makes the reinforcement learning agent unable to well learn the correct reasoning mode, thereby increasing the learning cost of the reinforcement learning agent. In summary, the technical problems existing in related technologies need to be solved. SUMMARY

[0003] Therefore, the embodiments of the present application provide an entity semantic correlation reasoning method and system based on reinforcement learning, so as to better process the semantic information of entities and their corresponding measurable quantity information, and reduce the learning cost of the reinforcement learning agent.

[0004] In one aspect, the present application provides an entity semantic correlation reasoning method based on reinforcement learning, which comprises:

[0005] obtaining a standard information document and a to-be-reasoned entity;

[0006] performing natural language processing on the standard information document to determine an entity, correlation information, and a measurable quantity information four tuple in the standard information document;

[0007] performing entity correlation processing on the entity and the correlation information to determine an entity contact data set;

[0008] constructing a polygon structure network according to the measurable quantity information four tuple;

[0009] using the polygon structure network as a reinforcement learning environment, and combining a reward function set based on the reinforcement learning environment to determine a reinforcement learning model;

[0010] inputting the to-be-reasoned entity into the reinforcement learning model according to the entity contact data set to perform correlation reasoning processing and determine a target entity set.

[0011] Optionally, the entity correlation processing on the entity and the correlation information to determine the entity contact data set comprises:

[0012] inputting the entity and the association information into a Chinese corpus to perform word embedding processing, to determine entity semantic information and association semantic information;

[0013] performing three-dimensional mapping processing on the entity semantic information and the association semantic information, to determine an entity contact dataset.

[0014] Optionally, the constructing a polygon structure network according to the measurable quantity information quadruple comprises:

[0015] performing mapping and filling processing on the measurable quantity information quadruple in combination with a first semantic formula, to determine semantic information of a network node;

[0016] performing semantic distance calculation processing on the semantic information of the network node, to determine a similar dataset;

[0017] selecting k network nodes from the similar dataset to construct a network layer, the k being an integer greater than or equal to 3;

[0018] determining semantic information of a connected edge according to the semantic information of the network node in combination with a second semantic formula;

[0019] determining a polygon structure network according to the semantic information of the network node, the semantic information of the connected edge, and the network layer.

[0020] Optionally, the taking the polygon structure network as a reinforcement learning environment and determining a reinforcement learning model in combination with a reward function set based on the reinforcement learning environment comprises:

[0021] taking the polygon structure network as a reinforcement learning model environment;

[0022] determining a reward function according to interaction of an agent in the reinforcement learning model environment;

[0023] determining a state definition of the agent according to node semantic information of a current node of the agent in the reinforcement learning model environment, a layer number of the current node in the network, and a distance already walked;

[0024] determining an action definition of the agent according to selection of the agent in the reinforcement learning model environment of a next step to jump from the current node to another node;

[0025] determining a reinforcement learning model according to the reinforcement learning model environment, the reward function, the state definition of the agent, and the action definition of the agent.

[0026] Optionally, the reward function set based on the reinforcement learning environment comprises:

[0027] The definition formula of the reward function Reward is:

[0028]

[0029] Wherein, subscript t represents a target entity, subscript i represents an entity represented by a network node at a position of an agent, E represents semantic information of the network node, D c represents a cosine similarity calculation, ∑d represents a path length walked by the agent, L represents a number of layers of the network, the number of layers of the network increases from inside to outside, K is a number of nodes contained in each layer of the network of the polygonal structure, and a, β, and γ are weight coefficients.

[0030] Optionally, before the inputting of the entity to be reasoned into the reinforcement learning model for the associated reasoning processing, the method further comprises pre-training the reinforcement learning model, specifically comprising:

[0031] Obtaining an agent state of an agent at a current network node in the reinforcement learning model;

[0032] Inputting the agent state into the reinforcement learning model, calculating action evaluation values corresponding to different network nodes walked by the agent, and determining a set of action evaluation values;

[0033] According to the set of action evaluation values, a greedy strategy is used to predict a next action of the agent, and a selected action of the agent is determined.

[0034] According to the selected action of the agent, the agent state and a reward value are updated until an action evaluation value function converges, and the reinforcement learning model is trained.

[0035] Optionally, the inputting of the entity to be reasoned into the reinforcement learning model for the associated reasoning processing, and the determination of a target entity set, comprises:

[0036] The entity to be reasoned is input into the reinforcement learning model, the agent interacts with the polygonal structure network, and a target entity set of the entity to be reasoned in the entity contact data set is output.

[0037] On the other hand, the embodiment of the application further provides a system, comprising:

[0038] A first module is configured to obtain a standard information document and an entity to be reasoned.

[0039] A second module is configured to perform natural language processing on the standard information document, and determine an entity, associated information, and a measurable quantity information four-tuple in the standard information document.

[0040] The third module is configured to perform entity association processing on the entity and the association information, and determine an entity contact dataset;

[0041] The fourth module is configured to construct a polygon structure network according to the quantifiable quantity information quadruple;

[0042] The fifth module is configured to use the polygon structure network as a reinforcement learning environment, and determine a reinforcement learning model in combination with a reward function set based on the reinforcement learning environment;

[0043] The sixth module is configured to input the to-be-reasoned entity into the reinforcement learning model for association reasoning processing according to the entity contact dataset, and determine a target entity set.

[0044] In another aspect, the embodiment of the present application further discloses an electronic device, comprising a processor and a memory;

[0045] The memory is configured to store a program;

[0046] The processor executes the program to implement the method as described above.

[0047] In another aspect, the embodiment of the present application further discloses a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method as described above.

[0048] In another aspect, the embodiment of the present application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method as described above.

[0049] Compared with the prior art, the above technical solution has the following technical effects: the embodiment of the present application can make each network node fuse semantic information of an entity and corresponding quantifiable quantity information, so as to more comprehensively and effectively process standard information documents. In addition, the embodiment of the present application can use the polygon structure network as a reinforcement learning environment, and determine a reinforcement learning model in combination with a reward function set based on the reinforcement learning environment, so that the training of an intelligent agent can find a target entity more quickly through semantic comparison of nodes in different network layers, thereby saving the training cost and time of reinforcement learning. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0051] Figure 1 is a flowchart of an entity semantic correlation reasoning method based on reinforcement learning provided by an embodiment of the present application;

[0052] Figure 2 is a schematic diagram of a polygon structure network provided by an embodiment of the present application;

[0053] Figure 3 is another schematic diagram of a polygon structure network provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0055] Firstly, some terms involved in the present application are analyzed:

[0056] Reinforcement learning (RL), also known as re-education, evaluation learning or enhancement learning, is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of maximizing the reward or achieving a specific goal through learning strategy in the interaction process between the agent and the environment.

[0057] Q learning algorithm is a reinforcement learning algorithm, and the Q function represents the action evaluation value (quality) of performing an action in a state. The goal of the algorithm is to maximize the Q value by selecting the best action among all possible actions in the state to maximize the reward.

[0058] In the related art, the reasoning method for entity correlation of entities in a text is usually implemented using knowledge graph technology. However, in a standard information document, not only entity information needs to be processed, but also the measurable quantity information corresponding to a certain entity needs to be processed. The knowledge graph cannot fuse the semantics of the entity and the semantics of the quantity information corresponding to the entity for processing, which also makes the reinforcement learning agent not able to learn the correct reasoning method well and increases the learning cost of the reinforcement learning agent.

[0059] With reference to Figure 1 The embodiment of the present application provides an entity semantic association reasoning method based on reinforcement learning, comprising the following steps:

[0060] S101, acquiring a standard information document and a to-be-reasoned entity;

[0061] S102, performing natural language processing on the standard information document to determine an entity, association information and a measurable quantity information quadruple in the standard information document;

[0062] S103, performing entity association processing on the entity and the association information to determine an entity contact dataset;

[0063] S104, constructing a polygon structure network according to the measurable quantity information quadruple;

[0064] S105, taking the polygon structure network as a reinforcement learning environment, and combining a reward function set based on the reinforcement learning environment to determine a reinforcement learning model;

[0065] S106, inputting the to-be-reasoned entity into the reinforcement learning model for association reasoning processing according to the entity contact dataset to determine a target entity set.

[0066] In the embodiment of the present application, first, a standard information document is acquired, and through natural language processing, an entity and association information in the document and a measurable quantity information quadruple corresponding to the entity in the document are obtained. The measurable quantity information quadruple comprises an entity, a numerical value, a unit and a comparison relationship. Then, semantic information of the entity and the association information is obtained through a Chinese corpus, and the semantic information is mapped to a three-dimensional space to construct an entity contact database. Next, the entity, the numerical value, the unit and the comparison relationship in the measurable quantity information quadruple are encoded and mapped and filled into network nodes to construct a polygon structure network. The polygon structure network is variable in size. Then, the polygon structure network is taken as a reinforcement learning environment, and a reinforcement learning model is constructed based on a reward function set in the reinforcement learning environment. The reinforcement learning model is a deep neural network model, which is used to train an agent to find a node corresponding to an associated entity of a certain entity in the entity contact dataset. Finally, the to-be-reasoned entity is input into the reinforcement learning model for association reasoning processing, and a target entity set in the entity contact dataset is output.

[0067] Further as a preferred implementation manner, the entity association processing on the entity and the association information to determine the entity contact dataset comprises the following steps:

[0068] inputting the entity and the association information into a Chinese corpus for word embedding processing to determine entity semantic information and association semantic information;

[0069] The entity semantic information and the associated semantic information are subjected to three-dimensional mapping processing to determine an entity contact dataset.

[0070] In the embodiment of the present application, the entity and association information are respectively input into a large corpus based on a large amount of high-quality data pre-trained, wherein the large corpus used in the embodiment of the present application can use a large amount of high-quality Chinese corpus, and the vocabulary semantic embedding of all entities and association information is obtained as their semantic information. The obtained semantic information is mapped into a three-dimensional space, and the entity association information is represented by the coordinates of the three-dimensional space to construct an entity contact dataset.

[0071] Further as a preferred implementation, the constructing a polygon structure network according to the measurable quantity information four-tuple comprises:

[0072] The measurable quantity information four-tuple is subjected to mapping and filling processing according to the first semantic formula to determine the semantic information of the network node;

[0073] The semantic information of the network node is subjected to semantic distance calculation processing to determine a similar dataset;

[0074] A network layer is constructed by selecting k network nodes from the similar dataset, wherein k is an integer greater than or equal to 3;

[0075] The semantic information of the connected edge is determined according to the semantic information of the network node and the second semantic formula;

[0076] The polygon structure network is determined according to the semantic information of the network node, the semantic information of the connected edge, and the network layer.

[0077] In the embodiment of the present application, according to the obtained measurable quantity information four-tuple, the entity, value, unit and comparison relationship in the four-tuple are respectively encoded and mapped and filled into the network nodes of the deformable polygon structure network to construct the semantic information of the network node. The semantic information E of the network node can be calculated by the first semantic formula, and the first semantic formula is as follows:

[0078] E=σ*S e +(1-σ)*(S n +S u +S r );

[0079] In the formula, σ is a parameter, the greater σ is set, the greater the proportion of entity information in the network node is, the more entity information the semantic of the node contains, the smaller σ is set, the less entity information the semantic of the node contains, and the more measurable quantity information the node contains. S eThe semantic vector of the entity itself is stretched or compressed to the same dimension as the network node to obtain. Similarly, S n , S u , and S r are the numerical values, units, and comparison relationship mapping semantic vectors of the same dimension as the network node. Through the above calculation, the network node effectively integrates the semantic information of the entity and the corresponding measurable quantity information of the entity.

[0080] According to the calculated semantic information of the network node, the semantic distance between two network nodes is calculated by cosine similarity. According to the size of the distance and the set threshold, it is judged whether the two network nodes are similar to a certain degree. If the semantic distance is within the specified threshold range, it is considered similar and added to the similar data set, and the similarity percentage is recorded.

[0081] From the above obtained similar data set, K network nodes with similar semantics are selected as a group, and a deformable polygon (i.e. each layer of the polygon structure network contains K nodes, and K is an integer greater than or equal to 3) structure network is constructed according to the value of K to serve as the environment for reinforcement learning. Among them, k network nodes are selected from the similar data set to construct a network layer, and the semantic information E r of the connected edge is determined according to the semantic information of the network node and a second semantic formula.

[0082]

[0083] In the formula, E e1 represents the semantic vector of a network node e1, E e2 represents the semantic vector of another network node e2 connected to e1. E r0 represents the semantic information of the association between entity 1 represented by network node e1 and entity 2 represented by network node e2 in the relationship between entity 1 and entity 2. This semantic information can be obtained by calculating the spatial coordinate distance between entity 1 and entity 2. p is a set percentage parameter, which can be reduced to make the information of the edge between network nodes e1 and e2 contain more information about the association between entity 1 and entity 2 represented by nodes e1 and e2. Conversely, increasing p will make E rThis primarily represents the relationship information between nodes after semantic fusion of entity and measurable quantity information. A polygonal network structure is determined based on the semantic information of the network nodes, the semantic information of the connected edges, and the network layer. In this embodiment, the semantics of the intersection of the connecting edges and the medians of the connecting edges between nodes in the network are also defined, expressed using the semantics of the current network layer. The semantics of the current network layer are obtained by averaging the semantics of all network nodes in the layer. The semantic definition of the network center point differs from the semantic calculation of other network nodes; the semantics of the network center point is variable and is obtained based on the semantics of the source entity nodes in entity reasoning. Each time a different source entity reasoning target entity node is given, the semantics of the center point changes accordingly to the fused semantics expressed by the network nodes of the source entity.

[0084] The polygonal structure neural network of this invention has variability, which is reflected in the fact that the network structure is variable. Each layer of the polygonal structure network can be manually configured to store K semantically similar network nodes. For a network layer of a polygonal structure network composed of K network nodes, as long as K is greater than or equal to 3, it is acceptable. Figure 2 As shown, specifying K=4 yields a polygonal neural network structure; as... Figure 3 As shown, specifying K=5 yields another polygonal structure neural network. This polygonal structure neural network also possesses structural strength. When K is specified, the network consists of K-sided network layers of the same shape. The semantics of nodes in each layer are similar, and the semantics of a network layer is the average of the semantics of all nodes in that layer. When the agent makes a judgment or selection action, it can compare the semantics of the target entity node with the semantics of the node at its current position, and also compare it with the semantics of the current network layer to determine whether the current network layer is likely the layer containing the target entity node. This strong structural strength allows the agent to find the target entity node much faster than a knowledge graph. Furthermore, the polygonal structure network also possesses connectivity. Based on the network structure, the agent in reinforcement learning can move between network nodes in the same layer and different layers. The midline of each edge in the network is also a path that the agent can traverse. Since no network nodes are placed on the midline, the agent can find the network layer containing the target entity node more quickly by comparing the average semantics of the network layers with the semantics of the target entity node. The semantic information of each edge connecting network nodes can be obtained using the semantics calculated by the first semantic formula. The center of the network is the default starting position for the agent. When the agent is given a source entity node for reasoning, if the path of the agent to explore the target entity node exceeds the threshold distance, the agent will return to the center starting point to re-explore and avoid the agent getting stuck in invalid exploration.

[0085] Further as a preferred implementation manner, the polygonal structure network is taken as a reinforcement learning environment, and a reward function is set based on the reinforcement learning environment, and a reinforcement learning model is determined, including:

[0086] The polygonal structure network is taken as a reinforcement learning model environment.

[0087] A reward function is determined according to the interaction of an agent in the reinforcement learning model environment.

[0088] A state definition of the agent is determined according to the node semantic information of a current node of the agent in the reinforcement learning model environment, the layer number of the current node in the network, and the distance that has been walked.

[0089] An action definition of the agent is determined according to the selection of the agent in the reinforcement learning model environment for jumping from the current node to other nodes in the next step.

[0090] A reinforcement learning model is determined according to the reinforcement learning model environment, the reward function, the state definition of the agent, and the action definition of the agent.

[0091] In the embodiment of the application, the polygonal structure network is taken as a reinforcement learning environment, a given entity (i.e., the obtained to-be-reasoned entity) is allowed to find a target entity associated with the given entity in the constructed entity contact data set, and the process is constructed as a Markov process. Reinforcement learning includes an environment, a state, an action, and a reward function definition. The state is determined according to the node semantic information of the network node where the agent is located, the layer number of the node where the agent is located in the network, and the distance that has been walked. The action is defined as the selection of the agent in the reinforcement learning model environment for jumping from the current node to other nodes in the next step. The path in the polygonal structure network constitutes an action space, the agent selects the next step to jump from the current network node to other nodes, and when the distance of the agent jumping to the next hop exceeds a specified threshold distance, the position is reset back to the center node to avoid invalid exploration.

[0092] Further as a preferred implementation manner, the reward function set based on the reinforcement learning environment includes:

[0093] The definition formula of the reward function Reward is:

[0094]

[0095] Wherein, subscript t represents a target entity, subscript i represents an entity of a network node where an agent is located, E represents semantic information of a network node, D represents a distance that has been walked, and Dth represents a specified threshold distance. cCosine similarity calculation, ∑d represents the length of the path that the agent has walked, L represents the number of layers of the network, the number of network layers increases from inside to outside, K is the number of nodes contained in each layer of the network of the polygonal structure, and a, b, and g are weight coefficients.

[0096] In the embodiments of the present application, based on the polygonal structure network as the environment in reinforcement learning, the reward is defined by the following three evaluation indicators in the setting of the reward function:

[0097] 1. Whether the target entity is found in the network node;

[0098] 2. The length of the path that the agent has walked;

[0099] 3. Whether the same layer of the network as the target entity node is reached.

[0100] Based on the three points, the reward function is defined as:

[0101]

[0102] The subscript t represents the target entity, the subscript i represents the entity represented by the network node where the agent is located, E represents the semantic information of the network node, and D c Cosine similarity calculation, ∑d represents the length of the path that the agent has walked, L represents the number of layers of the network, the number of network layers increases from inside to outside, K is the number of nodes contained in each layer of the network of the polygonal structure, and a, b, and g are weight coefficients, and a+ b+ g=1.

[0103] Further, as a preferred embodiment, before the method inputs the entity to be reasoned into the reinforcement learning model for associated reasoning processing, the method further comprises pre-training the reinforcement learning model, specifically comprising:

[0104] Obtaining the agent state of the agent at the current network node in the reinforcement learning model;

[0105] Inputting the agent state into the reinforcement learning model, calculating the action evaluation value corresponding to the different network nodes selected by the agent, and determining the action evaluation value set;

[0106] According to the action evaluation value set, the next step action selection of the agent is predicted and processed by using the greedy strategy, and the selected action of the agent is determined.

[0107] According to the selected action of the agent, the agent state and the reward value are updated until the action evaluation value function converges, and the reinforcement learning model is trained.

[0108] In the embodiment of the present application, the current time state of the network node is input into the reinforcement learning model, the action evaluation values (Q values) corresponding to the different actions of the agent in the current time state are calculated, a set of Q values is constructed, the action to be performed by the agent in the next step is determined according to the epsilon greedy strategy, the agent has an epsilon probability of selecting the action corresponding to the maximum Q value in the current set, and has a 1-epsilon probability of randomly exploring and selecting the action in the next step, the state of the network node where the agent is located at the current time, the reward value, and the action selected thereafter and the reward value corresponding to the selected action are recorded to encapsulate training data for training the neural network, the agent updates the reward value of the agent state after the action is performed and the current state, and the process is repeated until the Q value function converges.

[0109] Further as a preferred implementation, the inputting the entity to be inferred into the reinforcement learning model for association inference processing according to the entity contact data set to determine the target entity set comprises:

[0110] The inputting the entity to be inferred into the reinforcement learning model, the interaction of the agent with the polygon structure network, and the outputting the target entity set of the entity to be inferred in the entity contact data set.

[0111] In the embodiment of the present application, the reinforcement learning model is composed of four parts of environment, state, action and reward. The agent interacts with the environment in the process of exploration, and feedback is generated to the agent in the process of interaction. The agent selects the next action according to the feedback to find the maximum reward. Different from the reinforcement learning model in the related art, for entity reasoning of the standard information document, the reinforcement learning model of the embodiment of the present application improves the environment of reinforcement learning, which is not a knowledge graph but a polygonal structural network as the environment of reinforcement learning. Meanwhile, a reward function more suitable for entity reasoning of the standard information document is proposed based on the structural network, so as to better train the agent to reason similar entities. In order to obtain more rewards, the reinforcement learning agent explores in the direction set by the given reward. In a feasible implementation manner, the entity semantic correlation reasoning target is to find the similar entity of the input entity ("hydrochloric acid solution") in the standard information document. The input entity is "hydrochloric acid solution", and it is assumed that the similar entity "hydrogen chloride solution" of the input entity is in the fourth layer of the polygonal network structure. The agent quickly moves to the fourth layer where the "hydrogen chloride solution" is located by continuously comparing the semantics of the current network node and the semantics of the sought node, and comparing with the average semantics of the network layer. When the network layer with similar semantics is matched, the agent starts to find the similar entity in the network layer. When the moving distance of the agent exceeds the threshold distance, the agent returns to the center node at the beginning and re-explores. The threshold distance can be set according to the actual situation. Finally, all the target entity sets similar to the input entity obtained after reinforcement learning training are output. The entity semantic reasoning effect based on reinforcement learning in the embodiment of the present application is better than that using the knowledge graph, which can help the agent to find the similar entity faster.

[0112] Corresponding to the method of Figure 1 , the embodiment of the present application also provides an electronic device, which comprises a processor and a memory. The memory is used to store a program. The processor executes the program to realize the method as described above.

[0113] Corresponding to the method of Figure 1 , the embodiment of the present application also provides a computer readable storage medium, which stores a program. The program is executed by a processor to realize the method as described above.

[0114] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium. The processor executes the computer instructions, so that the computer device executes the method as shown in Figure 1 .

[0115] In summary, the embodiments of the present application have the following advantages: compared with using a knowledge graph, the deformable polygonal structure network proposed by the present application can train an entity reasoning agent more targetedly because of its strong structure, variability and connectivity, mainly reflected in the design of the network itself: K nodes with similar semantics are stored in the same network layer, the edges of the polygonal network contain similarity information between nodes, the semantics of each network layer is the average semantics of all nodes in the layer, and each network node semantics fusion, etc. Compared with the traditional use of knowledge graph as the environment of reinforcement learning, the semantic information is fully expressed while the training efficiency of reinforcement learning is optimized. When the data volume is large enough, the network structure can make the agent training find the target entity faster through the semantic comparison of nodes in different network layers, saving training cost and time.

[0116] In some alternative embodiments, the functions / operations mentioned in the block diagrams can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously with each other, or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of a larger operation are independently executed.

[0117] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is unnecessary for an understanding of the present application. Rather, given the properties, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be within the routine skill of the engineer, given the present disclosure. Thus, a person skilled in the art, using ordinary skill, can implement the present application as set forth in the claims without undue experimentation. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0118] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0119] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution systems, apparatus or devices. For the purpose of this specification, the "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0120] More specific examples (non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be electronically obtained and then stored in the computer memory.

[0121] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, can be used: a combination of discrete logic circuits having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having logic gates, field programmable gate arrays (FPGA), or other components, in combination or as the case can be.

[0122] In the description of the present application, the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The illustrative appearances of the above terms in various places in the specification are not intended to exclude that the terms in other places mean the same or similar features, structures, materials, or characteristics. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0123] Although embodiments of the present application have been shown and described, it would be recognized by those of ordinary skill in the art that various modifications, alternatives, replacements, and changes can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0124] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for reasoning of entity semantic association based on reinforcement learning, characterized in that, The method comprises: acquiring a standard information document and a to-be-reasoned entity; performing natural language processing on the standard information document to determine an entity, associated information, and a measurable quantity information quadruple in the standard information document; performing entity association processing on the entity and the associated information to determine an entity contact dataset; constructing a polygon structure network according to the measurable quantity information quadruple; taking the polygon structure network as a reinforcement learning environment, combining a reward function set based on the reinforcement learning environment, and determining a reinforcement learning model; inputting the to-be-reasoned entity into the reinforcement learning model for association reasoning processing according to the entity contact dataset to determine a target entity set; the polygon structure network is constructed according to the measurable quantity information quadruple, comprising: combining a first semantic formula to perform mapping and filling processing on the measurable quantity information quadruple to determine semantic information of a network node; performing semantic distance calculation processing on the semantic information of the network node to determine a similar dataset; selecting k network nodes from the similar dataset to construct a network layer, wherein k is an integer greater than or equal to 3; determining semantic information of a connected edge according to the semantic information of the network node and a second semantic formula; determining a polygon structure network according to the semantic information of the network node, the semantic information of the connected edge, and the network layer.

2. The method of claim 1, wherein, the entity association processing on the entity and the associated information to determine an entity contact dataset, comprising: inputting the entity and the associated information into a Chinese corpus for word embedding processing to determine entity semantic information and associated semantic information; performing three-dimensional mapping processing on the entity semantic information and the associated semantic information to determine an entity contact dataset.

3. The method of claim 1, wherein, the polygon structure network is taken as a reinforcement learning environment, and a reward function set based on the reinforcement learning environment is combined to determine a reinforcement learning model, comprising: taking the polygon structure network as a reinforcement learning model environment; determining a reward function according to the interaction of an agent in the reinforcement learning model environment; determining a state definition of the agent according to the node semantic information of the current node of the agent in the reinforcement learning model environment, the number of layers of the current node in the network, and the distance already walked; determining an action definition of the agent according to the selection of the agent in the reinforcement learning model environment to jump from the current node to other nodes next time; determining a reinforcement learning model according to the reinforcement learning model environment, the reward function, the state definition of the agent, and the action definition of the agent.

4. The method of claim 1, wherein, the reward function set based on the reinforcement learning environment, comprising: The reward function The definition formula is: ； wherein subscript represents the target entity, subscript represents the entity represented by the network node at the position where the agent is located, E represents the semantic information of the network node, represents the cosine similarity calculation, represents the length of the path that the agent has walked, represents the number of layers of the network, the number of layers of the network increases from inside to outside, is the number of nodes contained in each layer of the multi-edge structure network, , , is a weight coefficient.

5. The method of claim 1, wherein, before the to-be-reasoned entity is input into the reinforcement learning model for association reasoning processing, the method further comprises pre-training the reinforcement learning model, specifically comprising: acquiring an agent state of an agent in a current network node in the reinforcement learning model; inputting the agent state into the reinforcement learning model to calculate action evaluation values corresponding to the selection of the agent to walk to different network nodes to determine an action evaluation value set; The next step action of the agent is predicted by adopting a greedy strategy according to the action evaluation value set, and the selected action of the agent is determined. The state and reward value of the agent are updated according to the selected action of the agent until the action evaluation value function converges, and the reinforcement learning model is trained.

6. The method of claim 1, wherein, The method comprises the following steps: The method comprises the following steps:

7. A system for reasoning about entity semantic associations based on reinforcement learning, the system comprising: The system comprises: A first module is configured to obtain a standard information document and a to-be-reasoned entity. A second module is configured to perform natural language processing on the standard information document to determine an entity, associated information, and a measurable quantity information quadruple in the standard information document. A third module is configured to perform entity association processing on the entity and the associated information to determine an entity association data set. A fourth module is configured to construct a polygon structure network according to the measurable quantity information quadruple. A fifth module is configured to use the polygon structure network as a reinforcement learning environment, and determine a reinforcement learning model in combination with a reward function set based on the reinforcement learning environment. A sixth module is configured to input the to-be-reasoned entity into the reinforcement learning model for association reasoning processing according to the entity association data set to determine a target entity set. The fourth module is configured to construct a polygon structure network according to the measurable quantity information quadruple, and comprises the following steps: The measurable quantity information quadruple is mapped and filled in combination with a first semantic formula to determine semantic information of a network node. Semantic distance calculation processing is performed on the semantic information of the network node to determine a similar data set. k network nodes are selected from the similar data set to construct a network layer, where k is an integer greater than or equal to 3. Semantic information of a connected edge is determined according to the semantic information of the network node in combination with a second semantic formula. A polygon structure network is determined according to the semantic information of the network node, the semantic information of the connected edge, and the network layer.

8. An electronic device, comprising: The electronic device comprises a memory and a processor. The memory is configured to store a program. The processor executes the program to implement the method of any one of claims 1 to 6.

9. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images