Question and answer result generation method and device based on reinforcement learning, equipment and medium

Abstract

The invention relates to the technical field of intelligent decision making, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a question and answer result generation method, device, equipment and medium based on reinforcement learning, and the method comprises the following steps: obtaining a natural language question of a target user, and carrying out entity extraction on the natural language question to obtain candidate entities; optimizing a preset reinforcement learning model based on a preset business knowledge graph to obtain a path recall model; associating the candidate entity with a standard entity in the business knowledge graph by using the path recall model to obtain a candidate path corresponding to the natural language question; screening the candidate paths by using a preset path evaluation index to obtain a target recall path of the natural language question; and extracting a target entity in the target recall path, and generating a question and answer result of the natural language question according to the target entity. The question and answer result generation accuracy can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology, and in particular to a method, apparatus, device, and medium for generating question-and-answer results based on reinforcement learning. Background Technology

[0002] Knowledge-based question answering (KBQA) systems need to retrieve answers from structured knowledge bases using natural language questions. Path recall is a key step in this process, involving the identification of candidate paths related to the question. Existing path recall technologies typically rely on rule matching, graph traversal algorithms, or embedding-based similarity calculations, aiming to filter paths from knowledge bases containing a large number of entities and relationships, providing a foundation for retrieving answers to complex user questions in scenarios such as insurance claims.

[0003] In the healthcare field, existing path recall technologies struggle to accurately understand medication associations in complex scenarios like "comorbidities," and they are insufficient at capturing path dependencies in multi-hop reasoning, such as "diabetes → complications → contraindications for antihypertensive drug selection." Furthermore, they may recall a large number of irrelevant candidate paths due to path combination explosion, resulting in low accuracy in question-and-answer generation.

[0004] In the fintech business field, existing path recall technology has difficulty handling multi-hop reasoning such as "enterprise type → policy application → loan process → qualification requirements", and cannot accurately capture semantic relationships such as "the correlation between tax refund amount and loan amount" and "the impact of tax credit rating". It may also recall a large number of irrelevant paths due to the complexity of the knowledge base, resulting in low accuracy of question answering results.

[0005] Existing path retrieval technologies suffer from several drawbacks: path combination explosion, leading to an exponential increase in the number of candidate paths and high computational complexity; a semantic gap, making it difficult to accurately capture the complex semantic mapping between natural language questions and structured paths in the knowledge base; insufficient multi-hop reasoning capabilities, failing to effectively capture the dependencies between paths in complex questions; and a lack of a global optimization perspective, selecting paths based solely on local information. These shortcomings result in low efficiency in candidate path selection, poor semantic matching, and consequently, low accuracy in question-answering results. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and medium for generating question-answering results based on reinforcement learning, in order to solve the problem of low accuracy in generating question-answering results.

[0007] Firstly, a method for generating question-answering results based on reinforcement learning is provided, including: Obtain the natural language question from the target user, and perform entity extraction on the natural language question to obtain candidate entities; The path recall model is obtained by optimizing the preset reinforcement learning model based on the preset business knowledge graph. The path recall model is used to associate the candidate entities with the standard entities in the business knowledge graph to obtain the candidate paths corresponding to the natural language question. The candidate paths are filtered using preset path evaluation metrics to obtain the target recall path for the natural language problem; Extract the target entities from the target recall path, and generate the question-and-answer results for the natural language question based on the target entities.

[0008] Secondly, a question-answering result generation device based on reinforcement learning is provided, comprising: The candidate entity extraction module is used to obtain the natural language question of the target user, extract entities from the natural language question, and obtain candidate entities. The path recall model optimization module is used to optimize the preset reinforcement learning model based on the preset business knowledge graph to obtain the path recall model. The candidate path association module is used to associate the candidate entity with the standard entity in the business knowledge graph using the path recall model to obtain the candidate path corresponding to the natural language question. The target recall path filtering module is used to filter the candidate paths using preset path evaluation indicators to obtain the target recall path for the natural language problem. The question-and-answer result generation module is used to extract target entities in the target recall path and generate question-and-answer results for the natural language question based on the target entities.

[0009] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described reinforcement learning-based question-answering result generation method.

[0010] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described reinforcement learning-based question-answering result generation method.

[0011] The above-described solution for generating question-answering results based on reinforcement learning involves obtaining a natural language question from a target user through a client, extracting entities from the natural language question to obtain candidate entities, optimizing a preset reinforcement learning model based on a preset business knowledge graph to obtain a path recall model, associating the candidate entities with standard entities in the business knowledge graph using the path recall model to obtain candidate paths corresponding to the natural language question, filtering the candidate paths using preset path evaluation metrics to obtain the target recall path for the natural language question, extracting target entities from the target recall path, and generating question-answering results for the natural language question based on the target entities. In this invention, reinforcement learning dynamically optimizes the path exploration strategy, combines a semantic-aware reward function to improve the semantic matching degree between natural language and the knowledge base, utilizes multi-hop reasoning modeling to handle path dependencies in complex problems, and comprehensively evaluates path quality through a global optimization framework. This solution can accurately extract question entities, efficiently associate standard entities in the knowledge graph, and generate accurate question-answering results after path evaluation and filtering, significantly improving the accuracy of knowledge base question-answering path recall in insurance and solving the problem of low accuracy in generating question-answering results. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a question-answering result generation method based on reinforcement learning in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a question-answering result generation method based on reinforcement learning in one embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S1; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation of step S4; Figure 5 This is a schematic diagram of a question-answering result generation device based on reinforcement learning in one embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] The question-answering result generation method based on reinforcement learning provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain the target user's natural language question from the client, extract entities from the natural language question to obtain candidate entities, optimize a preset reinforcement learning model based on a preset business knowledge graph to obtain a path recall model, associate the candidate entities with standard entities in the business knowledge graph using the path recall model to obtain candidate paths corresponding to the natural language question, filter the candidate paths using preset path evaluation metrics to obtain the target recall path for the natural language question, extract the target entities in the target recall path, and generate question-and-answer results for the natural language question based on the target entities. In this invention, the path exploration strategy is dynamically optimized through reinforcement learning, the semantic matching degree between natural language and knowledge base is improved by combining a semantic awareness reward function, the path dependency of complex questions is handled by multi-hop reasoning modeling, and the path quality is comprehensively evaluated through a global optimization framework. This solution can accurately extract question entities, efficiently associate standard entities in the knowledge graph, and generate accurate question-and-answer results after path evaluation and filtering, significantly improving the accuracy of knowledge base question-and-answer path recall in insurance. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0016] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a question-answering result generation method based on reinforcement learning provided in an embodiment of the present invention includes the following steps: S1. Obtain the natural language question of the target user, and extract entities from the natural language question to obtain candidate entities.

[0017] In this embodiment of the invention, the target user refers to a user who uses the insurance KBQA (Knowledge Base Question Answering) system to query insurance-related questions; the natural language question is a query request made by the target user in everyday language.

[0018] In detail, by receiving unstructured everyday language query text input by the target user in the insurance KBQA system, the natural language questions of the target user are obtained. The obtained question text will be used as the raw input for subsequent entity extraction and path retrieval processes.

[0019] In this embodiment of the invention, the candidate entities are entities extracted from the natural language question text of the target user. These entities are composed of words that appear in the question, such as "out-of-town accident" and "materials" in the question "What materials are needed for an accident in another place". They are the output of the entity extraction model after processing the question text, and are used for subsequent entity mapping and path retrieval operations in the knowledge graph.

[0020] In this embodiment of the invention, reference is made to Figure 3 As shown, the entity extraction for the natural language problem to obtain candidate entities includes: S31. Perform text normalization processing on the natural language problem to obtain normalized problem text; S32. Use the pre-trained base model to perform semantic parsing on the normalized problem text to obtain potential entity fragments; S33. Perform boundary detection on the potential entity fragments to obtain accurate entity fragments; S34. Determine candidate entities for the natural language problem based on the precise entity fragment.

[0021] In detail, standardized problem text refers to the standardized text obtained after a series of processing steps to unify the format and regulate the content of the original natural language problem. This type of text removes redundant components, non-standard formats, or interfering information from the original expression. Standardizing natural language problems to obtain standardized problem text typically involves operations such as removing punctuation marks, unifying capitalization, handling abbreviations, eliminating stop words, standardizing number or date formats, correcting spelling errors, and unifying terminology. These operations transform the original problem text into a more structurally regular, semantically clear, and compliant standardized text.

[0022] Specifically, the pre-trained base model uses QWen2-72b (a Chinese large language model) as the foundation model, fine-tuned using LORA (Low-Rank Adaptive Algorithm) for entity extraction; latent entity fragments are text segments extracted from the normalized question text that may correspond to entities in the knowledge graph. By inputting the normalized question text into the pre-trained and fine-tuned QWen2-72b entity extraction model, the model's semantic parsing and entity recognition capabilities directly output n candidate entities extracted from the text as latent entity fragments.

[0023] Furthermore, precise entity fragments refer to text fragments whose specific start and end positions and ranges have been determined more precisely through boundary detection operations. Obtaining precise entity fragments by performing boundary detection on potential entity fragments involves using a pre-trained entity extraction model to detect the specific location of potential entity fragments in the text. By identifying the accurate boundaries of the fragment's start and end, the precise range of the entity is located from the normalized problem text, thus obtaining precise entity fragments with clear boundaries and accurate locations.

[0024] Furthermore, based on the precise boundaries determined by the precise entity fragments in the text, the corresponding entity text content is extracted, and these text contents are used as candidate entities, thereby completing the determination of candidate entities for natural language problems.

[0025] For example, in a medical scenario, if the target user's natural language question is "What medicine should I use for a child with a severe cough and a fever of 38.5 degrees Celsius?", the question is first normalized by removing punctuation and uniformly stating "What medicine should I use for a child with a severe cough and a fever of 38.5 degrees Celsius?". Then, a fine-tuned entity extraction model is used to perform semantic parsing on the normalized text to obtain potential entity fragments such as "child", "severe cough", "fever of 38.5 degrees Celsius", and "medication". Next, the model performs boundary detection on these fragments to accurately locate the entity boundaries of "child" corresponding to pediatric patients, "fever of 38.5 degrees Celsius" as a specific temperature symptom, and "medication" as the core need. Finally, based on the detection results, candidate entities such as "child", "fever", and "antipyretic" are determined.

[0026] For example, in a financial scenario, if the target user's natural language question is "I have a three-year fixed deposit and I want to withdraw it early, how is the interest calculated?", the question is normalized to "I have a three-year fixed deposit and I want to withdraw it early, how is the interest calculated?". Using a fine-tuned entity extraction model, the normalized text is semantically parsed to obtain potential entity fragments such as "three-year fixed deposit," "early withdrawal," and "interest calculation." The model then performs boundary detection on these fragments to accurately locate the entity boundaries for "three-year fixed deposit," which corresponds to the financial product type; "early withdrawal," which is the operational behavior; and "interest calculation," which is the core requirement. Finally, based on the detection results, candidate entities such as "early withdrawal of fixed deposit," "interest calculation formula," and "three-year deposit interest rate" are determined.

[0027] S2. Optimize the preset reinforcement learning model based on the preset business knowledge graph to obtain the path recall model.

[0028] In this embodiment of the invention, the preset business knowledge graph, namely the preset insurance knowledge graph, is a graph structure (such as a triple <Children's Million-Dollar Medical Insurance, Applicable Population, Students>) that represents insurance domain knowledge as nodes (entities) and edges (relationships) to carry structured knowledge; the preset reinforcement learning model is a model architecture composed of an initialized Q-network (used to estimate the value of actions in the current state), a target network (used to provide the target Q-value), an experience storage unit, and set hyperparameters (such as a discount factor of 0.99); the path recall model is a model that, after being trained and optimized through interaction between the reinforcement learning model and the business knowledge graph environment, can recall the optimal relevant path from the knowledge graph based on the problem entity.

[0029] In this embodiment of the invention, the optimization of a preset reinforcement learning model based on a preset business knowledge graph to obtain a path recall model includes: The pre-defined business knowledge graph is structured and modeled to obtain a graph structure representation; The state, action, and reward of the preset reinforcement learning model are parameterized according to the graph structure representation to obtain the model definition parameters; The interaction rules between the business knowledge graph and the reinforcement learning model are generated based on the model definition parameters. The network architecture and experience storage unit in the reinforcement learning model are initialized and configured according to the interaction rules to obtain the basic components of the model. Collect state transition data during the interaction between the basic components of the model and the graph structure representation, and store the state transition data in the experience storage unit; The state transition data is randomly sampled from the experience storage unit, and the weight parameters of the reinforcement learning model are iteratively optimized using the state transition data to obtain the path recall model.

[0030] In detail, the graph structure representation models the business knowledge graph as a graph structure where nodes represent entities and edges represent relationships. The pre-defined state of the reinforcement learning model is a vector formed by fusing the current problem and nodes through vector encoding and the Hadamard product (an operation of multiplying corresponding elements of a matrix). The action is to select an adjacent node from the current node as the next strategy. The reward is a positive reward when reaching the target node and stopping, and a negative reward otherwise. The model definition parameters are the parameterized definitions of the state, action, and reward in the above manner. By transforming the business knowledge graph into a graph structure composed of nodes and edges, and based on the connection relationships between entities (nodes) and relationships (edges) in this graph structure, the state, action, and reward of the reinforcement learning model are clearly parameterized, thus completing the process from structured modeling of the knowledge graph to the definition of key elements of the reinforcement learning model.

[0031] Specifically, interaction rules are the interaction mechanisms between the business knowledge graph and the reinforcement learning model, determined by the model definition parameters. These rules regulate the specific rules governing information transmission and processing between the two. The network architecture is the structural design of the neural network in the reinforcement learning model, including the composition, connection methods, and parameter configurations of each layer (such as convolutional layers and fully connected layers). The experience storage unit, or replay buffer, stores experience samples (states, actions, rewards, next states, etc.) generated by the reinforcement learning agent during the interaction process for reuse in subsequent training. The basic model components are the fundamental building blocks formed after initializing the parameters of the network architecture and experience storage units in the reinforcement learning model according to the interaction rules. By constructing specific rules for the interaction between the knowledge graph and the reinforcement learning model based on the model definition parameters, and then applying these rules to the parameter initialization of the neural network structure in the reinforcement learning model (such as the number of neurons in each layer, weight initialization methods, etc.) and the parameter configuration of the experience storage units (such as storage capacity, sampling strategy parameters, etc.), the basic model components are constructed.

[0032] Furthermore, state transition data refers to the data related to state changes generated during the interaction between the model's basic components and the graph representation, including information such as state, action, reward, and next state. Weight parameters are learnable parameters of the neural network in the reinforcement learning model, determining the model's decision-making ability. By collecting state transition data generated during the interaction between the model's basic components and the graph representation and storing it in an empirical storage unit, and then randomly sampling this data from the empirical storage unit, the weight parameters of the reinforcement learning model are iteratively optimized using the sampled data, ultimately resulting in a path recall model.

[0033] Furthermore, when randomly sampling the state transition data in the experience storage unit, a target Q-value needs to be calculated for each sample as a reference for updating the Q-network parameters. The specific formula is: ,in, These are the parameters of the target network.

[0034] Furthermore, the candidate path recall model training module is built around reinforcement learning: First, the problem environment is defined, and the business knowledge graph is represented as an entity-relationship graph. The state is a vector of "problem + node" (fused by Hadamard product). The action is to select adjacent nodes, and the reward is determined by whether the target node is reached. Then, the Q network, the target network (with consistent initial parameters), and the experience replay buffer are initialized, and hyperparameters such as the discount factor (γ=0.99) are set. During training, the system interacts with the environment and stores experience through an ε-greedy strategy (a decision strategy used to balance "exploration" and "exploitation"). The system randomly samples the state transition data from the experience storage unit to calculate the target Q value (the formula integrates reward and future value), updates the Q network, and periodically synchronizes the target network parameters.

[0035] S3. Using the path recall model, the candidate entity is associated with the standard entity in the business knowledge graph to obtain the candidate path corresponding to the natural language question.

[0036] In this embodiment of the invention, the standard entity is a normalized entity in the business knowledge graph entity library that corresponds to the candidate entity extracted from the user question; the candidate path is a path to be screened that is formed by starting from the standard entity in the knowledge graph through relational connections using a path recall model, and is related to the natural language question.

[0037] In this embodiment of the invention, the step of using the path recall model to associate the candidate entity with the standard entity in the business knowledge graph to obtain the candidate path corresponding to the natural language question includes: The candidate entities are mapped using the business knowledge graph to obtain standard entity nodes. Identify the neighboring nodes of the standard entity node, aggregate the neighboring nodes, and obtain the set of neighboring nodes of the standard entity node; The path recall model is used to calculate the action value of each node in the set of adjacent nodes, and the target action node of the standard entity node is determined based on the action value. The target action node is updated to the standard entity node, and the process returns to the step of identifying the neighboring nodes of the standard entity node until all standard entity nodes have been processed, and an entity relationship path is generated based on all target action nodes. Candidate paths corresponding to the natural language problem are determined based on the entity relationship paths.

[0038] In detail, standard entity nodes are graph structure nodes in the business knowledge graph used to represent normalized entities (entities exist in the knowledge graph as nodes). The entity relationship linking module matches and maps the candidate entities output by the problem entity extraction module with entities in the business knowledge graph entity library, thereby finding the node corresponding to the normalized entity of the candidate entity in the knowledge graph.

[0039] Specifically, an adjacent node refers to a node in the knowledge graph that is directly connected to a standard entity node via an edge; the adjacent node set is the set formed by aggregating these adjacent nodes. By identifying all nodes directly connected to a standard entity node in the knowledge graph via edges, and then aggregating these identified nodes, the adjacent node set of the standard entity node is obtained.

[0040] Furthermore, action value is a quantitative metric calculated by the path recall model for each node in the set of neighboring nodes. It measures the contribution or relevance of that node to solving the natural language problem when associated with the current standard entity node in the knowledge graph. The target action node is the optimal node determined from the set of neighboring nodes based on its action value; that is, the node with the highest action value and the most likely to help form an effective answer path. The path recall model calculates the action value for each node in the set of neighboring nodes of the standard entity node. This calculation may be based on factors such as the node's position in the knowledge graph, its semantic relevance to the question, and the weight of the connecting edges. Then, by comparing the action values ​​of each node, the node with the highest action value is determined as the target action node.

[0041] Furthermore, the entity relationship path is a path formed by connecting standard entity nodes through relationship edges in the business knowledge graph, used to represent the associations between entities in natural language problems. The currently determined target action node is converted into a new standard entity node. The neighboring nodes of this new standard entity node are re-identified and aggregated into a set. Then, the action value of each node in the neighboring node set is calculated using a path recall model to determine the new target action node. This process is repeated until all standard entity nodes have been processed. Finally, the entity relationship path is generated based on the connection order of all target action nodes.

[0042] Furthermore, based on the generated entity relationship paths, they are matched with the semantics of the natural language question to analyze whether the paths cover the entities and relationships mentioned in the question. This allows us to select paths that meet the requirements of the question as candidate paths. These candidate paths can reflect the relationships between the entities involved in the question and provide possible relationship path options for subsequent question answers.

[0043] For example, in a medical scenario, when a user asks "What tests do diabetic patients need to undergo?", the system first extracts candidate entities such as "diabetes" and "tests" from the question, and obtains the standard entity node of "diabetes" through medical knowledge graph mapping; it identifies its adjacent nodes such as "blood glucose test" and "glycated hemoglobin test" and aggregates them into a set; it uses a path recall model to calculate the action value of each node, determines "blood glucose test" as the target action node, updates it as a new standard entity node, and repeats the above process until all nodes are completed, generating entity relationship paths such as "diabetes → endocrine test → blood glucose test", and determines the candidate path corresponding to the question.

[0044] For example, in a financial scenario, when a user asks "What materials are needed for a personal housing loan application?", the system extracts candidate entities such as "personal housing loan" and "materials" from the question. It then maps these to a standard entity node for "personal housing loan" using a financial knowledge graph. Next, it identifies adjacent nodes such as "income certificate," "credit report," and "purchase contract" and aggregates them into a set. Using a path recall model, it calculates the action value of each node, identifies "income certificate" as the target action node, updates it to a new standard entity node, and repeats the process until all nodes are processed. This generates entity relationship paths such as "personal housing loan → application materials → income certificate," thereby determining the candidate path corresponding to the question.

[0045] S4. The candidate paths are screened using preset path evaluation metrics to obtain the target recall path for the natural language problem.

[0046] In this embodiment of the invention, the preset path evaluation index is a reward function designed by combining natural language questions and knowledge base path semantics, as well as factors such as path quality, semantic matching degree and computational efficiency. The target recall path is the optimal path obtained by screening candidate paths through the above evaluation index, which meets the requirements of semantic matching, multi-hop reasoning and takes into account computational efficiency.

[0047] In this embodiment of the invention, reference is made to Figure 4 As shown, the step of using preset path evaluation metrics to filter the candidate paths to obtain the target recall path for the natural language problem includes: S41. Perform multi-dimensional index analysis on the candidate paths using preset path evaluation indicators to obtain a comprehensive evaluation result for each candidate path. S42. Based on the semantic matching degree in the comprehensive evaluation results, the candidate paths are filtered to determine the target candidate path; S43. Based on the comprehensive score in the comprehensive evaluation result, the target candidate paths are prioritized to obtain a candidate path priority sequence; S44. Based on a preset recall path quantity threshold, the candidate paths in the candidate path priority sequence are filtered, and the filtered candidate paths are used as the target recall paths for the natural language problem.

[0048] In detail, the comprehensive evaluation result is a quantitative evaluation score of the candidate paths under multiple dimensions such as path quality, semantic matching degree, and computational efficiency. Using a reward function designed by combining natural language questions with knowledge base path semantics, and pre-defined path evaluation indicators that comprehensively consider factors such as path quality, semantic matching degree, and computational efficiency, multi-dimensional indicator analysis is performed on the candidate paths to obtain the comprehensive evaluation score for each candidate path.

[0049] Specifically, semantic matching degree is a quantitative value of semantic matching ability reflected in the reward function designed by combining the semantics of natural language questions and knowledge base paths; target candidate paths are candidate paths that meet the semantic requirements of natural language questions, selected based on the semantic matching degree index in the comprehensive evaluation results. Based on the semantic matching metric determined by the reward function designed by combining the semantics of natural language questions and knowledge base paths, the semantic matching degree index in the comprehensive evaluation results of candidate paths is compared and analyzed, and candidate paths that reach or exceed the preset semantic matching degree threshold are selected as target candidate paths.

[0050] Furthermore, the comprehensive score is a quantitative score obtained from the multi-dimensional analysis of candidate paths in the comprehensive evaluation results, used to measure the overall quality of the paths; the candidate path priority sequence is an ordered list formed by sorting the target candidate paths from high to low according to the comprehensive score, reflecting the priority level of each path. By performing a priority sorting operation on the target candidate paths based on the comprehensive score value in the comprehensive evaluation results, a candidate path priority sequence arranged in descending order of score is generated.

[0051] Furthermore, the preset threshold for the number of recall paths is a pre-set critical value used to limit the final number of recall paths. Based on this critical value, the paths in the priority sequence of candidate paths are filtered by quantity, and the candidate paths that rank in the top threshold number in the sequence are selected. These selected paths are determined as the target recall paths for the natural language problem.

[0052] For example, in a medical scenario, in response to a user's question, "What side effects should diabetic patients be aware of when taking metformin?", the candidate path recall model generates multiple candidate paths involving drug metabolism, interactions, contraindications, and other related aspects. Using pre-defined path evaluation metrics, the candidate paths are analyzed across multiple dimensions to obtain a comprehensive evaluation result for each path in terms of path quality, semantic matching degree, and computational efficiency. Based on the semantic matching degree in the comprehensive evaluation result, target candidate paths highly relevant to the semantics of "side effects" are selected. Then, based on the comprehensive score in the comprehensive evaluation result, the target candidate paths are prioritized, forming a priority sequence of candidate paths arranged from highest to lowest score. Finally, based on a pre-defined threshold for the number of recall paths (e.g., 5 paths), the top-ranked candidate paths are selected from the priority sequence and used as the target recall paths for this natural language question.

[0053] For example, in a financial scenario, in response to a user's question, "Which low-risk financial products are suitable for conservative investors?", the candidate path recall model generates candidate paths involving product types such as money market funds, government bonds, and bank wealth management products. Using pre-defined path evaluation metrics, the candidate paths are analyzed from multiple dimensions to obtain a comprehensive evaluation result for each path in terms of product risk level, semantic matching degree, and computational efficiency. Based on semantic matching degree, target candidate paths that clearly point to low-risk products are selected. These target candidate paths are then ranked according to their comprehensive scores to form a priority sequence. Finally, based on a pre-defined threshold (e.g., 3 paths), the top-ranked paths are selected from the sequence as the target recall paths for this question.

[0054] S5. Extract the target entities in the target recall path, and generate the question-and-answer results for the natural language question based on the target entities.

[0055] In this embodiment of the invention, the target entity is the tail entity after the path recall model recalls the path, that is, the entity returned as the answer.

[0056] In this embodiment of the invention, extracting the target entity in the target recall path includes: The target recall path is structurally parsed to obtain a structured path relationship graph, and the set of entity nodes of the target recall path is determined based on the node connection features of the structured path relationship graph. For each node in the entity node set, multidimensional feature weights are calculated to obtain an importance score; The target entity is selected from the set of entity nodes based on the importance score.

[0057] In detail, a structured path relationship graph is a graph structure obtained by structurally parsing the target retrieval path, where nodes represent entities and edges represent relationships. Node connection features are the connection methods between nodes in the graph (such as the relationship type represented by the edge, multi-hop connections, etc.). The entity node set is the set of all nodes representing entities in the graph. By performing structural parsing on the target retrieval path to obtain a structured path relationship graph composed of nodes and edges, and then determining all nodes representing entities in the graph based on the features reflected in the connection methods between nodes, an entity node set is formed.

[0058] Specifically, the importance score is a numerical value used to measure the importance of a node, calculated by weighting the multidimensional features of each node in the entity node set (such as the node's position in the path, the strength of the association, and the frequency of occurrence). For each node in the entity node set, features are extracted from multiple dimensions and corresponding weights are assigned to each feature. A specific algorithm is used to comprehensively calculate the multidimensional features and their weights to obtain the importance score for each node.

[0059] Furthermore, selecting target entities from the set of entity nodes based on importance scores means selecting nodes that meet preset conditions (such as scores higher than the threshold or higher ranking) from the set of entity nodes based on the importance scores of each node, by setting specific score thresholds or sorting them according to their scores. These selected nodes are the target entities.

[0060] In this embodiment of the invention, the question-and-answer result refers to the content generated by integrating relevant information based on the extracted target entity, which can directly answer natural language questions. This content is presented in the form of natural language expression and aims to provide accurate and effective answers to the questions raised by users.

[0061] In this embodiment of the invention, generating the question-and-answer result of the natural language question based on the target entity includes: Extract the semantic elements of the target entity in the business knowledge graph, and determine the entity semantic information of the target entity based on the semantic elements; The entity semantic information is matched with the intent information of the natural language question to obtain the target semantic information; Generate a structured answer framework corresponding to the natural language question based on the target semantic information; Using a preset natural language generation template, the structured answer framework is converted into initial question-and-answer result text; Logical validation is performed on the initial question-and-answer result text to obtain the question-and-answer result for the natural language question.

[0062] In detail, semantic elements are the basic constituent elements of a business knowledge graph, such as attributes, relationships, and types, that represent the semantic characteristics of a target entity. Entity semantic information is a structured semantic description formed by combining semantic elements, reflecting the connotation and associations of the target entity. By extracting semantic elements such as attribute values, relationship edges, and entity types corresponding to the target entity from the business knowledge graph, and then integrating and semantically parsing these elements, the complete entity semantic information of the target entity can be determined.

[0063] Specifically, intent information refers to the user's query purpose or need implied in the natural language question (such as inquiring about insurance product types, claim conditions, etc.); target semantic information is the semantic content that matches the question's intent, obtained by matching entity semantic information with intent information. By analyzing the natural language question to clarify its intent information, and then comparing entity semantic information with this intent information at the semantic level (such as similarity calculation, keyword matching, etc.), the part that matches the intent information is extracted or filtered from the entity semantic information, thus forming the target semantic information.

[0064] Furthermore, a structured answer framework is a framework structure that organizes answer content according to a certain logical structure (such as points, levels, or items) to clearly present the core points and hierarchical relationships of the answer. Based on the semantic content of the target semantic information, such as entity attributes, relationships, and types, as well as the intent of the natural language question, this information is organized into a hierarchical and orderly framework structure according to logical order or specific rules (such as primary and secondary, or categories), so that the key points of the answer are presented in a structured form.

[0065] Furthermore, the preset natural language generation template is a pre-designed text template containing answer content organization format and semantic element placeholders; the initial question-and-answer result text is natural language text generated by filling the semantic elements of the structured answer framework into the template placeholders. By mapping the semantic points such as entity attributes, relationships, and types in the structured answer framework to the corresponding placeholder positions in the template, the template automatically fills in the content according to preset grammatical rules and expression logic, thereby converting it into initial question-and-answer result text that conforms to natural language expression habits.

[0066] In addition, logical verification of the initial question-and-answer result text is performed by using preset logical rules (such as checking whether the entity relationships in the answer are contradictory, whether the attribute values ​​conform to business rules, and whether there are logical gaps in the semantic expression) or by comparing with a knowledge base to verify the semantic accuracy, information consistency, and logical coherence of the text, correct logical errors or information deviations, and thus obtain question-and-answer results that conform to logical norms and can accurately answer natural language questions.

[0067] As can be seen, the above scheme utilizes reinforcement learning to dynamically optimize path exploration strategies, combines a semantic-aware reward function to improve the semantic matching degree between natural language and the knowledge base, leverages multi-hop reasoning modeling to handle path dependencies in complex problems, and comprehensively evaluates path quality through a global optimization framework. This scheme can accurately extract question entities, efficiently associate standard entities from the knowledge graph, and generate accurate question-answering results after path evaluation and filtering, significantly improving the accuracy of knowledge base question-answering path retrieval in insurance.

[0068] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0069] In one embodiment, a reinforcement learning-based question-answering result generation device is provided, which corresponds one-to-one with the reinforcement learning-based question-answering result generation method described in the above embodiments. For example... Figure 5As shown, the reinforcement learning-based question-answering result generation device 100 includes a candidate entity extraction module 101, a path recall model optimization module 102, a candidate path association module 103, a target recall path filtering module 104, and a question-answering result generation module 105. Detailed descriptions of each functional module are as follows: The candidate entity extraction module 101 is used to obtain the natural language question of the target user, extract entities from the natural language question, and obtain candidate entities. The path recall model optimization module 102 is used to optimize the preset reinforcement learning model based on the preset business knowledge graph to obtain the path recall model. The candidate path association module 103 is used to associate the candidate entity with the standard entity in the business knowledge graph using the path recall model to obtain the candidate path corresponding to the natural language question. The target recall path filtering module 104 is used to filter the candidate paths using preset path evaluation indicators to obtain the target recall path of the natural language problem. The question-and-answer result generation module 105 is used to extract the target entity in the target recall path and generate the question-and-answer result of the natural language question based on the target entity.

[0070] In one embodiment, the candidate entity extraction module 101, when performing entity extraction on the natural language question to obtain candidate entities, is used to: The natural language problem is subjected to text normalization processing to obtain normalized problem text; The pre-trained base model is used to perform semantic parsing on the normalized problem text to obtain potential entity fragments; Boundary detection is performed on the potential entity fragments to obtain accurate entity fragments; Candidate entities for the natural language problem are determined based on the precise entity fragments.

[0071] In one embodiment, the path recall model optimization module 102, when optimizing a preset reinforcement learning model based on a preset business knowledge graph to obtain a path recall model, is used to: The pre-defined business knowledge graph is structured and modeled to obtain a graph structure representation; The state, action, and reward of the preset reinforcement learning model are parameterized according to the graph structure representation to obtain the model definition parameters; The interaction rules between the business knowledge graph and the reinforcement learning model are generated based on the model definition parameters. The network architecture and experience storage unit in the reinforcement learning model are initialized and configured according to the interaction rules to obtain the basic components of the model. Collect state transition data during the interaction between the basic components of the model and the graph structure representation, and store the state transition data in the experience storage unit; The state transition data is randomly sampled from the experience storage unit, and the weight parameters of the reinforcement learning model are iteratively optimized using the state transition data to obtain the path recall model.

[0072] In one embodiment, the candidate path association module 103, when performing the operation of associating the candidate entity with the standard entity in the business knowledge graph using the path retrieval model to obtain the candidate path corresponding to the natural language question, is configured to: The candidate entities are mapped using the business knowledge graph to obtain standard entity nodes. Identify the neighboring nodes of the standard entity node, aggregate the neighboring nodes, and obtain the set of neighboring nodes of the standard entity node; The path recall model is used to calculate the action value of each node in the set of adjacent nodes, and the target action node of the standard entity node is determined based on the action value. The target action node is updated to the standard entity node, and the process returns to the step of identifying the neighboring nodes of the standard entity node until all standard entity nodes have been processed, and an entity relationship path is generated based on all target action nodes. Candidate paths corresponding to the natural language problem are determined based on the entity relationship paths.

[0073] In one embodiment, the target recall path filtering module 104, when performing the filtering of candidate paths using preset path evaluation metrics to obtain the target recall path for the natural language problem, is configured to: The candidate paths are analyzed using preset path evaluation indicators to obtain a comprehensive evaluation result for each candidate path. The candidate paths are filtered based on the semantic matching degree in the comprehensive evaluation results to determine the target candidate path; The candidate paths are prioritized based on the comprehensive scores in the comprehensive evaluation results to obtain a candidate path priority sequence. Candidate paths in the priority sequence of candidate paths are filtered according to a preset recall path quantity threshold, and the filtered candidate paths are used as the target recall paths for the natural language problem.

[0074] In one embodiment, the question-and-answer result generation module 105, when performing the extraction of target entities in the target recall path, is used to: The target recall path is structurally parsed to obtain a structured path relationship graph, and the set of entity nodes of the target recall path is determined based on the node connection features of the structured path relationship graph. For each node in the entity node set, multidimensional feature weights are calculated to obtain an importance score; The target entity is selected from the set of entity nodes based on the importance score.

[0075] In one embodiment, the question-and-answer result generation module 105, when generating the question-and-answer result for the natural language question based on the target entity, is further configured to: Extract the semantic elements of the target entity in the business knowledge graph, and determine the entity semantic information of the target entity based on the semantic elements; The entity semantic information is matched with the intent information of the natural language question to obtain the target semantic information; Generate a structured answer framework corresponding to the natural language question based on the target semantic information; Using a preset natural language generation template, the structured answer framework is converted into initial question-and-answer result text; Logical validation is performed on the initial question-and-answer result text to obtain the question-and-answer result for the natural language question.

[0076] This invention provides a question-answering result generation device based on reinforcement learning. It dynamically optimizes path exploration strategies through reinforcement learning, enhances the semantic matching degree between natural language and the knowledge base by combining a semantically aware reward function, handles path dependencies for complex problems using multi-hop reasoning modeling, and comprehensively evaluates path quality through a global optimization framework. This solution can accurately extract question entities, efficiently associate standard entities from the knowledge graph, and generate accurate question-answering results after path evaluation and filtering, significantly improving the accuracy of knowledge base question-answering path retrieval in the insurance industry.

[0077] Specific limitations regarding the reinforcement learning-based question-answering result generation device can be found in the limitations of the reinforcement learning-based question-answering result generation method described above, and will not be repeated here. Each module in the aforementioned reinforcement learning-based question-answering result generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0078] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a reinforcement learning-based question-answering result generation method on the server side.

[0079] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a reinforcement learning-based question-answering result generation method.

[0080] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the natural language question from the target user, and perform entity extraction on the natural language question to obtain candidate entities; The path recall model is obtained by optimizing the preset reinforcement learning model based on the preset business knowledge graph. The path recall model is used to associate the candidate entities with the standard entities in the business knowledge graph to obtain the candidate paths corresponding to the natural language question. The candidate paths are filtered using preset path evaluation metrics to obtain the target recall path for the natural language problem; Extract the target entities from the target recall path, and generate the question-and-answer results for the natural language question based on the target entities.

[0081] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the natural language question from the target user, and perform entity extraction on the natural language question to obtain candidate entities; The path recall model is obtained by optimizing the preset reinforcement learning model based on the preset business knowledge graph. The path recall model is used to associate the candidate entities with the standard entities in the business knowledge graph to obtain the candidate paths corresponding to the natural language question. The candidate paths are filtered using preset path evaluation metrics to obtain the target recall path for the natural language problem; Extract the target entities from the target recall path, and generate the question-and-answer results for the natural language question based on the target entities.

[0082] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0083] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0085] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

[0086] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A question-answering result generation method based on reinforcement learning, characterized in that, include: Obtain the natural language question from the target user, and perform entity extraction on the natural language question to obtain candidate entities; The path recall model is obtained by optimizing the preset reinforcement learning model based on the preset business knowledge graph. The path recall model is used to associate the candidate entities with the standard entities in the business knowledge graph to obtain the candidate paths corresponding to the natural language question. The candidate paths are filtered using preset path evaluation metrics to obtain the target recall path for the natural language problem; Extract the target entities from the target recall path, and generate the question-and-answer results for the natural language question based on the target entities.

2. The question-answering result generation method based on reinforcement learning as described in claim 1, characterized in that, The entity extraction process for the natural language problem, yielding candidate entities, includes: The natural language problem is subjected to text normalization processing to obtain normalized problem text; The pre-trained base model is used to perform semantic parsing on the normalized problem text to obtain potential entity fragments; Boundary detection is performed on the potential entity fragments to obtain accurate entity fragments; Candidate entities for the natural language problem are determined based on the precise entity fragments.

3. The question-answering result generation method based on reinforcement learning as described in claim 1, characterized in that, The optimization of the preset reinforcement learning model based on the preset business knowledge graph to obtain the path recall model includes: The pre-defined business knowledge graph is structured and modeled to obtain a graph structure representation; The state, action, and reward of the preset reinforcement learning model are parameterized according to the graph structure representation to obtain the model definition parameters; The interaction rules between the business knowledge graph and the reinforcement learning model are generated based on the model definition parameters. The network architecture and experience storage unit in the reinforcement learning model are initialized and configured according to the interaction rules to obtain the basic components of the model. Collect state transition data during the interaction between the basic components of the model and the graph structure representation, and store the state transition data in the experience storage unit; The state transition data is randomly sampled from the experience storage unit, and the weight parameters of the reinforcement learning model are iteratively optimized using the state transition data to obtain the path recall model.

4. The question-answering result generation method based on reinforcement learning as described in claim 1, characterized in that, The step of using the path recall model to associate the candidate entities with standard entities in the business knowledge graph to obtain the candidate paths corresponding to the natural language question includes: The candidate entities are mapped using the business knowledge graph to obtain standard entity nodes. Identify the neighboring nodes of the standard entity node, aggregate the neighboring nodes, and obtain the set of neighboring nodes of the standard entity node; The path recall model is used to calculate the action value of each node in the set of adjacent nodes, and the target action node of the standard entity node is determined based on the action value. The target action node is updated to the standard entity node, and the process returns to the step of identifying the neighboring nodes of the standard entity node until all standard entity nodes have been processed, and an entity relationship path is generated based on all target action nodes. Candidate paths corresponding to the natural language problem are determined based on the entity relationship paths.

5. The question-answering result generation method based on reinforcement learning as described in claim 1, characterized in that, The step of filtering candidate paths using preset path evaluation metrics to obtain the target recall path for the natural language problem includes: The candidate paths are analyzed using preset path evaluation indicators to obtain a comprehensive evaluation result for each candidate path. The candidate paths are filtered based on the semantic matching degree in the comprehensive evaluation results to determine the target candidate path; The candidate paths are prioritized based on the comprehensive scores in the comprehensive evaluation results to obtain a candidate path priority sequence. Candidate paths in the priority sequence of candidate paths are filtered according to a preset recall path quantity threshold, and the filtered candidate paths are used as the target recall paths for the natural language problem.

6. The question-answering result generation method based on reinforcement learning as described in claim 1, characterized in that, The extraction of target entities from the target recall path includes: The target recall path is structurally parsed to obtain a structured path relationship graph, and the set of entity nodes of the target recall path is determined based on the node connection features of the structured path relationship graph. For each node in the entity node set, multidimensional feature weights are calculated to obtain an importance score; The target entity is selected from the set of entity nodes based on the importance score.

7. The question-answering result generation method based on reinforcement learning as described in claim 1, characterized in that, The step of generating the question-and-answer result for the natural language question based on the target entity includes: Extract the semantic elements of the target entity in the business knowledge graph, and determine the entity semantic information of the target entity based on the semantic elements; The entity semantic information is matched with the intent information of the natural language question to obtain the target semantic information; Generate a structured answer framework corresponding to the natural language question based on the target semantic information; Using a preset natural language generation template, the structured answer framework is converted into initial question-and-answer result text; Logical validation is performed on the initial question-and-answer result text to obtain the question-and-answer result for the natural language question.

8. A question-answering result generation device based on reinforcement learning, characterized in that, include: The candidate entity extraction module is used to obtain the natural language question of the target user, extract entities from the natural language question, and obtain candidate entities. The path recall model optimization module is used to optimize the preset reinforcement learning model based on the preset business knowledge graph to obtain the path recall model. The candidate path association module is used to associate the candidate entity with the standard entity in the business knowledge graph using the path recall model to obtain the candidate path corresponding to the natural language question. The target recall path filtering module is used to filter the candidate paths using preset path evaluation indicators to obtain the target recall path for the natural language problem. The question-and-answer result generation module is used to extract target entities in the target recall path and generate question-and-answer results for the natural language question based on the target entities.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the question-answering result generation method based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the question-answering result generation method based on reinforcement learning as described in any one of claims 1 to 7.

Images