Multi-aspect reasoning graph method of NLP interaction model based on reinforcement learning

Abstract

The invention discloses a multi-aspect reasoning graph method of an NLP interaction model based on reinforcement learning. The method comprises the following steps: extracting entities and relationships from an input text to construct an initial reasoning graph comprising a plurality of reasoning aspects; a reinforcement learning agent is initialized, a state space of the reinforcement learning agent comprises a graph state and user query, and an action space comprises reasoning operation on a graph; the agent updates a graph or generates a path based on a state selection action, and improves performance through continuous interaction based on a comprehensive reward signal optimization policy; according to the method, the limitation that the prior art focuses on a single reasoning dimension is overcome by constructing the multi-dimensional reasoning graph integrating semantics, logic, context and the like and cooperatively operating the sub-graphs by utilizing the reinforcement learning agent, so that the model can carry out all-dimensional and multi-angle deep analysis on complex queries, and the accuracy of the analysis is improved. The reasoning comprehensiveness and the comprehensive judgment accuracy are greatly improved, and therefore more accurate and complete responses are given in tasks such as questions and answers or dialogues.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a multi-aspect reasoning graph method of an NLP interaction model based on reinforcement learning. BACKGROUND

[0002] Natural language processing (NLP) technology is to enable computers to understand, interpret and generate human language, which is applied in intelligent question answering, automatic dialogue system, knowledge retrieval and other fields. With the complication of application scenarios, complex reasoning tasks that require multi-step logical deduction, context association and multi-dimensional semantic analysis have become one of the core challenges in the current NLP field. Traditional sequence modeling methods often have difficulty in explicitly capturing and maintaining the rich structure and logical relationship between knowledge entities when processing such tasks, thereby limiting the deep reasoning ability of the model. In recent years, the method combining reinforcement learning and structured representation has provided a new idea for coping with this challenge. Through the interaction between the agent and the environment, the optimal decision strategy is learned, which is expected to realize more efficient and accurate semantic understanding and reasoning.

[0003] Many existing methods focus on single-aspect reasoning (such as only logical reasoning or only semantic matching), and lack a unified framework to cooperatively process reasoning information in multiple dimensions such as semantics, logic, context and even emotion, resulting in difficulty in giving comprehensive and accurate responses when facing comprehensive user queries. At the same time, the existing reasoning process is often a "black box" operation, and the internal state and decision path are not transparent, which not only makes the model less interpretable, but also makes the reasoning process difficult to dynamically adjust and optimize according to immediate feedback. And the existing method usually relies heavily on a large amount of high-quality labeled data for supervised training, and for reasoning tasks that require flexible use of knowledge and rules, the cost of constructing such data is extremely high and the coverage is limited.

[0004] Therefore, in view of the above problems, the present application provides a multi-aspect reasoning graph method of an NLP interaction model based on reinforcement learning. SUMMARY

[0005] In order to overcome the problems of multi-aspect reasoning cooperation, model interpretability and dynamic interaction optimization, the present application provides a multi-aspect reasoning graph method of an NLP interaction model based on reinforcement learning, which constructs a dynamic graph structure integrating multiple reasoning aspects, and uses a reinforcement learning agent to learn how to operate and optimize the graph in continuous interaction with the environment, thereby realizing more comprehensive, robust and controllable natural language deep reasoning.

[0006] The technical scheme of the present application is: a multi-aspect reasoning graph method of an NLP interaction model based on reinforcement learning, comprising the following steps: S1, extracting a plurality of entities and relationships from an input natural language text, and constructing an initial multi-aspect reasoning graph, wherein the multi-aspect reasoning graph comprises a plurality of sub-graphs, each sub-graph corresponding to an aspect of reasoning, the aspect of reasoning including a semantic aspect, a logical aspect and a contextual aspect, wherein the semantic aspect represents semantic roles and meanings of the text, the logical aspect represents logical relationships and reasoning rules, and the contextual aspect represents dialogue history or text context information; S2, initializing a reinforcement learning agent, wherein the state space of the reinforcement learning agent comprises a current graph state of the multi-aspect reasoning graph, a vector representation of a user query and a user historical interaction record, and the action space comprises reasoning operations performed on the multi-aspect reasoning graph, the reasoning operations including node expansion, edge update, path traversal or sub-graph selection, wherein the node expansion is used to add new entities or concepts, the edge update is used to modify relationships between entities, the path traversal is used to generate multi-hop reasoning chains, and the sub-graph selection is used to activate a sub-graph of a specific aspect of reasoning; S3, interacting with the user through the reinforcement learning agent, receiving a natural language query input by the user, encoding the user query into a vector representation, combining the current graph state in the state space, using a policy network to select an action, and executing the selected action to update the multi-aspect reasoning graph or generate a reasoning path; S4, calculating a reward signal based on the result after executing the action, wherein the reward signal integrates reasoning accuracy, response relevance, user feedback and external knowledge base verification, wherein the reasoning accuracy is calculated by comparing the similarity between the generated response and the standard answer, the response relevance is evaluated by cosine similarity, the user feedback is obtained by explicit scoring or implicit behavior data, and the external knowledge base verification confirms the correctness of reasoning by querying a knowledge graph; S5, using the reward signal to update the policy network and value network of the reinforcement learning agent through a reinforcement learning algorithm to optimize the reasoning strategy, wherein the reinforcement learning algorithm includes a deep Q network, a policy gradient method or an actor-critic method; S6, repeating steps S3 to S5 to realize continuous learning and interaction optimization, thereby improving the performance of natural language processing tasks.

[0007] Preferably, in step S1, the extraction of entities and relationships adopts a combination of pre-trained language models and domain-specific dictionaries to ensure comprehensiveness and accuracy; it is worth noting that the initial structure of the multi-aspect reasoning graph can be preheated based on a general knowledge graph to accelerate model convergence.

[0008] Preferably, the reward signal in step S4 is a composite reward function, the weights of which can be dynamically adjusted according to specific application scenarios; for example, in a question-answering system, the weight of reasoning accuracy is high; in a casual conversation system, the weights of response relevance and user feedback are high; it is worth noting that the introduction of a sparse reward processing mechanism also gives positive rewards to key intermediate steps that lead to the final correct reasoning, thereby solving the problem of delayed rewards.

[0009] It is worth noting that the policy network can adopt a graph neural network architecture, which can directly process state inputs of graph structures, thereby more accurately evaluating the value of performing actions on the graph.

[0010] As a preference, the construction of the multi-aspect reasoning graph uses graph neural networks for representation learning, where the graph neural networks include graph convolutional networks or graph attention networks for learning node and edge embedding vectors to capture distributed representations of entities and relationships.

[0011] Preferably, the graph neural network adopts a multi-layer structure, with the bottom layer capturing local neighborhood information and the high layer integrating global graph structure information, using a graph attention network (GAT) that can assign different attention weights to different neighbor nodes, thereby highlighting more important connection relationships in a specific reasoning context; it is worth noting that the initial features of the nodes can be fused with the embeddings obtained from the language model via the language model, thereby enhancing semantic information.

[0012] As a preference, the path traversal in the reasoning operation includes multi-hop reasoning, which generates a reasoning chain by traversing multiple edges in the multi-aspect reasoning graph, where the reasoning chain consists of a series of connected nodes and edges, representing the logical derivation process from the user query to the response.

[0013] Preferably, the path traversal operation adopts a path search strategy based on random walk or reinforcement learning to efficiently find reasonable reasoning paths in a huge graph search space; to prevent the path from being infinitely extended, the system sets a maximum hop limit, and a total confidence score can be calculated for the generated reasoning chain, which integrates the relationship strength of each edge on the path, for subsequent reward calculation and response generation.

[0014] As a preference, the external knowledge base verification in the reward signal includes querying an external knowledge graph, where the external knowledge graph is a structured database used to verify the correctness of the reasoning result and adjust the reward value based on the verification result.

[0015] As a preference, the reinforcement learning agent is trained using a deep Q network, where the DQN includes a target network and an experience replay mechanism.

[0016] As preferred, the multi-aspect reasoning graph is dynamically updated, adjusting nodes and edges in real-time according to user interactions, wherein the updating operation includes adding new nodes, deleting redundant nodes, or modifying edge weights, for reflecting the latest context and reasoning requirements.

[0017] As preferred, the reasoning aspects further include an emotion aspect for processing emotion-related reasoning, wherein the emotion aspect subgraph includes emotion polarity nodes and emotion relationship edges, constructed based on an emotion dictionary or a deep learning model.

[0018] As preferred, the method is applied to a question-answering system or a dialogue system, wherein the question-answering system is used to answer complex multi-step queries, and the dialogue system is used for context maintenance and reasoning in multi-round dialogue.

[0019] As preferred, the user historical interaction records in the state space include encoded representations of past dialogues, which are sequentially encoded using a recurrent neural network or a Transformer model to capture long-term dependencies.

[0020] As preferred, the action space further includes an action of generating a natural language response, wherein the natural language response is generated using a sequence-to-sequence model or a pre-trained language model to convert the reasoning path into a natural language text output.

[0021] Advantages of the present application: 1. The present application overcomes the limitations of existing technologies that focus on a single reasoning dimension by constructing a reasoning graph that integrates multiple dimensions such as semantics, logic, and context, and using a reinforcement learning agent to operate these subgraphs collaboratively, enabling the model to conduct comprehensive and multi-angle analysis on complex queries, significantly improving the comprehensiveness of reasoning and the accuracy of comprehensive judgment, and thus providing more accurate and complete responses in question-answering or dialogue tasks.

[0022] 2. The present application makes the reasoning process tangible as a series of interpretable operations (such as node expansion, path traversal) on a graph structure, making the internal decision-making mechanism of the model transparent, allowing users and developers to clearly trace the complete reasoning chain from the question to the answer, significantly enhancing the explainability and credibility of the model, facilitating error diagnosis and system optimization.

[0023] 3. The present application uses a reinforcement learning framework, enabling the model to learn and dynamically optimize online based on composite reward signals (such as accuracy, relevance, user feedback) through continuous interaction with users or the environment, which not only reduces the dependence on large amounts of static labeled data, but also enables the system to evolve and adapt to new domains and user preferences, improving the robustness and practicality of the system.

[0024] 4.The application ensures that the reasoning process and its results are based on reliable facts by taking external knowledge base verification as a key component of reward calculation, avoids errors caused by models relying only on internal parameters, and greatly improves the authenticity and reliability of system output. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A workflow schematic of the application is shown. DETAILED DESCRIPTION

[0026] To make the purposes, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described in detail below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some but not all of the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the application.

[0027] The application provides an embodiment of a multi-aspect reasoning graph method of an NLP interaction model based on reinforcement learning: In this embodiment, the reasoning graph structure design is described: The multi-aspect reasoning graph of the application is a dynamic graph model with multi-level and multi-subgraph structures. The overall structure of the graph can be formally represented as G = {G_s, G_l, G_c, G_e}, where G_s represents a semantic subgraph, G_l represents a logical subgraph, G_c represents a context subgraph, and G_e represents an emotion subgraph.

[0028] Each subgraph itself is also a complete graph structure, including a node set V and an edge set E, the nodes represent entities, concepts or propositions, and the edges represent the relationships between the nodes. Different types of nodes and edges are represented by feature vectors, which will be continuously optimized in the training process.

[0029] The semantic subgraph G_s focuses on the meaning analysis of language itself, the nodes include entities, actions and attributes mentioned in the text, and the edges represent semantic relationships between entities, such as agent relationships, patient relationships, attribute relationships, etc. The core function of this subgraph is to accurately capture the semantic roles and meaning structures of sentences.

[0030] The logical subgraph G_l is responsible for representing and processing logical reasoning relationships, the nodes include fact propositions and logical operators, and the edges represent logical connection relationships, such as implication relationships, equivalence relationships, conjunction relationships, etc. This subgraph enables the system to perform logical operations such as deductive reasoning and inductive reasoning.

[0031] The context subgraph G_c records and maintains the contextual information of the dialogue or text, with nodes including entities and propositions mentioned in the historical dialogue, and edges representing temporal order, anaphora relations, and other contextual associations. This subgraph ensures that the system remains coherent and consistent in expanding interactions.

[0032] The emotion subgraph G_e processes information related to emotions and attitudes, with nodes including emotion subjects, emotion objects, and emotion polarities, and edges representing emotion expression relationships, emotion target relationships, and the like. This subgraph enables the system to understand and respond to the emotional state of the user.

[0033] In this embodiment, the initialization and updating mechanism of the graph are described: The initialization of the reasoning graph is based on the parsing results of the input text. The system uses a pre-trained language model combined with a dependency syntax analysis tool to extract entities and relationships from the text. To improve the accuracy of the initialization, the system also incorporates domain-specific dictionaries and rules to ensure that professional terms and domain concepts are correctly identified.

[0034] The updating of the graph is dynamic and mainly includes the following operations: Node expansion, when new entities or concepts appear in the input, the system adds new nodes in the corresponding subgraph, and the initial feature vector of the new node is obtained by encoding the node name or description text.

[0035] Edge update, including adding new edges, deleting existing edges, and adjusting edge weights. The adjustment of edge weights is based on the decision of the reinforcement learning agent and the subsequent reward signal. The edge weights of important relationships will be enhanced, while the edge weights of unimportant or incorrect relationships will be weakened.

[0036] Subgraph coordination, the nodes between different subgraphs are connected through cross-reference edges, forming a unified reasoning network. When reasoning in one subgraph, relevant information and conclusions can be propagated to other subgraphs through these cross-reference edges, realizing the coordination of multi-aspect reasoning.

[0037] In this embodiment, the state space design of the reinforcement learning agent is described: The state space S of the reinforcement learning agent is a high-dimensional continuous space, which includes the following components: The graph state representation s_g is a compressed representation of the current multi-aspect reasoning graph, obtained by graph pooling operation on the feature vectors of all nodes and edges. To preserve the structural information of the graph, the system adopts a hierarchical pooling strategy, first pooling each subgraph, and then integrating the pooling results of all subgraphs.

[0038] The user query representation s_q is a vector representation of the user's current input natural language query, using a pre-trained language model to obtain the semantic vector of the query sentence.

[0039] The historical interaction representation s_h is the encoding of the user's historical interaction with the system. The system uses an LSTM network to encode the past multi-turn dialog, capturing long-term dependencies.

[0040] The complete representation of the state vector is the concatenation of s = [s_g, s_q, s_h], ensuring that the agent can consider the current reasoning state, user's immediate needs, and interaction history context when making decisions.

[0041] In this embodiment, the action space design for the reinforcement learning agent is described: The action space of the agent contains a series of operations that can be performed on the reasoning graph, mainly including: The node expansion action a_expand adds a new node in the graph, and the parameters of this action include the type of node to be added, the content of the node, and the connection position.

[0042] The edge update action a_update adds, deletes, or modifies edges in the graph, and the parameters of this action include the edge type, weight adjustment amount, and involved node pair.

[0043] The path traversal action a_traverse performs multi-hop reasoning along a specific path in the graph, and the parameters of this action include the starting node, traversal direction, and maximum number of hops.

[0044] The subgraph selection action a_select activates or focuses on a specific subgraph, and the parameters of this action include the target subgraph identifier and focus intensity.

[0045] The response generation action a_generate generates a natural language response based on the current reasoning state, and this action triggers a conditional text generation model to convert the reasoning result into natural language.

[0046] The action space contains both discrete actions (such as selecting which type of operation to perform) and continuous actions (such as weight adjustment amount), so it is a hybrid action space. To handle this complexity, the system uses the Proximal Policy Optimization (PPO) algorithm, which can output the probability distribution of the hybrid action.

[0047] In this embodiment, the reward function design is described: The invention adopts a multi-dimensional composite reward function, which specifically includes: The reasoning accuracy reward r_accuracy measures the correctness of the system's reasoning result, and calculates the similarity score by comparing the reasoning chain generated by the system with the annotated standard answer. For scenarios lacking annotated data, multi-model consensus or expert evaluation can be used as an alternative solution.

[0048] The relevance reward r_relevance is used to evaluate the relevance of the system response to the user query. The cosine similarity is used to calculate the semantic similarity between the query vector and the response vector, and traditional indicators such as keyword overlap rate are combined.

[0049] The feedback reward r_feedback is based on the explicit or implicit feedback of the user. Explicit feedback includes user ratings, likes / dislikes, etc. Implicit feedback includes subsequent dialogue length, user problem solving rate, and other behavior data.

[0050] The knowledge verification reward r_verification is used to verify the factual correctness of the reasoning result by querying an external knowledge graph. Positive rewards are given to reasoning steps that can find support in the knowledge graph, and negative rewards are given to reasoning steps that contradict the facts in the knowledge graph.

[0051] The exploration reward r_exploration encourages the agent to try new reasoning paths and avoid getting stuck in local optima. This reward is positively correlated with the novelty of the action, and is achieved by calculating the difference between the current action and the historical action distribution.

[0052] The total reward is the weighted sum of the above rewards, R = w_a·r_accuracy + w_r·r_relevance +w_f·r_feedback + w_v·r_verification + w_e·r_exploration, where the weight parameters w_a, w_r, w_f, w_v and w_e can be adjusted according to the specific application scenario.

[0053] In this embodiment, the network structure and training algorithm are described: The agent's policy network uses the Actor-Critic architecture, including a feature extraction layer, an Actor network and a Critic network, The feature extraction layer uses a multi-layer perceptron to process the user query and historical interaction part of the state vector; uses a graph attention network to process the graph state part of the state vector, capturing important relationships between nodes in the graph. The Actor network is used to receive the output of the feature extraction layer, and outputs the probability distribution of discrete actions through softmax, and the mean and variance of continuous actions through Gaussian distribution parameterization. The Critic network shares the feature extraction layer with the Actor network, and outputs the estimated value of the state value function, which is used to calculate the advantage function.

[0054] The training process adopts a proximal policy optimization algorithm, and a generalized advantage estimation is used to calculate an advantage function, in order to improve the training efficiency, the system also includes storing the interaction experience (s, a, r, s') of the agent, breaking the time correlation between data by random sampling, providing a stable learning goal for the Critic network, periodically copying parameters from the main network, and starting training from a simple inference task, gradually increasing the complexity of the task, and accelerating the convergence process.

[0055] Please refer to Figure 1 The workflow of the application is described as follows: S1, extracting multiple entities and relationships from the input natural language text to construct an initial multi-aspect reasoning graph, wherein the multi-aspect reasoning graph includes multiple subgraphs, each subgraph corresponds to an aspect of reasoning, the aspect of reasoning includes semantic aspect, logical aspect and context aspect, wherein the semantic aspect represents the semantic role and meaning of the text, the logical aspect represents the logical relationship and reasoning rule, and the context aspect represents the dialogue history or text context information; S2, initializing a reinforcement learning agent, the state space of the reinforcement learning agent includes the current graph state of the multi-aspect reasoning graph, the vector representation of the user query and the user historical interaction record, the action space includes the reasoning operation performed on the multi-aspect reasoning graph, the reasoning operation includes node expansion, edge update, path traversal or subgraph selection, wherein the node expansion is used to add new entities or concepts, the edge update is used to modify the relationship between entities, the path traversal is used to generate a multi-hop reasoning chain, and the subgraph selection is used to activate the subgraph of a specific reasoning aspect; S3, interacting with the user through the reinforcement learning agent, receiving the user input natural language query, encoding the user query into a vector representation, and combining the current graph state in the state space to select actions using a policy network, and updating the multi-aspect reasoning graph or generating a reasoning path by executing the selected action; S4, based on the result after executing the action, calculating a reward signal, the reward signal integrates reasoning accuracy, response relevance, user feedback and external knowledge base verification, wherein the reasoning accuracy is calculated by comparing the similarity between the generated response and the standard answer, the response relevance is evaluated by cosine similarity, the user feedback is obtained by explicit scoring or implicit behavior data, and the external knowledge base verification is used to confirm the correctness of reasoning by querying the knowledge graph; S5, using the reward signal, updating the policy network and value network of the reinforcement learning agent through the reinforcement learning algorithm to optimize the reasoning strategy, wherein the reinforcement learning algorithm includes deep Q network, policy gradient method or actor-critic method; S6, repeating steps S3 to S5 to realize continuous learning and interaction optimization, thereby improving the performance of the natural language processing task.

[0056] The application provides example 1: The application is applied to the intelligent question and answer system in the medical and health field, provides disease related consultation and suggestion for users, uses medical knowledge graph (such as UMLS) as external knowledge source in the embodiment, increases medical professional subgraph in inference graph, and contains medical concepts such as symptom, disease, medicine and treatment scheme.

[0057] The user query faced by the system is "I have headache and fever recently, what is the possible reason, and how to relieve it?", it is a typical medical consultation problem containing symptom description and seeking etiological diagnosis and preliminary coping measure, and the complexity lies in that a plurality of symptoms need to be combined with a plurality of potential diseases for association reasoning, and a safe and effective relief scheme is further derived, and meanwhile, the system must consider medical accuracy and popularization in response, and avoid misunderstanding or risk caused by improper expression.

[0058] When receiving the user query, the system firstly starts a natural language processing procedure, utilizes integrated BERT pre-training model and medical field dictionary to deeply analyze the query text, accurately extracts key entities "headache" and "fever", and creates corresponding nodes for the two symptoms in the semantic subgraph, and analyzes the user intention of "possible reason" and "relief method"; then, the system activates the medical professional subgraph, links the "headache" and "fever" nodes with known medical concepts in the knowledge base by querying the built-in UMLS knowledge graph, and initially retrieves a common disease candidate set strongly related to the two symptoms, such as "cold", "flu" or "migraine", to prepare a structured information environment for subsequent deep reasoning.

[0059] After the initialization, the reinforcement learning agent starts to operate. Its state perception module first constructs the current state vector, which integrates the current graph state containing the "headache" and "fever" nodes, the encoded user query vector, and the empty historical interaction record. Based on this state, the agent's internal policy network starts to calculate and sequentially selects and executes a series of fine-grained reasoning actions: the first action is node expansion, adding disease nodes such as "cold", "flu", and "migraine" in the medical subgraph; then the edge update action is performed, establishing "possible indication" relationship edges between the "headache" and "fever" symptom nodes and the newly added disease nodes, with their initial weights strictly set according to the association strength in the UMLS knowledge base; then the agent selects the path traversal action, starting from the "headache" node and exploring one hop along the newly established "possible indication" edges to access nodes such as "cold" and "flu", simulating the preliminary diagnosis reasoning chain from symptoms to diseases; to further answer "relief methods", the agent performs the node expansion action again, adding treatment or relief measure nodes such as "rest", "hydration", and "fever-reducing medicine", and connecting these measure nodes to the corresponding disease nodes with "can be used to relieve" relationship edges through the edge update action. Finally, the agent triggers the response generation action, inputting the path and node information obtained by walking and reasoning on the graph into the fine-tuned pre-trained language model to generate the final natural language response.

[0060] The natural language response generated by the system is "Headache and fever may be symptoms of cold or flu. It is recommended to rest more and drink more water. If the body temperature exceeds 38.5°C, consider using fever-reducing medicine. If the symptoms persist or worsen, please seek medical attention promptly." This response clearly lists the possible causes of reasoning and gives prudent action recommendations. Then the reward calculation module is activated to evaluate the quality of this interaction. It first verifies the factual correctness of the reasoning steps "headache and fever indicate cold / flu" and "rest, hydration, and fever-reducing medicine relieve cold / flu" by querying the UMLS knowledge base, and gives a high reasoning accuracy reward after confirming the correctness. At the same time, the system calculates the semantic relevance reward between the generated response and the user query. In addition, the system monitors the user's subsequent behavior, such as whether the user expresses satisfaction, asks follow-up questions, or directly ends the conversation, and incorporates these implicit feedback into the user feedback reward. Finally, all dimensions of rewards are combined according to the pre-set weights to form a comprehensive reward signal, which is used to guide the subsequent policy update.

[0061] After completing this round of interaction and obtaining the comprehensive reward signal, the system enters the learning and optimization phase, which stores the complete sequence of this interaction, including the initial state, the series of actions performed, the rewards obtained, and the end state, as an experience sample into the experience replay buffer. During the training period, the system samples a batch of such experiences from the buffer and updates the parameters of the policy network and value network in the reinforcement learning agent using the proximal policy optimization algorithm. The core goal is to increase the probability of selecting actions that can lead to high rewards (e.g., accurately associating symptoms with diseases and proposing safe and effective relief solutions) in similar future states, while suppressing actions that may lead to low or negative rewards. Through continuous optimization, the system's decision-making ability in handling medical question and answer tasks will continuously improve.

[0062] After quantitative evaluation on a professionally constructed medical question and answer test set, the system of the embodiment achieved significant results, with a final diagnosis accuracy of 87.5%, which is a significant improvement over the baseline question and answer system using only BERT, which has an accuracy of 76.2%. This fully demonstrates the great advantage of the multi-aspect reasoning graph and reinforcement learning collaborative framework in handling medical problems that require complex symptom reasoning. In addition, since the system can provide clear and interpretable reasoning paths (such as "headache -> possible indication -> cold -> can be used to relieve -> rest"), rather than just giving a final answer, it greatly enhances the user's trust and acceptance of the system's response, and achieves higher satisfaction scores in user experience surveys.

[0063] The present application provides embodiment 2: The present application is applied to an intelligent customer service system in the field of e-commerce, which handles customer inquiries, complaints, and after-sales service tasks, etc. In this embodiment, an enterprise product knowledge base and a user database are integrated, and a user emotion subgraph and a transaction history subgraph are added to the reasoning graph.

[0064] The user query received by the system is "I bought a mobile phone screen last week, and the customer service attitude is very poor, I am very dissatisfied!", which is a typical composite customer complaint, containing both specific product quality problems (mobile phone screen failure) and service attitude complaints, as well as expressing strong negative emotions. The difficulty in handling it lies in the need to coordinate and handle both factual problems (screen problem diagnosis and solution) and emotional problems (apology, pacification, and compensation). Any lack or improper handling of either side may lead to further decline in customer satisfaction, and even trigger more serious escalation of complaints.

[0065] When receiving the user's emotional complaint, the system first starts deep semantic analysis, accurately extracts the key entities "mobile phone screen", "customer service" and emotional keywords "very bad", "very dissatisfied" using the domain-adapted BERT model, and creates corresponding entity nodes in the semantic subgraph. At the same time, the system specially strengthens the processing of the emotion analysis module, creates a high-intensity "angry" node in the emotion subgraph and associates it with the user entity, accurately quantifies the user's dissatisfaction, and queries the transaction database to establish specific record nodes of the user's purchase of mobile phones last week in the transaction history subgraph, and through cross-reference edges, it is associated with the current problem description, forming an initial state of comprehensive reasoning that integrates facts, emotions and historical context.

[0066] Based on the initialized multi-aspect reasoning graph, the reinforcement learning agent starts its decision cycle. The state vector constructed by the state perception module of the agent contains the product quality problem node, the high-intensity negative emotion node, the complete user purchase history, and the current complaint intent code. In the face of this complex state, the agent's strategy network calculates and selects and executes a series of carefully ordered reasoning actions: first, it starts the subgraph selection action, activates the emotion subgraph and focuses on the high-intensity "angry" node, ensures that the system first identifies and pays attention to the user's emotional state, then executes the node expansion action, adds "sincere apology", "problem solving solution" and "compensation measures" nodes in the dialogue strategy subgraph, then the agent executes the path traversal action, starting from the "mobile phone screen" node, connecting to the company's "warranty policy" node through the "belongs to warranty range" edge, confirming the policy basis for repair or replacement, and through the edge update action, enhancing the association strength between the "poor customer service attitude" node and the "compensation measures" node, establishing a logical basis for providing compensation, finally the agent triggers the response generation action, inputs all reasoning results including problem confirmation, apology expression, specific solution and compensation commitment into the emotion-sensitive text generation model as conditions, and forms the final response.

[0067] The final response generated by the system is "I'm very sorry for the bad experience you've had! For the phone screen issue, we can immediately start the warranty process for you and arrange a replacement. Regarding the customer service attitude issue, we have recorded it and will handle it seriously. As an apology, we have prepared a 50 yuan coupon for you, hoping to make up for it." This response fully covers the four key elements of problem confirmation, apology, specific solution, and compensation measures, and then the reward calculation module starts the evaluation process, which first monitors the user's immediate feedback, such as whether the user accepts the solution and whether the mood is eased, as the main source of user feedback rewards, while verifying the factual correctness of "the phone screen issue is within the scope of warranty" and "providing a coupon is a feasible compensation measure" by querying the knowledge base, giving a knowledge verification reward. The system also tracks the final resolution of this complaint (such as whether the replacement is completed and whether the customer withdraws the complaint) as a long-term reward signal. All these dimensions are combined into a comprehensive reward according to the pre-set weight, which is used to guide strategy optimization.

[0068] After completing this round of customer interaction and obtaining multi-dimensional reward signals, the system enters the learning phase, storing the complete interaction experience (from receiving complaints to generating responses and obtaining feedback) as a sequence sample into the experience replay buffer. During the training period, the system uses the Proximal Policy Optimization (PPO) algorithm to sample a batch of experience similar to customer complaint cases from the buffer, calculates the advantage function and updates the policy network parameters. Its essence is to strengthen action sequences that can effectively calm user emotions, solve actual problems, and prevent complaints from escalating (such as timely apologies and providing practical compensation solutions), while weakening action choices that may exacerbate conflicts or not fully resolve problems. Through this continuous online learning, the system's response strategy becomes more and more accurate and efficient when handling various customer complaints.

[0069] In the actual deployment of the system for three months, through statistical analysis of more than 100,000 customer interaction data, it has achieved significant business results, with the most critical user satisfaction index reaching 92.3%, an increase of 25.6 percentage points compared to the previous traditional customer service system based on rules. And because the system can accurately identify user emotions and take appropriate calming and compensation measures in a timely manner, the proportion of customers choosing to escalate complaints (such as complaining to regulatory agencies or requiring higher-level supervisors to intervene) after initial complaints has decreased by 18.7%, which directly reduces the enterprise's public relations risks and customer churn costs. At the same time, the handling solutions provided by the system also shorten the average problem resolution time by about 35%, improving the overall operational efficiency of the customer service team.

[0070] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in other forms. Any skilled person in the art can modify or change the disclosed technical content into equivalent embodiments with equivalent changes, and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solution content of the present application, still falls within the protection scope of the present application.

Claims

1. A multi-faceted inference graph method for NLP interaction models based on reinforcement learning, characterized in that, It includes the following steps: S1. Extract multiple entities and relations from the input natural language text to construct an initial multi-faceted reasoning graph. The multi-faceted reasoning graph includes multiple subgraphs, each of which corresponds to a reasoning aspect. The reasoning aspect includes a semantic aspect, a logical aspect, and a contextual aspect. The semantic aspect represents the semantic role and meaning of the text, the logical aspect represents logical relations and reasoning rules, and the contextual aspect represents the dialogue history or text context information. S2, initialize a reinforcement learning agent. The state space of the reinforcement learning agent includes the current graph state of the multi-aspect reasoning graph, the vector representation of the user query, and the user's historical interaction records. The action space includes reasoning operations performed on the multi-aspect reasoning graph. The reasoning operations include node expansion, edge update, path traversal, or subgraph selection. Node expansion is used to add new entities or concepts, edge update is used to modify the relationships between entities, path traversal is used to generate multi-hop reasoning chains, and subgraph selection is used to activate subgraphs for specific reasoning aspects. S3 interacts with the user through a reinforcement learning agent, receives natural language queries input by the user, encodes the user query into a vector representation, and combines it with the current graph state in the state space. It then uses a policy network to select an action and executes the selected action to update the multi-faceted inference graph or generate an inference path. S4. Based on the results after the action is performed, a reward signal is calculated. This reward signal integrates reasoning accuracy, response relevance, user feedback, and external knowledge base verification. Reasoning accuracy is calculated by comparing the similarity between the generated response and the standard answer. Response relevance is evaluated by cosine similarity. User feedback is obtained through explicit ratings or implicit behavioral data. External knowledge base verification is confirmed by querying a knowledge graph to confirm the correctness of the reasoning. S5 uses reward signals to optimize the inference policy by updating the policy network and value network of the reinforcement learning agent through reinforcement learning algorithms, including deep Q-networks, policy gradient methods, or actor-critic methods. S6, repeat steps S3 to S5.

2. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The construction of the multi-faceted reasoning graph uses graph neural networks for representation learning, including graph convolutional networks or graph attention networks, to learn the embedding vectors of nodes and edges to capture distributed representations of entities and relationships.

3. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The path traversal in the reasoning operation includes multi-hop reasoning, which generates a reasoning chain by traversing multiple edges in a multi-faceted reasoning graph. The reasoning chain consists of a series of connected nodes and edges, representing the logical deduction process from user query to response.

4. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The external knowledge base verification in the reward signal includes querying an external knowledge graph, which is a structured database used to verify the correctness of the reasoning results and adjust the reward value based on the verification results.

5. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning agent is trained using a deep Q-network, where the DQN includes a target network and an experience replay mechanism.

6. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The multi-faceted reasoning graph is dynamically updated, adjusting nodes and edges in real time based on user interaction. The update operations include adding new nodes, deleting redundant nodes, or modifying edge weights to reflect the latest context and reasoning requirements.

7. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The reasoning aspect also includes an emotion aspect for handling emotion-related reasoning, wherein the emotion aspect subgraph includes emotion polarity nodes and emotion relation edges, and is constructed based on an emotion dictionary or a deep learning model.

8. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The method is applied to question-answering systems or dialogue systems, where question-answering systems are used to answer complex multi-step queries, and dialogue systems are used for context maintenance and reasoning in multi-turn dialogues.

9. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The user history interaction records in the state space include encoded representations of past conversations, using recurrent neural networks or Transformer models for sequence encoding to capture long-term dependencies.

10. The multi-faceted inference graph method for an NLP interaction model based on reinforcement learning according to claim 1, characterized in that: The action space also includes actions for generating natural language responses, wherein generating natural language responses uses a sequence-to-sequence model or a pre-trained language model to convert inference paths into natural language text output.

Images