A method and device for intelligent interaction and decision-making based on a large language model
By combining a large language model and the DQN algorithm to create an intelligent interaction method, structured dialogue states are generated and the policy network is optimized. This solves the problems of insufficient state representation and low training efficiency in complex business scenarios of the DQN dialogue system, and enables the dialogue system to make efficient, coherent and intelligent decisions in industrial collaboration scenarios.
Patent Information
- Application Number
- CN202610777258.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-01
- Publication Date
- 2026-11-17
AI Technical Summary
Existing DQN dialogue systems suffer from insufficient state representation in complex business scenarios, inability to effectively handle unstructured user language input, low training efficiency, and unstable parameter updates, making them unsuitable for the actual needs of industry collaboration scenarios.
Natural language understanding is performed using a large language model to generate structured dialogue states. These states are then encoded using historical context, and the DQN algorithm is used to select the optimal action. Finally, the policy network parameters are optimized through an experience replay pool, achieving efficient and standardized transitions between dialogue states.
It enables real-time continuation of dialogue information and smooth progress of multi-round dialogues, ensuring intelligent adaptation of dialogue actions and effective utilization of user feedback, thereby improving dialogue efficiency and task completion rate, and adapting to the coherent needs of complex business scenarios.
Smart Images

Figure CN123115302A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal interaction and dynamic learning technology, specifically relating to an intelligent interaction and decision-making method and device based on a large language model. Background Technology
[0002] In complex business scenarios such as industrial collaboration, supply chain management, and enterprise services, users often need to complete information collection, solution comparison, compliance verification, and final decision-making through multiple rounds of interaction.
[0003] Reinforcement learning (RL) has been attempted to be used for dialogue policy optimization, among which DeepQ-Network (DQN) has attracted attention because it can handle high-dimensional state spaces and has an experience replay mechanism.
[0004] However, DQN dialogue systems based on related technologies generally suffer from the following technical problems: First, the state representation only includes the slot filling situation in the current round, ignoring the behavioral trends and intention evolution in multiple rounds of context.
[0005] Secondly, dialogue systems using the DQN algorithm cannot convert unstructured natural language input from users into standardized numerical data that the algorithm can process. This renders the DQN policy network ineffective and unable to achieve intelligent action selection. Furthermore, the system lacks the ability to systematically manage the entire dialogue process data, and the absence of historical experience data leads to unstable parameter updates, low training efficiency, and the inability to quickly apply trained parameters to real-world dialogues. This results in a disconnect between the algorithm and practical applications, hindering the transformation from experience to intelligent decision-making and making it difficult to adapt to the actual business needs of industry collaboration scenarios.
[0006] Third, the experience playback and training update process did not take into account the business semantic structure of the industry scenario, resulting in high-value samples being diluted by a large number of neutral interactions, and the model converging slowly or even getting stuck in local optima. Summary of the Invention
[0007] This invention provides an intelligent interaction and decision-making method based on a large language model. The method improves task completion rate and dialogue efficiency, optimizes strategy stability through experience playback and dual network architecture, and achieves a leap from rule-driven to intelligence-driven approaches. It is suitable for complex decision-making needs in industrial collaboration scenarios.
[0008] The methods include: S1: Receives the current dialogue information input by the user, performs semantic recognition on the current dialogue information through the natural language understanding component, and generates a structured dialogue state that represents the current dialogue process by combining the historical dialogue context. S2: Encode the structured dialogue state into a current state vector; S3: Input the current state vector into the policy network, and the policy network selects the current optimal action from the predefined set of dialogue actions based on the current state vector. The set of dialogue actions includes reply, query, or confirmation. S4: Execute the current optimal action to generate a system response and return it to the user; S5: Receive user feedback on system response and calculate instant reward value based on feedback and preset reward rules; S6: Based on user feedback and the current dialogue state, update the dialogue context and generate the next structured dialogue state based on subsequent user input; S7: Encode the next structured dialogue state into a next state vector; S8: Generate an experience sample containing the current state vector, the current best action, the immediate reward value, and the next state vector, and store the experience sample in the experience replay pool; S9: In training mode, historical experience samples are sampled from the experience replay pool, and the parameters of the policy network are iteratively updated using the historical experience samples. S10: Use the next state vector as the new current state vector, return to step S3, and continue processing subsequent dialogue interactions.
[0009] According to another embodiment of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the intelligent interaction and decision-making method based on a large language model.
[0010] As can be seen from the above technical solutions, the present invention has the following advantages: The intelligent interaction and decision-making method based on a large language model provided by this invention parses the user's current input through a natural language understanding component, generates a structured dialogue state by linking historical dialogue context, updates the context based on user feedback and the current state, and generates the next structured state by combining subsequent input, ensuring that dialogue information can continue in real time. The combination of these two approaches allows the system to grasp user intent and dialogue progress, enabling smooth multi-turn dialogues and meeting the coherent needs of complex business dialogues in industry collaboration scenarios.
[0011] This invention achieves efficient and standardized conversion of dialogue states into algorithm-processable data. The generated structured state is encoded into a current state vector, and dimensionality verification and standardization ensure that the vector format conforms to the input requirements of the policy network. The encoding module and rules are reused to encode the generated next structured state into a next state vector, guaranteeing complete consistency in encoding logic and dimensionality specifications between the two vectors.
[0012] This invention achieves intelligent adaptation and filtering of dialogue actions through a policy network action selection mechanism. The current state vector is input into the policy network, and the value of each action is evaluated using a dual-network architecture. The optimal action is selected from a predefined action set, and the action verification unit verifies the action's suitability for the current dialogue process. This ensures that action selection aligns with the current dialogue state and business requirements, efficiently driving the dialogue towards completing business tasks.
[0013] This invention provides an action execution and response generation logic that achieves a natural transformation from optimal actions to user-perceptible responses. It receives the selected optimal action, retrieves a script template tailored to the industry collaboration scenario, fills in relevant parameters to generate a personalized system response, and returns it to the user via an interface after verification. This transforms abstract algorithmic actions into fluent and easily understood natural language responses, allowing users to clearly understand the system's intent and ensuring smooth and accurate dialogue interaction.
[0014] The feedback processing and reward calculation logic of this invention enables the effective utilization and quantification of user feedback. It receives user feedback on system responses, categorizes and processes it, and then, in conjunction with preset business scenario reward rules, converts qualitative feedback into quantitative, real-time reward values. This allows the system to perceive the effectiveness of its actions, transforming user feedback into data usable for subsequent optimization, supporting the generation of experience samples and the updating of strategy network parameters, and enabling the system to recognize its own shortcomings through user feedback.
[0015] This invention's experience sample generation and storage enables the effective accumulation and orderly management of dialogue interaction experience. It collects the current state vector, optimal action, immediate reward value, and next state vector, encapsulates them into standardized experience samples, and stores them in an orderly manner according to the dialogue scenario in an experience replay pool using a bucketed storage mechanism, establishing an associated index with the vector database. This allows experience samples to be efficiently retrieved and reused, providing high-quality historical experience support for policy network training.
[0016] This invention employs a training optimization method to achieve continuous evolution and performance improvement of the policy network. In training mode, samples are selectively sampled from the experience replay pool based on training priorities. After preprocessing, the policy network parameters are iteratively updated in batches. Through dual-network synchronization, training verification, and parameter rollback mechanisms, the stability and effectiveness of parameter updates are ensured, and the updated parameters are synchronized to the Rasa decision engine. This allows the policy network to continuously optimize its parameters based on historical dialogue experience, gradually improving the accuracy and adaptability of action selection, enabling the system to continuously improve its intelligence level with the accumulation of dialogue experience. Based on cyclic triggering, continuous advancement of multi-turn dialogue and stable system operation are achieved. The output next state vector is updated to the new current state vector. After verification, re-execution is triggered, initiating a new round of dialogue interaction. By monitoring the cyclic process in real time, anomalies are promptly identified and addressed, ensuring the system can continuously adapt to the dialogue needs of industry collaboration scenarios. Attached Figure Description
[0017] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 The system execution flowchart for realizing intelligent interaction and decision-making methods; Figure 2 A flowchart for intelligent interaction and decision-making methods based on large language models; Figure 3 A flowchart illustrating an implementation of an intelligent interaction and decision-making method based on a large language model; Figure 4 This is a flowchart illustrating another embodiment of an intelligent interaction and decision-making method based on a large language model. Figure 5 Flowchart of another embodiment of intelligent interaction and decision-making method based on large language model; Figure 6 This is a schematic diagram of an electronic device. Detailed Implementation
[0019] like Figure 1 As shown, the system implementing the intelligent interaction and decision-making method of this invention mainly provides services externally through an HTTP protocol interface. Users initiate a conversation and input text through a computer page or mobile app. The front end then transmits user information and conversation content to the system through the interface. After analysis, the system returns the results. For the user, it appears as if they are having a conversation with a person. All the content proposed in this invention is implemented in the backend system.
[0020] The entire system of this invention is divided into a model layer, a database layer, and an application framework layer. The model layer is built using open-source large models, and the data layer uses the ChromaDB vector database as the system's knowledge base. The application architecture layer mainly uses the Rasa framework to handle complex dialogue scenarios.
[0021] The system is mainly divided into three components: NLU stands for Natural Language Processing: converting raw user input into structured information; Dialogue management: storing and restoring dialogue context; Decision Engine: The core decision-making layer, managing the state of the conversation and deciding on the next action.
[0022] This invention introduces the DQN algorithm, which uses a neural network to learn a Q-function that evaluates the long-term value of each action in a given state. Using the DQN algorithm in Rasa's decision engine enhances the system's intelligence.
[0023] Rasa, the subject of this invention, is an open-source chatbot framework primarily responsible for handling dialogue, understanding user intent, and managing multi-turn conversations. For example: User input: Book a flight; System reply: What is your destination? User input: Shanghai; System reply: What time is it? This paper introduces the DeepQ-Network (DQN) algorithm, a classic algorithm in deep reinforcement learning, which can self-evolve through experience. It achieves continuous interaction in complex scenarios through a domain-adaptive dialogue strategy. The DQN algorithm is integrated to construct a decision tree reinforcement learning model. Experience replay mechanisms and a dual-network architecture are used to optimize policy selection, and the Q-value function is adjusted based on real-time environmental feedback, enabling decision-making from demand perception, policy deduction, and execution verification.
[0024] The intelligent interaction and decision-making method based on a large language model involved in this application will be described in detail below. Specific details such as particular system structures and technologies are proposed for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.
[0025] It should be understood that, when used in this specification, terms include indicating the presence of a described feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms include, encompass, have, and variations thereof mean including but not limited to, unless otherwise specifically emphasized.
[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] Please see Figure 2 and Figure 3 The diagram shows a flowchart of an intelligent interaction and decision-making method based on a large language model in a specific embodiment. The method includes: S1: Receives the current dialogue information input by the user, performs semantic recognition on the current dialogue information through the natural language understanding component, and generates a structured dialogue state that represents the current dialogue process by combining the historical dialogue context.
[0028] In some embodiments, the system may be configured with a dialogue information receiving module, a natural language understanding (NLU) component, a history context invocation module, and a structured state generation module, which may be deployed at the system application framework layer and establish data connections with the front-end interaction module and the dialogue caching module.
[0029] Furthermore, the current dialogue information is sent through the front-end computer page or mobile APP and transmitted to the receiving module via the HTTP protocol interface. The receiving module performs format validation on the input information, and after successful validation, it is transmitted to the NLU component.
[0030] Furthermore, the NLU component performs word segmentation on the current dialogue information to extract business-related entities. Then, it combines this with the historical dialogue context retrieved from the dialogue cache module, matching it against a pre-defined intent library for industry collaboration scenarios, including intents such as data acquisition, decision recommendation, and question consultation, ultimately generating a structured dialogue state.
[0031] Optionally, the structured state includes: the user's current intent, slot name, slot value, list of filled slots with filling status, list of necessary slots that are not filled, current dialogue stage, key nodes of historical dialogue, and user identity identifier.
[0032] This step primarily addresses the technical issue of ambiguous semantics and lack of contextual association in user natural language input, which prevents the system from capturing dialogue requirements. It avoids the limitation of the Rasa framework's NLU component, which can only recognize single-turn inputs and cannot link to historical context. By associating with historical dialogue context, it ensures the coherence and accuracy of semantic recognition, allowing the system to grasp the current dialogue progress and meet the needs of multi-turn, complex business dialogues in industry collaboration scenarios.
[0033] S2: Encode the structured dialogue state into a current state vector.
[0034] In some embodiments, the structured dialogue state data output is received, the feature information of each field is extracted, and the data is converted into numerical features according to preset association rules.
[0035] Optionally, the user's current intent corresponds to an integer code of 1-10, the list of filled slots is converted to a value of 0-8 according to the number of slots filled and the importance of the slots, the list of unfilled necessary slots is converted to a value of 0-4 according to the urgency, and the current dialogue stage corresponds to an integer code of 1-4. Here, you can set the initial greeting 1, information gathering 2, solution recommendation 3, and end confirmation 4.
[0036] Key nodes in historical dialogues are converted into values from 0 to 6 according to their impact level, and user identity identifiers correspond to values from 0 to 2. Optionally, ordinary users are represented by 0, administrators by 1, and visitors by 2. After integrating all numerical features, an initial vector with fixed dimensions is generated.
[0037] Alternatively, a min-max normalization method can be used to correlate all values of the initial vector to the [0,1] interval, eliminating encoding bias caused by differences in the scale of different feature values. The standardized vector is then the current state vector. The current state vector can also be synchronously transmitted to the execution mode of subsequent steps, while the encoding correlation rules and normalization parameters are recorded in the encoding log for reference in subsequent troubleshooting and encoding rule optimization.
[0038] This step primarily addresses the technical challenge of structured dialogue states being in non-numerical format and thus unable to be directly input into the DQN policy network for computation. This embodiment, considering the business characteristics of industry collaboration scenarios, designs fixed-dimensional encoding association rules to convert various text and list features of the structured state into numerical vectors of a uniform scale. Through dimensionality verification and standardization, it ensures that the vector format and scale meet the input requirements of the policy network, enabling the policy network to capture key information of the current dialogue process through numerical vectors, thus achieving the conversion of structured information into algorithm-processable data.
[0039] S3: Input the current state vector into the policy network, and the policy network selects the current optimal action from a predefined set of dialogue actions based on the current state vector. The set of dialogue actions includes reply, inquiry, or confirmation.
[0040] In some embodiments, this step addresses the technical problem that the Rasa framework requires preset rules to select actions and cannot dynamically select the optimal action based on the real-time dialogue state. This embodiment utilizes a DQN dual-network policy network to evaluate the value of each action through numerical calculation. Combined with a dynamic exploration-exploitation strategy, it filters out the optimal action that fits the current dialogue state. Action verification ensures that the action is adapted to the dialogue process, achieving intelligent action selection. The dual-network architecture of the policy network ensures the stability of action value evaluation, allowing the selected optimal action to adapt to the current dialogue requirements.
[0041] S4: Execute the current optimal action to generate a system response and return it to the user.
[0042] In some embodiments, a predefined script template library for industry collaboration scenarios can be provided. Combining the current optimal action code and execution parameters output by S3, the execution logic and script template corresponding to the action are retrieved from the predefined action library. Optionally, this can be done by querying the template corresponding to a key slot action, such as "Please provide {slot name}" to provide relevant services.
[0043] Extract the slot name, filled slot value, business data, etc. required for action execution from the current dialogue context, fill the parameters into the corresponding placeholders in the dialogue template, and generate personalized system response text. The response text must conform to the natural language expression habits.
[0044] Furthermore, after the response is generated, the response verification unit verifies the completeness and accuracy of the response text to ensure that there are no missing parameters or semantic errors. After the verification is passed, the response return unit synchronously returns the system response to the front-end page or mobile APP through the HTTP protocol interface.
[0045] S5: Receive user feedback on system responses and calculate instant reward values based on the feedback and preset reward rules.
[0046] In some embodiments, this step addresses the technical problem that the system cannot obtain user evaluations of responses and cannot optimize subsequent strategies based on user feedback. By obtaining user feedback through front-end interaction, classifying the feedback information, and designing targeted reward rules in conjunction with the business needs of industry collaboration scenarios, qualitative user feedback is converted into quantitative, real-time reward values that reflect the user's evaluation of the current action's performance. This allows the system to perceive the performance of actions through user feedback, overcoming Rasa's limitation of not being able to continuously optimize using user feedback.
[0047] S6: Based on user feedback and the current dialogue state, update the dialogue context and generate the next structured dialogue state based on subsequent user input.
[0048] In some embodiments, the system receives the output user feedback information and the current dialogue state, and then updates the core fields of the dialogue context.
[0049] Specifically, it involves supplementing user feedback identifiers and feedback content, updating the current dialogue stage, updating the slot status after obtaining a valid slot value for an action such as asking a question, marking the slot as filled, and recording the current action execution effect and reward value.
[0050] The system receives subsequent input from the user in response to the system's response via a front-end HTTP interface, verifies the validity of the input, and reuses it only after successful verification.
[0051] Furthermore, reuse involves semantic recognition of subsequent input information, combining it with the updated dialogue context to extract new intentions, slot values, and other features. Here, the structured state generation rules of S1 can be combined to generate a structured dialogue state that represents the next dialogue process. The structured dialogue state includes updated context information, core features of subsequent user input, new dialogue stages and slot states. After generation, it is synchronously transmitted to the execution process of S7 to overwrite the old dialogue context and ensure that subsequent dialogues can be based on the latest context.
[0052] This can solve the technical problem that the dialogue context cannot be updated in real time, resulting in the inability to connect subsequent dialogues and insufficient context awareness.
[0053] Step S6 combines user feedback with the current dialogue state to update various information in the dialogue context in real time, retains the feedback, slots, and stages of the dialogue process, and receives subsequent user input. It reuses the NLU component for semantic recognition to generate the next structured dialogue state, achieving seamless connection of the dialogue process and ensuring that the system can remember the latest dialogue information.
[0054] S7: Encode the next structured dialogue state as a next state vector.
[0055] In some embodiments, S7 and S2 encode only the next structured dialogue state, receive the next structured dialogue state output by S6, and reuse the encoding association rules and standardized parameters of S2.
[0056] Furthermore, the user's subsequent intent, updated slot filling status, unfilled slots, new dialogue stages, latest historical nodes, and user identity identifiers are extracted for the corresponding state, converted into numerical features according to preset association rules, and an initial vector is generated.
[0057] Furthermore, the dimension and completeness of the initial vector are verified. Missing dimensions are supplemented with the default value 0, and redundant features are pruned for dimensions that exceed the limit. The standardization processing unit adopts the same min-max normalization method as S2 to associate the initial vector with the [0,1] interval, eliminate numerical scale differences, and generate the next state vector.
[0058] Furthermore, the next state vector is synchronously transmitted to process S8 for generating experience samples and to process S10 for subsequent dialogue loops. The encoding process and vector data are also recorded in the encoding log and merged with the encoding log of S2 for storage, ensuring the consistency and traceability of the encoding process.
[0059] S8: Generate an experience sample containing the current state vector, the current best action, the immediate reward value, and the next state vector, and store the experience sample in the experience replay pool.
[0060] In some embodiments, this step addresses the technical problem that the system cannot record dialogue interaction experience and the policy network cannot continuously optimize through historical experience. This step captures the complete process of dialogue interaction, generates standardized experience samples, and stores the samples in an orderly manner in the experience replay pool. This preserves dialogue experience and facilitates on-demand sampling during training, thus solving the limitations of Rasa in accumulating experience and continuously optimizing.
[0061] S9: In training mode, historical experience samples are sampled from the experience replay pool, and the parameters of the policy network are iteratively updated using the historical experience samples.
[0062] As one implementation of the present invention, step S9 specifically includes the following: S91: Perform hierarchical conditional sampling based on business context metadata and priority weights in the experience samples.
[0063] In some embodiments, when sampling from the experience replay pool, the business subtask type of the current training phase can be read by the global training controller. For example, the business subtask type could be qualification review or order confirmation.
[0064] Within a subset of the experience replay pool, a weighted random selection is performed based on priority: high-priority samples are given a 3x sampling probability, medium-priority samples are given a 1x probability, and low-priority samples do not participate in this sampling.
[0065] Furthermore, each sampling generates an empirical sample set with a batch size of 32, ensuring that the batch contains at least error-correcting or related samples; if this is not met, resampling is performed until the condition is met.
[0066] As can be seen, this embodiment transforms the random sampling of general reinforcement learning into instructional sampling oriented towards business objectives. In industry collaboration scenarios, the dialogue logic of different sub-tasks, such as compliance-focused review and comparison-focused recommendation, differs greatly, and mixed sampling can lead to policy confusion. Mandatory inclusion of failure cases ensures that the model continuously learns and corrects errors, rather than merely imitating successful paths.
[0067] S92: Input the sampled empirical batches into the evaluation network and the target network to analyze the TD error with target correction.
[0068] In some embodiments, the system splits the sampled batch data into four groups of tensors: state sbatch, action batch, reward batch, and next state snextbatch.
[0069] Input sbatch into the evaluation network Qeval to obtain the current Q-value matrix.
[0070] Input snextbatch into the target network Qtarget to obtain the Q-value matrix of the next state.
[0071] For each sample, the system checks its `slotchangedflag` field: if true, meaning a new slot was filled in this round, an additional fixed gain constant is added when calculating the target Q value to enhance the value of acquiring effective information. The final TD error is calculated as the mean squared error between the predicted Q value and the corrected target Q value, and the TD error serves as the loss signal for backpropagation. It can be seen that step S92 enhances the policy network's sensitivity to structural progress in the task without modifying the reward function.
[0072] S93: Lock the output layer, fine-tune the hidden layers, jointly fine-tune all parameters, and update the target network.
[0073] In some embodiments, the parameter update process is divided into two consecutive stages. The first stage involves locking the output layer of the evaluation network, i.e., the last fully connected layer, and updating only the hidden layer parameters using gradient descent with a learning rate of η1 = 0.0005, iterating once. This stage aims to adjust the feature extraction capability without altering the action preference distribution.
[0074] In the next stage, all layers are unfrozen, and a complete backpropagation is performed using a small learning rate η2=0.0001 to fine-tune the volumetric correlation.
[0075] Furthermore, after each two-phase update, the system performs a soft update of the target network: θ'←τ·θ+(1 τ)·θ', where τ=0.005, θ is the evaluation network parameter, and θ' is the target network parameter. This approach borrows the hierarchical fine-tuning concept from transfer learning, where the hidden layer is responsible for state representation and the output layer is responsible for action relationships. Stable representations are then used to adjust decisions, aligning with cognitive learning principles. A small learning rate and soft updates further suppress parameter mutations, ensuring training stability.
[0076] S10: Use the next state vector as the new current state vector, return to step S3, and continue processing subsequent dialogue interactions.
[0077] In some embodiments, the next state vector output in step S7 is received, and the format, scale, and integrity of the vector are verified to ensure that the vector meets the input requirements of the policy network in step S3. After the verification is passed, the next state vector is marked as the new current state vector, overwriting the original current state vector.
[0078] Furthermore, the dialogue round identifier can be incremented by 1 from the original round number, synchronously updating the current dialogue round number in the system's global variables, which will be used to adjust the exploration probability when selecting subsequent actions.
[0079] Furthermore, a loop trigger signal is generated and sent to the S3 process, triggering the S3 process to re-execute, inputting the new current state vector into the policy network, and starting a new round of action selection, execution, and feedback processes. The system monitors the loop process in real time. If a state vector update failure or loop triggering anomaly occurs, the loop is paused, an alarm is triggered, the user is notified to investigate, and the loop update process is recorded in the loop log.
[0080] As can be seen, this step solves the technical problem that multi-turn dialogue cannot be continuously looped and the dialogue process cannot be advanced to complete the business task. S10 updates the next state vector to the new current state vector. Through the loop triggering method, step S3 and subsequent steps are restarted to realize the continuous loop of dialogue interaction. This ensures that the system can continuously interact with the user in multiple rounds based on the latest dialogue state, and gradually complete tasks such as gradually acquiring all key slots, recommending decision solutions, confirming user needs, and updating the dialogue round identifier. This forms a state update, loop triggering, and new round of interaction method, realizing multi-turn and coherent intelligent dialogue.
[0081] In one embodiment of the present invention, based on step S5, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. For example... Figure 4 As shown, S5 specifically includes the following steps: S51: Captures user interaction events, text content, and response times in the current round. Outputs include the probability distribution of confirmation and follow-up intents, and a sentiment polarity scalar. All signals are encapsulated in real-time into feedback data units {event code, intent vector, sentiment value, response interval}.
[0082] In some embodiments, the behavior acquisition thread executed in step S51 is registered on the user interface framework in event listener mode to capture standardized event codes triggered by user operations. The semantic analysis thread executed in step S51, upon receiving user text, employs a Transformer encoder architecture, with the output layer connected to two parallel fully connected layers.
[0083] Specifically, we can use one for intent classification, outputting a probability distribution vector covering multiple categories such as confirmation, denial, follow-up questioning, correction, silence, and termination. We can also use one for sentiment regression, outputting a sentiment score between -1 and 1.
[0084] Furthermore, the timer thread starts when the system message is sent and stops when the user begins to input, recording the precise time difference.
[0085] Furthermore, the outputs of these three threads are synchronized within a preset time window, configured with data content having fixed fields, and transmitted to subsequent execution processes.
[0086] S52: Input the generated feedback data units into the relationship processing module. The relationship processing module configures the feedback indicator correlation matrix, where each rule defines the correlation between a specific signal combination and the basic reward indicator.
[0087] For example, an intent that includes confirmation and an affinity score >0.5 is considered an explicit affirmative metric, with a base value +3. The key enhancement is configuring a decay policy function for each metric. The decay policy function takes the current conversation turn and the current conversation stage as input.
[0088] For example, for information acquisition metrics, the final value is calculated as: Value = Base * φ(T, Stage), where φ is a decay function that decays slowly in the later stages of the information gathering phase to encourage the collection of key information, and decays rapidly in the final confirmation phase to prevent redundant inquiries.
[0089] In some embodiments, the relationship processing module is configured with a feedback indicator association matrix, in which each record contains a matching condition, a target indicator, a base value calculation formula, and a decay function identifier.
[0090] The matching condition is a logical expression for the S51 output structure. When the condition is met, the base value of the metric is derived from a simple formula that may be a constant or include variables such as the response interval.
[0091] Furthermore, each indicator is associated with a decay function identifier, pointing to a predefined function library. For example, the information acquisition success indicator is associated with the function φinfo(T,Stage)=exp(-λ1*T)*(1+β*I(Stage='Information Acquisition')), where I is the indicator function. exp(-λ1*T) simulates global time-dependent decay, while (1+β*I(...)) provides gain at specific stages. Another indicator, user-expressed confusion, is associated with the function φconfuse(T,Stage)=exp(-λ2*T), where λ2>λ1, meaning the negative impact of the confusion signal decays faster.
[0092] It can be seen that by slowing down the decay of information-gathering rewards in the later stages of the information gathering phase, the learning focuses on the problem in the middle of the dialogue. By allowing negative feedback to decay rapidly, the agent avoids becoming overly conservative due to individual mistakes in the early stages.
[0093] S53: Input the indicators output from S52, including their decayed values, into the reward component generator. The generator contains four independent calculation channels, each corresponding to one of the four components: Task completion component: determined by the task termination indicator.
[0094] User satisfaction component: calculated by weighting and summing sentiment and positive indicators.
[0095] Information gain component: An information entropy variation model is adopted. The dialogue state S is defined as the probability distribution of all slots to be filled, and its entropy is H(S).
[0096] Furthermore, when new information, namely the slot value, is determined, the distribution changes, and the entropy decreases by ΔH.
[0097] The value of this component is calculated as Rinfo=η*ΔH, where η is a coefficient.
[0098] Furthermore, the key enhancement points define a baseline calibration based on historical experience. The system uses a buffer of historical ΔH values indexed by dialogue state similarity.
[0099] Calculate the offset of the current ΔH relative to the historical mean μh. If the offset exceeds the threshold, dynamically fine-tune η. The fine-tuning method is ηadj=η*(1+α*(μh-ΔH) / σh)) to keep the scale of Rinfo within an empirically reasonable range.
[0100] The dialogue efficiency component is calculated as Reff = -γ*(T+ξ*C), where T is the total number of rounds, C is the number of confirmation requests, and γ and ξ are coefficients.
[0101] S54: The four component values [Rcomplete, Rsatisfy, Rinfo, Reff] output by S53 are used to form a vector, which is then concatenated with the current dialogue stage encoding, the global progress P, and the short-term trends of each component. This vector is then fed into a weighted synthesis network as input.
[0102] The weight synthesis network is a small feedforward neural network that outputs a four-dimensional dynamic weight vector [ωc,ωs,ωi,ωe]. The final uncorrected reward Rraw = Σ(ωi*Ri).
[0103] S55: Perform range pruning, first-order exponential smoothing, and online normalization on Rraw, and output the final instantaneous reward value Rfinal used for policy network updates.
[0104] In some embodiments, range clipping can set a reasonable reward value range [Rlow, Rhigh]. Any input value Rraw that exceeds this range is set to the nearest boundary value Rclip = min(max(Rraw, Rlow), Rhigh). This is to prevent extreme values from disrupting the gradient calculation of the neural network.
[0105] First-order exponential smoothing is a time-series filter that controls the strength of historical memory. It is based on the assumption that the expected value of the reward signal changes gradually over a short period, filtering out high-frequency noise to obtain a smoother signal, Rsmooth.
[0106] Online normalization maintains a fixed-length FIFO queue of length L, storing the L most recent Rsmooth values. At each step, the mean μL and standard deviation σL of the data within this window are calculated. The final output reward is Rfinal = (Rsmooth - μL) / (σL + ε), where ε is a very small constant to prevent division by zero. This makes the distribution of Rfinal approximately zero-mean and unit-variance within the recent window.
[0107] In this embodiment, user feedback is full of noise and uncertainty in a real online interactive environment. This embodiment reduces the sensitivity to hyperparameter tuning and improves the reproducibility and robustness of the system.
[0108] As one implementation of the present invention, the reward component generation step S53 based on the multi-theoretical model specifically includes the following: S531: Based on the value space of all information slots to be filled in the current dialogue, construct an updated joint probability distribution model.
[0109] Furthermore, for each slot, the prior probability of its possible values is initialized. Based on the valid information submission category index with decay value output by S52 and the corresponding specific slot information, the probability distribution of the slot is updated, transforming the information provided by the user into an increase in the probability of a specific value.
[0110] Furthermore, by utilizing the updated edge distribution of each slot, the joint distribution entropy H(Scurrent) of the entire dialogue state is approximately calculated or estimated by sampling.
[0111] Furthermore, by comparing the state entropy H(Sprevious) of the previous round, the amount of state entropy reduction caused by this interaction is calculated, which is the original measure of information gain ΔHraw=H(Sprevious)-H(Scurrent).
[0112] In some embodiments, the flight booking task includes slots such as {destination, departure time, passenger name}. Each slot is associated with a list of possible values and their initial probabilities (which can be uniformly distributed or a prior distribution based on historical data). The system models the entire dialogue state S as a joint random variable of these slot values. When the user provides information, such as stating the city name, the probability of the city name value in the corresponding destination slot is significantly increased, while the probability of other city values is decreased.
[0113] S532: An experience buffer with the hash value of the determined slot as the key and historical ΔH observations as the queue. Calculate the current state feature fingerprint and retrieve the set of historical ΔH values for all similar states within the buffer.
[0114] Furthermore, statistics for the historical set are calculated, including the mean μh and standard deviation σh. Based on the relative deviation of the current ΔHraw from the historical mean, a smooth scaling function is applied to dynamically adjust the base coefficient η of the information gain.
[0115] Furthermore, the formula is adjusted to: ηadj=η*[1+δ*tanh((ΔHraw-μh) / (σh+ε))], where δ is the calibration intensity coefficient, ε is a small constant to prevent division by zero, and tanh is the hyperbolic tangent function. If historical samples are insufficient, then ηadj=η. Finally, the information gain component value is calculated as Rinfo=ηadj*ΔHraw.
[0116] In some embodiments, the calibrator includes an index structure and a calculation unit.
[0117] The index structure is a key-value store, where the key is the feature fingerprint of the dialogue state and the value is a fixed-capacity circular queue used to store historical ΔH values observed near that feature state.
[0118] The computation unit is triggered whenever Rinfo needs to be calculated. It generates a feature fingerprint based on the current state, searches the index for records where the Hamming distance of all keys is less than a preset threshold, aggregates all ΔH values in the queues corresponding to these records, and calculates their mean μh and standard deviation σh.
[0119] Furthermore, if a sufficient number of historical records are found, the calibration logic is initiated.
[0120] Furthermore, the design intent of the calibration formula ηadj=η*[1+δ*tanh((ΔHraw-μh) / (σh+ε))] is: when ΔHraw is not much different from the historical empirical mean μh, the output of the tanh function is close to 0, ηadj≈η, and the original coefficient is maintained.
[0121] Furthermore, when ΔHraw is significantly higher than the historical average, the tanh output approaches 1, ηadj=η*(1+δ), and the reward is appropriately amplified to encourage this information acquisition.
[0122] Furthermore, when ΔHraw is significantly lower than the historical average, the tanh output approaches -1, and ηadj=η*(1-δ), which appropriately suppresses rewards and avoids overestimating information of extremely low value.
[0123] δ controls the maximum amplitude of amplification or suppression, and can be set to 0.2-0.3 to ensure that the adjustment is gentle.
[0124] S533: When calculating the information gain component, the following three components are calculated in parallel: Task completion component calculation process: Monitor the task termination signal indicator from S52. If the task termination signal indicator is activated and has a negative value, then Rcomplete takes a large negative constant.
[0125] If the task termination signal indicator is not activated, but the system's internal state machine detects that all required slots have been effectively filled and the user has no intention to ask follow-up questions in this round, then Rcomplete takes a large positive number. Otherwise, Rcomplete = 0.
[0126] The user satisfaction component calculation process receives a series of attenuated emotion and interaction index values from the S52. The process is configured with a weight vector Ws and performs a weighted summation of the index values, i.e., Rsatisfy = Σ(Wsi * Valuei). The weights Ws can be preset based on domain knowledge or set as learnable parameters.
[0127] The dialogue efficiency component calculation process involves obtaining the current total number of dialogue rounds T from the system's global variables and obtaining the count C of confirmation request events from the S52 metrics. The component is calculated as a penalty term: Refficiency = -[γ1*T + γ2*C]. Here, γ1 is the round penalty coefficient, and γ2 is the confirmation penalty coefficient. This ensures that each round of dialogue or each additional confirmation request accumulates a negative reward, incentivizing efficiency.
[0128] In some embodiments, the task completion component employs a sparse, high-amplitude pulsed reward / penalty model, directly corresponding to the termination state and round reward in reinforcement learning, defining the final success and failure criteria for the entire dialogue task. The user satisfaction component uses a continuous, multi-feature linear model, synthesizing an overall satisfaction score from multiple aspects of user feedback, focusing on the quality of the interaction process. If it is found that the learned strategy can complete the task but the user score is not high, the weight vector or input index of the user satisfaction component can be adjusted; if the dialogue is found to be too lengthy, the efficiency penalty coefficient γ1 can be increased.
[0129] S534: Input the component values Rcomplete, Rsatisfy, Rinfo, and Refficiency generated by the above four parallel computing channels into their respective processors.
[0130] For each processor, online mean and standard deviation statistics are performed on the historical output values of the corresponding component. The original component values are normalized to zero mean and unit variance using the formula Rinormalized=(Ri-μi) / σi to obtain the standardized component values Rc', Rs', Ri', Re'.
[0131] Furthermore, the aim is to eliminate the influence of differences in dimensions and numerical ranges among different components, ensuring that the component values are at similar scales, which facilitates balancing and comparison in the weight synthesis network of step S54. The normalized four-dimensional component vector [Rc', Rs', Ri', Re'] is output to step S54.
[0132] In some embodiments, the aim is to eliminate the inherent magnitude and scale differences between different reward components due to their different intrinsic properties. Here, the task completion component Rcomplete may only vary between ±15, while the user satisfaction component Rsatisfy may vary continuously between -2 and +2, and the information gain component Rinfo may be a decimal less than 1.
[0133] If these values at different scales are directly input into the subsequent weight synthesis network, the network focuses on components with large absolute values while ignoring components with small values but potentially significant changes. Online normalization, by independently normalizing each component to a similar scale, ensures that the weight synthesis network can perform normalization on each component, rather than being dominated by its original numerical range.
[0134] As one implementation of the present invention, the dynamic weight synthesis step based on the meta-learning network in step S54 specifically includes the following: S541: Receive the normalized four-dimensional reward component vector F=[Rc',Rs',Ri',Re'] from S53.
[0135] Obtain contextual information from the state machine: one-hot encoding of the dialogue phase S, global progress scalar P (range 0-1), and short-term trend vectors of each component T=[ΔRc,ΔRs,ΔRi,ΔRe].
[0136] These four types of features are input into four independent embedding layers: the component vector F is passed through a four-dimensional to eight-dimensional fully connected layer EmbF; The stage code S is passed through an embedding lookup table EmbS with a dimension of 3×4; the progress scalar P is passed through a linear transformation layer EmbP from one dimension to four dimensions. The trend vector T is passed through a fully connected layer EmbT, which is from four to eight dimensions. The outputs of all embedding layers are concatenated along the feature dimension to form a unified context feature representation C=concat(EmbF(F),EmbS(S),EmbP(P),EmbT(T)).
[0137] In some embodiments, hierarchical embedding unifies contextual information of different types and scales into a continuous high-dimensional feature space. Component embedding transforms reward values into features, stage embedding transforms discrete categories into vectors, progress embedding transforms continuous scalar relationships into vectors, and trend embedding encodes rate of change information into features. This hierarchical processing allows the network to use different transformation methods for different types of information, preserving the specificity of each type of information. The concatenation operation then fuses these heterogeneous features into a unified representation, providing a comprehensive input containing multi-perspective information for subsequent weight generation.
[0138] S542: Input the contextual feature representation C into a two-layer gated attention network. The first layer is a gated linear unit layer, which calculates G = σ(Wg*C + bg)⊙(Wh*C + bh), where σ is the sigmoid function, ⊙ is the element-wise multiplication, and Wg, bg, Wh, and bh are learnable parameters.
[0139] The second layer is the attention weighting layer, which maintains four queryable weight prototype vectors Proto=[pc,ps,pi,pe], each with the same dimension as the output dimension of G.
[0140] By calculating the cosine similarity between the context representation G and each prototype vector, and then normalizing it using softmax, the initial attention weights αi = softmax(cos(G,pi) / τ) of each component are obtained, where τ is the temperature parameter.
[0141] The final weight vector W=[ωc,ωs,ωi,ωe] is adjusted by the initial attention weights through a residual connection: ωi=αi+β*gatei, where gatei is the gate value of the dimension corresponding to the output G of the gated layer, and β is the scaling factor.
[0142] In some embodiments, the gated linear unit performs soft selection on features using a sigmoid gating signal, filtering out noise information that is irrelevant to the current context and retaining key features.
[0143] Furthermore, the attention mechanism achieves content addressing through learnable prototype vectors: each prototype vector can be seen as a template for a weight allocation pattern, and the network selects the most matching weight allocation pattern by calculating the similarity between the current context and these templates.
[0144] S543: Calculate the uncorrected reward Rraw=Σ(ωi*Ri') using the synthesized weight vector W and the normalized component vector F.
[0145] During the training phase, the parameter updates of the weight synthesis network adopt a meta-learning strategy: every K dialogue rounds, the input features C1,...,CK, the generated weights W1,...,WK, and the long-term rewards G1,...,GK of the corresponding policy network after taking actions in these rounds are collected.
[0146] Define a meta-loss function
[0147] The first term encourages weight allocation to predict long-term returns, the second is KL divergence regularization to prevent excessive weight concentration, and the third is a smoothness constraint. This meta-loss function is then used to update all parameters of the weight synthesis network via backpropagation.
[0148] In some embodiments, the weight synthesis network is trained using a two-layer optimization framework: the inner layer is for conventional dialogue policy learning, and the outer layer is for meta-learning of the weight synthesis network.
[0149] Specifically, the system maintains a circular buffer to store the meta-training data for the most recent K dialogue rounds. In each training round, the weight synthesis network first generates weights based on the current parameters, which are used to calculate the immediate reward and update the policy network. Every K rounds, the system samples a batch of meta-training samples from the buffer. Each sample contains: the contextual features C of that round, the generated weight vector W, and the long-term reward G generated by the action in that round.
[0150] The meta-loss function consists of three parts: the first term Lpred=-ΣiGi*log(Σjω{i,j}*R'{i,j}) is a prediction loss that encourages the weighting to predict long-term returns, so that the weighted reward is positively correlated with the long-term return; The second term Lreg1 = λ1 * KL(W) i ||W uniform ) is the KL divergence regularization, where W uniform It is a uniformly distributed vector [0.25, 0.25, 0.25, 0.25]. This term prevents the weights from being overly concentrated on a certain component and maintains the balance of multi-objective learning. The third term, Lreg2 = λ2 * ||Wi - W{i-1}||^2, is a smoothness constraint, penalizing drastic changes in weights between adjacent rounds. The total loss is Lmeta = Lpred + Lreg1 + Lreg2. The parameters of the weight synthesis network are updated using gradient descent: θnew = θold - η* θLmeta, where η is the meta-learning rate, typically an order of magnitude smaller than the learning rate of the policy network. This meta-learning training alternates with the main policy training, forming a co-evolution.
[0151] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. For example... Figure 5 As shown, S3 specifically includes the following steps: S31: Receive the current state vector St from step S2. The current state vector St contains the encoded information of the current dialogue.
[0152] Retrieve the state vectors [S{t-1},S{t-2},S{t-3}] from the most recent K rounds in the dialogue history buffer, and extract the temporal pattern features Fseq of the state sequence through a lightweight temporal convolutional network (TCN).
[0153] The current state vector St is concatenated with the temporal feature Fseq to obtain the enhanced state representation Senriched=concat(St,Fseq).
[0154] In some embodiments, the causal convolutional structure of the TCN ensures that feature extraction depends only on the current and historical states, conforming to the causality of time series. The linear projection layer allows the network to recalibrate the importance of each dimension of the current state. The concatenation operation integrates current information and historical trends.
[0155] For example, factors such as whether the user gradually loses patience, whether the conversation stalls in acquiring key information, and whether the information entropy fails to decrease significantly over multiple rounds can influence the agent's decision-making. For instance, the agent can initiate a confirmation or summary phase early when it detects a decline in user patience, or try a different questioning strategy when information acquisition stalls.
[0156] S32: Input the enhanced state Senriched into two parallel neural networks simultaneously, and evaluate the network Qeval and the target network Qtarget.
[0157] Both networks have the same structure, being feedforward networks with two hidden layers, and their output dimension is equal to the size of the predefined set of dialogue actions, |A|.
[0158] The Qeval network outputs an instantaneous value estimate for each action a, Qeval(S,a;θ), where θ represents the network parameters.
[0159] The Qtarget network outputs a target value estimate for each action a, Qtarget(S,a;θ'), where θ' is the target network parameter.
[0160] Calculate the combined value of each action a: Qcombined(a) = α * Qeval(S,a;θ) + (1-α) * Qtarget(S,a;θ'), where α is the dynamic mixing coefficient, which is dynamically adjusted based on the consistency of historical predictions of the two networks.
[0161] In some embodiments, the evaluation network Qeval and the target network Qtarget have the same structure: the input layer dimension is equal to the dimension of the augmented state Senriched, the first hidden layer has 256 neurons, the second hidden layer has 128 neurons, and the output layer has |A| neurons.
[0162] Furthermore, both networks use the ReLU activation function, with linear activation in the output layer. The key innovation lies in the introduction of a dynamic mixing coefficient α. The system defines a sliding window to record the predictions of the two networks for the same state-action pair in the most recent N times.
[0163] Furthermore, the correlation coefficient ρ between the two predicted sequences is calculated, and then α is dynamically adjusted based on ρ: α = 0.5 + 0.3 * tanh(ρ), thus α ranges from 0.2 to 0.8. When the two networks predict highly consistent results (ρ close to 1), α approaches 0.8, indicating greater trust in the evaluation network; when the prediction discrepancy ρ approaches 0 or is negative, α approaches 0.5, giving both networks equal weight and reducing the risk of estimation bias from a single network. The comprehensive value Qcombined(a) is calculated in real time during each forward propagation. By utilizing the predictions of two independent networks, the system becomes more resistant to overfitting or anomalous outputs from a single network. Because the dialogue strategy needs to maintain a certain degree of exploratory nature, this embodiment directly translates into higher task success rates and a better user experience.
[0164] S33: For each action a, calculate the uncertainty measure of its value estimate U(a) = |Qeval(S,a;θ) - Qtarget(S,a;θ')| / (|Qeval(S,a;θ)| + ε).
[0165] Define the decision score for each action as Score(a) = Qcombined(a) + β*U(a), where β is the exploration coefficient, which is larger in the early stages of training and decays with increasing training rounds. A softmax selection strategy with a temperature parameter is adopted: selection probability P(a) = exp(Score(a) / τ) / Σ{a'}exp(Score(a') / τ), where τ is the temperature parameter, controlling the degree of randomness in the selection. The current action at is selected based on sampling according to the probability distribution P(a).
[0166] S34: Query the set of available actions, Afeasible, in the current dialogue state. The set of available actions is provided by the dialogue management module of the Rasa framework based on the current slot filling state and dialogue strategy.
[0167] If at If Afeasible is selected, the action with the highest decision score is chosen to replace at. The finally selected action at is then parameter-filled: relevant slot information from the state vector St is injected into the action template to generate the specific system response. An executable action instruction with complete parameters is output.
[0168] In some embodiments, the original action at selected in S33 is compared with Afeasible. If at is not in the feasible set, the module recalculates the decision score for each action in Afeasible and selects the action with the highest score as the replacement.
[0169] Furthermore, during the parameter filling phase, each dialogue action has a corresponding template. Relevant slot information is extracted from the structured dialogue state decoded from the state vector St. For actions requiring parameters, such as providing flight recommendations, the system inserts the already filled slot values (departure location, destination, time, etc.) into the corresponding placeholders in the response template. For confirmation actions, a confirmation summary is generated. Finally, the output is a complete instruction package containing the action type, the specific response text, and any necessary information to be passed to subsequent processes.
[0170] In one embodiment of the present invention, based on step S8, the following is a possible embodiment and its specific implementation will be described in a non-limiting manner. S8 specifically includes the following steps: S81: Synchronously acquire the current state vector output by process S2, the current optimal action output by process S3, the instantaneous reward value output by process S5, and the next state vector output by process S7.
[0171] The completeness and validity of the four types of data are verified one by one: the current state vector is a multi-dimensional standardized vector, the current optimal action must contain valid encoding and complete parameters, the immediate reward value must be within the range of [-2,2], and the next state vector must have the same dimension as the current state vector and be encoded in a standardized manner.
[0172] After verification, the four types of data are uniformly encapsulated: the current state vector and the next state vector retain the original standardized format, the current optimal action is converted into binary code that is common to the Rasa framework and DQN algorithm, and the immediate reward value is appended with a feedback type identifier.
[0173] In some embodiments, steps S2 to S7 are linked via a high-speed data link to ensure that the four types of experience samples can be acquired synchronously, ensuring that the samples can truly reflect the interaction process and meet the requirements of the DQN algorithm's experience replay mechanism. By verifying the completeness and validity of each type of data, invalid data with missing dimensions, incorrect parameters, or abnormal values are filtered out in advance, reducing the interference of invalid data on the training of the policy network. The binary encoding conversion takes into account the requirements of both the Rasa framework and the DQN algorithm.
[0174] S82: Receives the experience samples encapsulated by S81 and obtains the business information of the current dialogue from the Rasa dialogue management module: initial greeting, information collection, solution recommendation, end confirmation, number of filled slots, total number of necessary slots, data acquisition, decision recommendation, question consultation, etc.
[0175] Based on the sample weighting rules in the industry collaboration business rule base, each experience sample is labeled with a scenario-based weight: samples in the information collection stage that involve key slot inquiries are assigned a weight of 1.2; Samples with positive user feedback during the solution recommendation phase are assigned a weight of 1.1. The samples in the initial greeting and closing confirmation stages are assigned a weight of 0.8. Samples with invalid feedback are assigned a weight of 0.5. The weight of samples in other scenarios is assigned to 1.0.
[0176] Furthermore, after the weight labeling is completed, the validity of the samples is verified: the matching between the sample weight and the dialogue scenario and feedback type is verified, and samples with incorrect weight assignment are excluded; the action execution log is retrieved to confirm the actual execution effect of the current optimal action. If the action execution fails, it is determined to be an invalid sample, marked and temporarily stored in the abnormal sample pool, and will not enter the subsequent storage process; valid samples that pass the verification are given a weight identifier and a verification pass stamp, and output to the experience replay pool.
[0177] In some embodiments, different weights are assigned to samples based on the corresponding dialogue stage, action type, and feedback effect, so that the weights of the samples can truly reflect their value to the policy network training. High-value samples receive higher weights, and low-value samples receive lower weights. Secondary verification ensures the accuracy of weight assignment, further filters out invalid samples, and reduces the interference of invalid samples on the policy network training.
[0178] S83: Based on the dialogue stage and user intent type, the experience replay pool is divided into the initial greeting unit, information gathering unit, solution recommendation unit, and end confirmation unit. Each main unit is further divided into sub-units according to user intent type. For example, the information gathering unit is divided into the data acquisition sub-unit and the slot inquiry sub-unit.
[0179] The valid samples output by S82 are stored in the corresponding sub-units according to the dialogue stage and intent type. The storage path follows the structure of main unit, sub-unit, and sample timestamp. The maximum storage capacity of each sub-unit is set in the experience replay pool. When the number of samples in a sub-unit reaches the upper limit, overflow processing is triggered. Samples with a weight lower than 0.8 are deleted first. If the weight of all samples is ≥0.8, the earliest stored sample is deleted.
[0180] Furthermore, every 100 rounds of dialogue, invalid samples are cleaned up for all sub-units, deleting samples marked as invalid, stored for more than a preset time, and not used in training.
[0181] This embodiment categorizes and stores samples according to dialogue stage and user intent, enabling samples to be aggregated by scenario and solving the problem of chaotic sample classification. Combined with the efficient retrieval characteristics of the ChromaDB vector database, it achieves rapid association between samples and business data through correlation indexes, facilitating the synchronous retrieval of business data during training and improving the relevance and efficiency of training.
[0182] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0183] like Figure 6 As shown, this application also provides an electronic device, including a display module 103, a memory 102, a processor 101, a communication module 104, and a computer program stored in the memory and executable on the processor 101. When the processor 101 executes the program, it implements the steps of an intelligent interaction and decision-making method based on a large language model.
[0184] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments described and / or claimed herein.
[0185] In this embodiment, processor 101 may be implemented using at least one of an application-specific integrated circuit, a programmable logic device, a field-programmable gate array, a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such an implementation may be implemented within a controller. For software implementation, implementations such as processes or functions may be implemented with separate software modules that allow the performance of at least one function or operation. Software code may be implemented by a software application (or program) written in any suitable programming language, and the software code may be stored in memory and executed by the controller.
[0186] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like.
[0187] The memory 102 can be used to store software programs and various data. The memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0188] The communication module 104 transmits radio signals to and / or receives radio signals from at least one of a base station, an external terminal, and a server. Such radio signals may include voice call signals, video call signals, or various types of data sent and / or received according to text and / or multimedia messages.
[0189] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An intelligent interaction and decision-making method based on a large language model, characterized in that, The methods include: S1: Receives the current dialogue information input by the user, performs semantic recognition on the current dialogue information through the natural language understanding component, and generates a structured dialogue state that represents the current dialogue process by combining the historical dialogue context. S2: Encode the structured dialogue state into a current state vector; S3: Input the current state vector into the policy network, and the policy network selects the current optimal action from the predefined set of dialogue actions based on the current state vector. The set of dialogue actions includes reply, query, or confirmation. S4: Execute the current optimal action to generate a system response and return it to the user; S5: Receive user feedback on system response and calculate instant reward value based on feedback and preset reward rules; S6: Based on user feedback and the current dialogue state, update the dialogue context and generate the next structured dialogue state based on subsequent user input; S7: Encode the next structured dialogue state into a next state vector; S8: Generate an experience sample containing the current state vector, the current best action, the immediate reward value, and the next state vector, and store the experience sample in the experience replay pool; S9: In training mode, historical experience samples are sampled from the experience replay pool, and the parameters of the policy network are iteratively updated using the historical experience samples. S10: Use the next state vector as the new current state vector, return to step S3, and continue processing subsequent dialogue interactions.
2. The intelligent interaction and decision-making method based on a large language model according to claim 1, characterized in that, S5 specifically includes the following steps: S51: Capture user interaction events, text content, and response time in the current round, and encapsulate them into a feedback data unit containing event code, intent vector, sentiment value, and response interval; S52: Input the feedback data unit into the relationship processing module. The relationship processing module converts the feedback signal into a basic reward indicator value according to the preset feedback indicator association rules and the decay function corresponding to the dialogue round and dialogue stage. S53: The basic reward index values are fed into four parallel computing channels to generate reward components for task completion, user satisfaction, information gain, and dialogue efficiency. S54: Combine the four reward components with the dialogue phase encoding, task progress and short-term change trends of the components, input them into the weighted synthesis network to obtain the synthesis coefficients of each component, and generate uncorrected instant reward values by weighted summation. S55: Performs range pruning, exponential smoothing, and online normalization on the uncorrected instant reward value, and outputs the final instant reward value used for updating the policy network parameters.
3. The intelligent interaction and decision-making method based on a large language model according to claim 2, characterized in that, S53 specifically includes the following steps: S531: Based on the slot information determined by the user feedback in the current round, update the slot probability distribution and calculate the current state entropy. By comparing it with the state entropy of the previous round, obtain the original measure of information gain of the current round of interaction. S532: Retrieve information gain observations from historical similar states using the feature fingerprint of the current dialogue state, and smoothly scale the information gain base coefficients according to the deviation of the current metric from the historical mean to generate calibrated information gain component values. S533: In parallel with the calculation of information gain components, it generates task completion components, user satisfaction components, and dialogue efficiency components by detecting task termination signals, weighted summation of sentiment indicators, and counting dialogue rounds and confirmation times, respectively. S534: Input the reward components into the processor, perform normalization calculations based on the statistical distribution of the historical output values of each component, and output a standardized reward component vector with uniform scale.
4. The intelligent interaction and decision-making method based on a large language model according to claim 2, characterized in that, S54 specifically includes the following steps: S541: Receive the normalized four-dimensional reward component vector from step S53, and obtain the dialogue stage encoding, global progress scalar and short-term trend vectors of each component from the state machine. Input the four types of features into the corresponding embedding layers, perform dimensional transformation and then concatenate to generate context feature representation. S542: Input the context feature representation into the gated linear unit layer for feature filtering, and then calculate its cosine similarity with the four learnable weight prototype vectors through the attention weighting layer. After normalization and residual adjustment, output the synthetic weight vector corresponding to each reward component. S543: The uncorrected reward value is obtained by weighted summation of the weight vector and the normalized reward component vector. During the training phase, the long-term rewards of the network input, output and policy network are collected at fixed intervals. The parameters of the weight synthesis network are updated by backpropagation through the meta-loss function which includes constraints on prediction fitting, distribution balance and temporal smoothing.
5. The intelligent interaction and decision-making method based on a large language model according to claim 2, characterized in that, S3 specifically includes the following steps: S31: Receive the current state vector from step S2 and combine it with the historical state features extracted from the dialogue history to generate an enhanced state representation; S32: Input the enhanced state representation into two neural networks with the same structure. The two networks generate value estimates for each dialogue action, and the outputs of the two networks are weighted and combined. S33: Based on the comprehensive value generated in step S32 and the difference between the two network predictions, combined with the adjustable exploration coefficient and temperature parameter, calculate the selection probability of each action and sample accordingly; S34: Compare and adapt the action selected in step S33 with the set of currently available actions provided by the dialogue management module, and fill the relevant information in the dialogue state into the action template to generate the final executable instruction.
6. The intelligent interaction and decision-making method based on a large language model according to claim 5, characterized in that, In S32, the enhanced state Senriched is simultaneously input into two parallel neural networks to evaluate the network Qeval and the target network Qtarget. Both networks have the same structure, being feedforward networks with two hidden layers, and their output dimension is equal to the size of the predefined set of dialogue actions |A|. The Qeval network outputs an instantaneous value estimate for each action a, Qeval(S,a;θ), where θ is the evaluation network parameter. The Qtarget network outputs a target value estimate for each action a, Qtarget(S,a;θ'), where θ' is the target network parameter; Calculate the combined value of each action a: Qcombined(a) = α * Qeval(S,a;θ) + (1-α) * Qtarget(S,a;θ'), where α is the dynamic mixing coefficient, which is dynamically adjusted based on the consistency of the historical predictions of the two networks. The evaluation network Qeval and the target network Qtarget have the same structure: the input layer dimension is equal to the dimension of the augmented state Senriched, the first hidden layer has 256 neurons, the second hidden layer has 128 neurons, and the output layer has |A| neurons. Calculate the correlation coefficient ρ between the two predicted sequences, and dynamically adjust α based on ρ: α = 0.5 + 0.3 * tanh(ρ). When the two networks have ρ close to 1, trust the evaluation network. When the predicted divergence ρ is close to 0 or negative, both are given equal weight.
7. The intelligent interaction and decision-making method based on a large language model according to claim 5, characterized in that, S33 specifically includes the following steps: For each action a, calculate the uncertainty measure of its value estimate U(a) = |Qeval(S,a;θ) - Qtarget(S,a;θ')| / (|Qeval(S,a;θ)|+ε); Define the decision score for each action as Score(a) = Qcombined(a) + β*U(a), where β is the exploration coefficient. A softmax selection strategy with a temperature parameter is adopted, and the selection probability is P(a) = exp(Score(a) / τ) / Σ{a'}exp(Score(a') / τ), where τ is the temperature parameter, which controls the degree of randomness in the selection.
8. The intelligent interaction and decision-making method based on a large language model according to claim 1, characterized in that, S8 specifically includes the following steps: S81: Synchronously collects output data from the state coding, action selection, reward calculation and next state coding modules, and encapsulates the data into experience samples after completing integrity verification; S82: Receive the encapsulated experience samples, combine the current dialogue scenario information with the preset business rule base, calculate and label a scenario-based weight that reflects the learning value of each experience sample, and verify the validity of the sample. S83: Based on the dialogue stage and user intent, the experience replay pool is divided into multi-level structural units, and the effective samples with labeled weights are classified and stored in the corresponding units, and storage maintenance based on capacity and timeliness is performed.
9. The intelligent interaction and decision-making method based on a large language model according to claim 1, characterized in that, S9 specifically includes the following steps: S91: Perform hierarchical conditional sampling based on business context metadata and priority weights in the experience samples; S92: Input the sampled empirical batches into the evaluation network and the target network to analyze the TD error with target correction; S93: Lock the output layer, fine-tune the hidden layers, jointly fine-tune all parameters, and update the target network.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the intelligent interaction and decision-making method based on a large language model as described in any one of claims 1 to 9.
