AI interaction system based on large language model
By introducing a dynamic target mapping layer and a counterfactual coupling mechanism into the interactive system of a large language model, the problems of multidimensional target adaptation and counterfactual simulation are solved, the collaborative optimization of reinforcement learning is realized, and the learning stability and policy generalization ability of the AI interactive system are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA SERVICE JINKE (SHANDONG) INFORMATION TECH CO LTD
- Filing Date
- 2026-02-26
- Publication Date
- 2026-06-02
AI Technical Summary
Existing interactive systems based on large language models lack multi-dimensional goal adaptation and counterfactual simulation mechanisms, making it difficult to achieve strategy optimization and goal transfer in multi-round interactions. Furthermore, user feedback in real-world environments is sparse and delayed, making it difficult for the learning process to converge stably.
By introducing a dynamic target mapping layer and a counterfactual coupling mechanism, a causal relationship of "target-value function-policy" is formed through multimodal input processing, dynamic target mapping, counterfactual trajectory generation, and reward difference calculation, thereby achieving collaborative optimization and policy adaptation in reinforcement learning.
It improves the learning stability and policy generalization ability of AI interaction systems in multi-task and multi-round interaction scenarios, enhances the level of intelligent decision-making, and improves sample efficiency and policy consistency without increasing the cost of real interaction.
Smart Images
Figure CN122132536A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an AI interaction system based on a large language model. Background Technology
[0002] In recent years, large language models have made significant progress in natural language understanding, semantic generation, and multimodal reasoning, and have been widely applied in scenarios such as intelligent question answering, task planning, and human-computer interaction. However, most existing interactive systems based on large language models rely on static instruction generation or supervised fine-tuning methods, performing well only in single-turn or fixed tasks, and lacking adaptive learning capabilities for multi-turn interactions.
[0003] Reinforcement learning can optimize strategies through environmental feedback and demonstrates strong learning capabilities in dynamic decision-making problems. However, traditional reinforcement learning methods typically rely on fixed reward functions and a single objective optimization direction, making it difficult to cope with the multi-dimensional and dynamically changing objective requirements in human-computer interaction. At the same time, user feedback in real-world environments is sparse and delayed, making it difficult for the policy training process to converge stably.
[0004] Currently, there is a lack of reinforcement learning frameworks that can integrate multi-dimensional goal adaptation and counterfactual simulation mechanisms on the basis of semantic understanding of large language models, so as to achieve a unified closed loop of policy optimization, goal transfer and experience self-reinforcement.
[0005] Therefore, how to provide an AI interaction system based on a large language model is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] One objective of this invention is to propose an AI interaction system based on a large language model. This invention introduces a dynamic target mapping layer and a counterfactual coupling mechanism into a large language model-driven interaction framework, achieving adaptive optimization of multi-dimensional targets and counterfactual reinforcement learning of policies. The system can dynamically adjust the optimization direction of reinforcement learning based on task type, user feedback, and environmental changes, and utilizes the large language model to generate semantically similar candidate trajectories for reward difference calculation, thereby improving sample efficiency and policy generalization ability without increasing the cost of real interaction. By embedding target mapping and gating weight adjustment mechanisms in the actor-critic structure, a causal relationship of "target-value function-policy" is formed, improving the interpretability, stability, and convergence speed of the reinforcement learning process, achieving synergistic optimization of the large language model and reinforcement learning, and significantly enhancing the intelligent decision-making and adaptive learning capabilities of the AI interaction system.
[0007] An AI interaction system based on a large language model according to an embodiment of the present invention includes the following modules:
[0008] A multimodal input processing unit is used to collect multimodal input information from users and perform feature extraction and semantic encoding on the input information to obtain the current interaction state vector;
[0009] The target parsing and weight calculation unit is used to determine a multi-dimensional set of interactive targets based on the interaction state vector, task type and user feedback information, and to calculate the weight coefficient vector of each target.
[0010] An actor subnetwork is used to receive the interaction state vector and output the corresponding action strategy;
[0011] The comment subnetwork is used to receive the interaction state vector and the action policy, and output the current state-action value function;
[0012] The dynamic target mapping layer is used to receive the weight coefficient vector, generate a mapping matrix for adjusting the value function relationship between the actor subnetwork and the comment subnetwork, and perform real-time transformation on the state-action value function output by the comment subnetwork according to the mapping matrix;
[0013] The counterfactual trajectory generation unit is used to call the large language model to generate candidate counterfactual trajectories similar to the current interaction state while the system executes the action strategy;
[0014] The counterfactual coupling layer is used to input the real interaction trajectory and the candidate counterfactual trajectory, calculate the reward difference between the two, and generate a gating weight signal based on the reward difference.
[0015] The parameter joint update unit is used to jointly update the parameters of the actor subnetwork and the commentary subnetwork based on the value function corrected by the dynamic target mapping layer and the gated weight signal generated by the counterfactual coupling layer.
[0016] Furthermore, the modules are integrated through the following methods:
[0017] Collect multimodal input information from users, and perform feature extraction and semantic encoding on the input information to obtain the current interaction state vector;
[0018] Based on the current interaction status, task type, and user feedback, determine the set of multi-dimensional interaction goals and calculate the weight coefficient vector of each goal;
[0019] The interaction state vector is input into the actor subnetwork, and the corresponding action strategy is output.
[0020] Input the interaction state vector and action strategy into the comment subnetwork, and output the current state-action value function;
[0021] The weight coefficient vectors of each target are input into the dynamic target mapping layer to generate a mapping matrix for adjusting the value function relationship between the actor subnetwork and the comment subnetwork. The state-action value function output by the comment subnetwork is then transformed in real time based on the mapping matrix.
[0022] While executing the action strategy, a large language model is used to generate candidate counterfactual trajectories similar to the current interaction state. The counterfactual trajectories include candidate states, actions, and reward estimates.
[0023] The real interaction trajectory and the counterfactual trajectory are input into the counterfactual coupling layer, the reward difference between the two is calculated, and a gating weight signal is generated based on the reward difference.
[0024] Based on the gradient feedback of the value function corrected by the dynamic target mapping layer and the gating weight signal adjusted by the counterfactual coupling layer, the parameters of the actor subnetwork and the commentary subnetwork are jointly updated.
[0025] Furthermore, the step of generating the interaction state vector includes:
[0026] Modality recognition and normalization preprocessing are performed on speech, text, image, and structured data in the multimodal input information, respectively;
[0027] The speech signal is input into the acoustic feature extraction subunit to extract spectral features and intonation parameters.
[0028] The text is input into the text feature extraction subunit, which extracts word vectors and syntactic dependency features;
[0029] The image is input into the visual coding subunit, and the visual embedding vector is extracted.
[0030] Structured data is input into the feature mapping subunit to generate numerical feature vectors;
[0031] The extracted features are semantically aligned and concatenated using a fusion encoder to obtain the current interaction state vector.
[0032] Furthermore, the step of calculating the weight coefficient vector for each objective includes:
[0033] User feedback information is extracted from the multimodal input collected during the previous or current round of interaction, and the feedback information is subjected to semantic recognition and standardization processing.
[0034] Based on the interaction state vector and task context information, identify the task type corresponding to the current interaction;
[0035] Based on task type and standardized user feedback information, a multi-dimensional set of interactive goals is generated, including task dimension, experience dimension and resource dimension.
[0036] Feature correlation calculations are performed on the interaction state vector and the multidimensional interaction target set respectively to obtain the correlation score of each target dimension;
[0037] The initial weights of each target dimension are calculated based on the relevance scores, and the weight coefficient vector is obtained through weighted normalization.
[0038] Furthermore, the action strategy corresponding to the output includes:
[0039] The interaction state vector is input into the input layer of the actor subnetwork;
[0040] Normalization and feature mapping are performed on the input state vector to obtain a standardized state representation;
[0041] Based on the standardized state representation, the action probability distribution is calculated through the hidden layer;
[0042] Based on the action probability distribution, a set of candidate actions is obtained by sampling.
[0043] Perform constraint judgment and filtering on the candidate action set, and remove actions that do not meet the current task conditions;
[0044] The selected actions are then passed through the output layer of the actor subnetwork to generate corresponding action strategies.
[0045] Furthermore, the steps of the output state-action value function include:
[0046] The interaction state vector is combined with the action strategy output by the actor subnetwork to form a state-action pair;
[0047] The state-action pair execution features are concatenated and normalized to obtain a joint feature representation;
[0048] The joint feature representation is input into the input layer of the comment sub-network to calculate the corresponding state-action value;
[0049] The state-action value function output is obtained through multi-layer nonlinear mapping, and the current value function estimation result is generated in the output layer.
[0050] Furthermore, the steps of generating the mapping matrix and performing real-time transformation include:
[0051] The weight coefficient vectors of each target are concatenated with the interaction state vector to form the target condition input;
[0052] Perform linear mapping and normalization on the target condition input to generate a mapping weight matrix;
[0053] The state-action value function output by the comment subnetwork is subjected to matrix operations using the mapping weight matrix to obtain the value function modified by the target conditions.
[0054] During each interactive update cycle, the mapping weight matrix is recalculated based on the latest weight coefficient vector, and the mapping result of the previous cycle is replaced.
[0055] Furthermore, the steps for generating the counterfactual trajectory include:
[0056] During the execution of the action strategy, the current interaction state, input content, and system response are recorded;
[0057] The interaction state vector and the system response content are input into the large language model;
[0058] By calculating the similarity between the current state and the current state in the semantic space using a large language model, historical state samples with similarity higher than a preset threshold are selected.
[0059] Several candidate states are generated based on the historical state samples obtained through screening.
[0060] For each candidate state, the large language model is invoked to infer and generate the corresponding candidate action and reward estimate;
[0061] The candidate states, candidate actions, and estimated returns are combined to form a set of candidate counterfactual trajectories.
[0062] Furthermore, the step of generating the gating weight signal includes:
[0063] Extract the state, action, and reward information from the actual interaction trajectory and the candidate counterfactual trajectory respectively;
[0064] Semantic embedding mapping is performed on the states and actions of the two types of trajectories to obtain the corresponding embedding vectors;
[0065] Calculate the distance metric between the true trajectory and the counterfactual trajectory in the semantic space to obtain the trajectory semantic difference;
[0066] The reward difference is calculated based on the difference between the trajectory semantic difference and the corresponding reward value.
[0067] The reward difference value is input into the gating function to generate a gating weight signal;
[0068] The gating weight signal is then used to adjust the gradient backpropagation strength from the comment subnetwork to the actor subnetwork.
[0069] Furthermore, the joint update includes:
[0070] The value function corrected by the dynamic target mapping layer and the gated weight signal output by the counterfactual coupling layer are input to the parameter update module;
[0071] Calculate the value function loss of the comment subnetwork, and update the parameters of the comment subnetwork based on the loss;
[0072] Calculate the policy gradient of the actor subnetwork and adjust the gradient according to the gating weight signal;
[0073] The weighted policy gradient is used to update the parameters of the actor subnetwork;
[0074] After completing one round of parameter updates, the target weight coefficient vector is re-evaluated, and the next round of interactive training begins.
[0075] The beneficial effects of this invention are:
[0076] This invention introduces a dynamic target mapping layer to establish a mapping relationship between multi-dimensional interactive targets and reinforcement learning value functions. This allows the optimization direction to be dynamically adjusted according to task type, user feedback, and environmental changes, enabling the system to maintain stable learning performance and policy consistency in different interactive scenarios.
[0077] During training, a counterfactual trajectory generation and coupling mechanism is adopted. A candidate trajectory with semantic similarity to the real trajectory is generated by a large language model, and an extended sample set is formed by reward difference calculation. This enhances the policy learning effect and improves the generalization ability without increasing the cost of real interaction.
[0078] By setting up a dynamic target mapping layer and a counterfactual coupling layer in the actor-critic structure, a traceable relationship of "target-value function-policy" is formed, which makes the policy update process have clear target orientation and causal explanation, and facilitates policy optimization and system verification.
[0079] The dynamic target mapping and gating weight adjustment mechanism continuously corrects the gradient feedback direction during training, avoiding learning bias caused by a single reward signal, and effectively improving the convergence speed and learning stability in multi-task and multi-round interaction scenarios.
[0080] This invention uses a large language model as the core component for counterfactual generation and semantic alignment, enabling semantic-level policy expansion and feedback modeling. This allows language understanding and policy optimization to form a collaborative closed loop, thereby significantly improving the intelligent decision-making level of AI systems in open interactive tasks. Attached Figure Description
[0081] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0082] Figure 1 This is a structural diagram of an AI interaction system based on a large language model proposed in this invention;
[0083] Figure 2 This is a flowchart of the interaction state vector generation process for an AI interaction system based on a large language model, as proposed in this invention.
[0084] Figure 3 This is a flowchart of a dynamic target mapping process for an AI interaction system based on a large language model, as proposed in this invention.
[0085] Figure 4 This is a flowchart illustrating the counterfactual generation and coupling process of an AI interaction system based on a large language model, as proposed in this invention. Detailed Implementation
[0086] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0087] refer to Figure 1 - Figure 4 An AI interaction system based on a large language model includes the following modules:
[0088] A multimodal input processing unit is used to collect multimodal input information from users and perform feature extraction and semantic encoding on the input information to obtain the current interaction state vector;
[0089] The target parsing and weight calculation unit is used to determine a multi-dimensional set of interactive targets based on the interaction state vector, task type and user feedback information, and to calculate the weight coefficient vector of each target.
[0090] The actor subnetwork is used to receive the interaction state vector and output the corresponding action strategy.
[0091] The comment subnetwork is used to receive the interaction state vector and the action policy, and output the current state-action value function;
[0092] The dynamic target mapping layer is used to receive the weight coefficient vector, generate a mapping matrix for adjusting the value function relationship between the actor subnetwork and the comment subnetwork, and perform real-time transformation on the state-action value function output by the comment subnetwork according to the mapping matrix;
[0093] The counterfactual trajectory generation unit is used to call the large language model to generate candidate counterfactual trajectories similar to the current interaction state while the system executes the action strategy;
[0094] The counterfactual coupling layer is used to input the real interaction trajectory and the candidate counterfactual trajectory, calculate the reward difference between the two, and generate a gating weight signal based on the reward difference.
[0095] The parameter joint update unit is used to jointly update the parameters of the actor subnetwork and the commentary subnetwork based on the value function corrected by the dynamic target mapping layer and the gated weight signal generated by the counterfactual coupling layer.
[0096] In this embodiment, the modules are interconnected through the following method:
[0097] Collect multimodal input information from users, and perform feature extraction and semantic encoding on the input information to obtain the current interaction state vector;
[0098] Based on the current interaction status, task type, and user feedback, determine the set of multi-dimensional interaction goals and calculate the weight coefficient vector of each goal;
[0099] The interaction state vector is input into the actor subnetwork, and the corresponding action strategy is output.
[0100] Input the interaction state vector and action strategy into the comment subnetwork, and output the current state-action value function;
[0101] The weight coefficient vectors of each target are input into the dynamic target mapping layer to generate a mapping matrix for adjusting the value function relationship between the actor subnetwork and the comment subnetwork. The state-action value function output by the comment subnetwork is then transformed in real time based on the mapping matrix.
[0102] While executing the action strategy, a large language model is used to generate candidate counterfactual trajectories similar to the current interaction state. The counterfactual trajectories include candidate states, actions, and reward estimates.
[0103] The real interaction trajectory and the counterfactual trajectory are input into the counterfactual coupling layer, the reward difference between the two is calculated, and a gating weight signal is generated based on the reward difference.
[0104] Based on the gradient feedback of the value function corrected by the dynamic target mapping layer and the gating weight signal adjusted by the counterfactual coupling layer, the parameters of the actor subnetwork and the commentary subnetwork are jointly updated.
[0105] In this embodiment, the step of generating the interaction state vector includes:
[0106] Modality recognition and normalization preprocessing are performed on speech, text, image and structured data in multimodal input information respectively. Modality recognition automatically classifies the input signal according to the channel identifier and data format. Speech signal is detected by sampling rate, text data is parsed by encoding format, image data is detected by resolution, and structured data is confirmed by field parsing before entering the corresponding processing flow.
[0107] Normalization preprocessing includes time scale standardization, amplitude normalization, and code length unification;
[0108] The speech signal is input into the acoustic feature extraction subunit to extract the spectral features and intonation parameters. The speech signal is pre-emphasized and framed windowed to calculate the Mel frequency cepstral coefficients. The spectral and intonation information is extracted by combining the energy envelope and fundamental frequency curve. After mean normalization, a fixed-dimensional acoustic feature vector is formed.
[0109] The text input is processed by a text feature extraction subunit, which extracts word vectors and syntactic dependency features. The text input is segmented and part-of-speech tagged. A semantic encoding model is called to generate context-related word vectors. The inter-word dependency matrix is calculated based on the dependency syntax tree. The text features are aligned with the speech features in dimension after linear transformation.
[0110] The image is input into the visual encoding subunit. Before input, the image undergoes size normalization and color correction. The image is then processed by a convolutional feature extraction network to output a multi-scale feature map, which is then processed by global average pooling to generate a visual embedding vector.
[0111] Structured data is input into the feature mapping subunit. Before input, the structured data undergoes missing value imputation and standard deviation normalization. Then, it is mapped to the same feature dimension as the text and speech features through linear projection to obtain a unified numerical feature vector.
[0112] The extracted features are semantically aligned and concatenated into vectors by a fusion encoder. The fusion encoder uses a multi-head attention mechanism to calculate the correlation between speech, text, image and structured features in a weighted manner to form a cross-modal alignment matrix. The features of each modality are concatenated according to their weights and processed by a normalization layer to output a unified semantic representation and generate an interaction state vector.
[0113] In this embodiment, the step of calculating the weight coefficient vector of each target includes:
[0114] User feedback information is extracted from the multimodal input collected during the previous or current round of interaction. The feedback information is then subjected to semantic recognition and standardization. The user feedback information includes the user's confirmation statements, rejection statements, rating content, and operation records in the previous round of dialogue. The system uses a large language model to perform intent discrimination and sentiment polarity recognition on the feedback text to obtain a set of feedback tags. The feedback tags are then concatenated with the task state vector to form a feedback embedding representation, which is then input into the task recognition process after being processed by min-max normalization.
[0115] Based on the interaction state vector and task context information, the task type corresponding to the current interaction is identified. The identification process adopts a dual-channel task classification network: the first channel performs semantic clustering on the interaction state vector to identify the semantic scene category, and the second channel uses task keyword matching and hierarchical classification model to determine the task level and outputs the task type identifier through weighted voting.
[0116] Based on the task type and standardized user feedback information, a multi-dimensional interactive target set including task dimension, experience dimension and resource dimension is formed. The system loads the corresponding target template according to the task type, matches the template parameters with the feedback features to generate task dimension indicators, calculates the score of the feedback tag in the satisfaction mapping table to generate experience dimension indicators, and extracts the number of calls, latency and computing power usage based on the system monitoring log to form resource dimension indicators. The three types of indicators together constitute a multi-dimensional target matrix.
[0117] Feature correlation calculations are performed on the interaction state vector and the multidimensional interaction target set respectively to obtain the correlation score of each target dimension. The correlation calculation adopts the embedding similarity algorithm, which expands the interaction state vector and the target matrix by dimension, calculates the cosine similarity, and normalizes to obtain the correlation score vector.
[0118] The initial weights of each target dimension are calculated based on the relevance scores, and the weight coefficient vector is obtained through weighted normalization: the relevance score vector is input into the Softmax layer, the probabilistic initial weights are output, the weight smoothing coefficient β is set according to the task type, and the initial weights are updated with time weighting to obtain the weight coefficient vector.
[0119] In this embodiment, the action strategy corresponding to the output includes:
[0120] The interaction state vector is input to the input layer of the actor sub-network. Normalization and feature mapping are performed on the input state vector to obtain a standardized state representation. The normalization process adopts a combination of batch normalization and layer normalization to eliminate input distribution bias. The feature mapping adopts the nonlinear activation function ReLU.
[0121] Based on the standardized state representation, the action probability distribution is calculated through the hidden layer. The hidden layer consists of several fully connected layers and an attention mechanism layer. The response intensity of the state to different actions is calculated by weighted summation, and the action distribution vector is output, where each dimension corresponds to the probability value of an optional action.
[0122] Based on the action probability distribution, a set of candidate actions is obtained by sampling. The sampling process adopts a mixed strategy of random sampling and greedy sampling, that is, random sampling is performed with probability p to maintain exploration, and greedy sampling is performed with probability (1-p) to maintain utilization.
[0123] The candidate action set is subjected to constraint judgment and screening to remove actions that do not meet the current task conditions. The constraint judgment includes two types: task context constraints and system security constraints. Task context constraints are used to detect whether the action meets the preconditions of the task logic, and system security constraints are used to detect whether the action involves restricted interface calls or operations that exceed the execution permissions. After the constraint judgment, the set of compliant actions is retained.
[0124] The selected actions are passed through the output layer of the actor subnetwork to generate corresponding action policies. The output layer then normalizes the set of compliant actions to form the final policy distribution. The action with the highest probability is selected as the system execution instruction, and the policy parameters are recorded in the model update queue.
[0125] In this embodiment, the step of outputting the state-action value function includes:
[0126] The interaction state vector is combined with the action strategy output by the actor sub-network to form a state-action pair. The combination process involves a connection operation on the feature dimension. The state vector s and the action vector a are concatenated column-wise to generate the input matrix [s,a]. The matrix dimension is consistent with the weight matrix of the input layer of the comment sub-network.
[0127] The state-action pair is subjected to feature concatenation and normalization to obtain a joint feature representation. After feature concatenation, the system performs zero-mean normalization and variance standardization on the joint representation. After normalization, the joint feature dimension is projected to a fixed length through a linear mapping layer.
[0128] The joint feature representation is input into the input layer of the comment sub-network to calculate the corresponding state-action value. The input layer of the comment sub-network adopts a fully connected structure and performs linear weighting and non-linear activation operations on the input vector to generate the first layer of hidden representation. During the forward propagation process, the network estimates the expected value corresponding to the input state-action pair and passes the result to the next layer for calculation.
[0129] The state-action value function output is obtained through multi-layer nonlinear mapping, and the current value function estimation result is generated in the output layer. The multi-layer nonlinear mapping consists of several fully connected layers, residual connections and activation functions. LeakyReLU is used to implement nonlinear feature extraction, and the output layer is a single-node linear layer.
[0130] In this embodiment, the steps of generating the mapping matrix and performing real-time transformation include:
[0131] The weight coefficient vectors of each target and the interaction state vector are concatenated to form the target condition input. The concatenation operation is performed sequentially along the feature dimension. The weight coefficient vectors and the interaction state vectors are concatenated to form a joint feature, which serves as the input to the dynamic target mapping layer.
[0132] The target condition input is subjected to linear mapping and normalization processing to generate a mapping weight matrix. The linear mapping uses a trainable weight matrix for affine transformation. The normalization processing stabilizes the input distribution through layer normalization and uses the Sigmoid activation function to constrain the mapping weights in the [0,1] interval to obtain the mapping weight matrix.
[0133] The state-action value function output by the comment subnetwork is subjected to matrix operation using the mapping weight matrix to obtain the value function modified by the target conditions. The value function vector output by the comment subnetwork is multiplied by the mapping weight matrix to form the target-corrected value function. The matrix multiplication causes the weights corresponding to different target dimensions to affect each component of the value function.
[0134] Within each interactive update cycle, the mapping weight matrix is recalculated based on the latest weight coefficient vector, and the mapping result of the previous cycle is replaced. In each training or inference cycle, the system calls the latest weight coefficient vector, regenerates the mapping weight matrix through the above linear mapping and normalization process, and replaces the matrix of the previous cycle in a sliding update manner within the dynamic target mapping layer.
[0135] In this embodiment, the step of generating the counterfactual trajectory includes:
[0136] During the execution of the action strategy, the current interaction state, input content, and system response are recorded;
[0137] The interaction state vector and the system response content are input into the large language model. The large language model adopts an instruction fine-tuning structure, and the input format is "<state description> + <system response>". The state features and response text are transformed into semantic vector representations through an embedding layer. The model generates a set of candidate state descriptions that are adjacent to the current interaction semantics in the latent space.
[0138] The similarity between the current state and the current state in the semantic space is calculated using a large language model. Historical state samples with similarity higher than a preset threshold are selected. The cosine similarity index is used for similarity calculation.
[0139] Based on the historical state samples obtained from the screening, several candidate states are generated. The system inputs the candidate states into the generator decoder part of the language model, expands the context through conditional generation, and generates state descriptions that are similar to but not exactly the same as the real trajectory in semantic structure, thus forming a set of counterfactual states.
[0140] For each candidate state, the large language model is invoked to generate the corresponding candidate action and reward estimate. During the inference phase, the candidate state is used as the input condition, the language model is used to output possible system responses or action plans, and the corresponding reward value is estimated through the internal evaluation function. The reward estimate is calculated based on the semantic score of the large language model and the matching degree of the task target, forming a set of candidate action-reward pairs.
[0141] The candidate states, candidate actions, and estimated returns are combined to form a set of candidate counterfactual trajectories.
[0142] In this embodiment, the step of generating the gating weight signal includes:
[0143] Extract the state, action, and reward information from the actual interaction trajectory and the candidate counterfactual trajectory respectively;
[0144] Semantic embedding mapping is performed on the states and actions of the two types of trajectories to obtain corresponding embedding vectors. The semantic embedding mapping is completed through a weighted embedding network, which maps the states and actions to a unified latent space.
[0145] The distance between the true trajectory and the counterfactual trajectory in the semantic space is calculated to obtain the semantic difference of the trajectory. The system uses a joint metric of Euclidean distance and cosine similarity to comprehensively evaluate the semantic similarity between the two types of trajectories.
[0146] ;
[0147] in, This represents the semantic difference between the i-th counterfactual trajectory and the true trajectory. This is the true state vector at the current time step. Let i be the candidate counterfactual state vector. Let Euclidean distance be the state vector. The cosine similarity is the similarity between the state vectors. This is a balance coefficient between Euclidean distance and cosine similarity; a smaller value indicates a better balance. This indicates that the counterfactual trajectory is semantically closer to the actual interaction trajectory.
[0148] The reward difference is calculated based on the difference between the trajectory semantic difference and the corresponding reward value.
[0149] ;
[0150] in, Let represent the reward difference between the i-th counterfactual trajectory and the true trajectory. The actual reward value of the true trajectory. The counterfactual reward value estimated by the large language model. This represents the semantic difference between the i-th counterfactual trajectory and the true trajectory.
[0151] The reward difference value is input into the gating function to generate a gating weight signal. The gating function adopts the Sigmoid form.
[0152] The gating weight signal is then used to adjust the gradient backpropagation strength from the comment subnetwork to the actor subnetwork.
[0153] In this embodiment, the joint update includes:
[0154] The value function corrected by the dynamic target mapping layer and the gated weight signal output by the counterfactual coupling layer are input to the parameter update module;
[0155] Calculate the value function loss of the comment subnetwork, and update the parameters of the comment subnetwork based on the loss, wherein the value function loss is in the form of mean squared error;
[0156] ;
[0157] in, The value function is corrected by the dynamic target mapping layer. The target return value.
[0158] The system calculates the policy gradient of the actor subnetwork and adjusts the gradient by weighting it according to the gating weight signal. Based on the policy distribution of the actor subnetwork, the system calculates the gradient direction corresponding to each state-action pair and uses the gating weight signal as an adjustment factor to weight the magnitude of the policy gradient.
[0159] The weighted policy gradient is used to update the parameters of the actor subnetwork. After the parameter update is completed, the system synchronously records the loss change trend and gradient convergence index of the current training round.
[0160] After completing one round of parameter updates, the target weight coefficient vector is re-evaluated, and the next round of interactive training begins. The system recalculates the target weight coefficients based on the latest value function changes and policy performance, and inputs the updated weights into the dynamic target mapping layer.
[0161] Example 1:
[0162] To verify the feasibility of this invention in practice, it was applied to a certain intelligent customer service interaction system. The performance differences between the "dynamic target mapping + antifactual coupling reinforcement learning" architecture proposed in this invention and the conventional reinforcement learning interaction system - DDPG and PPO baseline model were compared and analyzed.
[0163] The large language model uses the LLaMA2-7B, which has been fine-tuned with instructions, as the generation core; the multimodal input data comes from real dialogue logs of the intelligent customer service platform, totaling 8,000 rounds of task interaction samples, including voice text (45%), plain text (40%), and image-assisted interaction (15%).
[0164] The parameter settings for each submodule are as follows:
[0165] Actor subnetwork: 3 fully connected layers, hidden layer dimensions 128, 64, and 32, learning rate... ;
[0166] Comment subnetwork: 4-layer residual fully connected network, learning rate ;
[0167] The dimension of the dynamic target mapping layer mapping matrix is ;
[0168] Number of candidate trajectories sampled by the counterfactual coupling layer Counterfactual generation threshold similarity ;
[0169] Discount factor Gated temperature parameters Target smoothing coefficient .
[0170] Table 1 Comparison Table
[0171] Model Number Model Name Strategy Mechanism Does it include a dynamic mapping layer? Does it contain counterfactual coupling? Large language model participation M1 DDPG-Basic Basic reinforcement learning no no no M2 DDPG-LLM Reinforcement learning + contextual guidance of large language models no no yes M3 Proposed (Solution of this invention) Dynamic target mapping + counterfactual coupling dual-path optimization yes yes yes
[0172] All three models were trained for 2000 epochs on the same task set. Evaluation metrics included:
[0173] Average task success rate, average user satisfaction score, strategy stability, average computation latency, and average convergence rounds.
[0174] Table 2 Experimental Results
[0175] Model Success rate (%) Satisfaction Strategy fluctuation Delay (ms) Convergence cycles M1 78.3 0.71 0.183 41.5 1570 M2 85.6 0.79 0.156 52.7 1320 M3 93.2 0.86 0.093 48.9 910
[0176] As shown in Table 2, the system of this invention (M3) improves the average task success rate by approximately 14.9% compared to the basic reinforcement learning model (M1), and by 7.6% compared to M2, which has a large language model but no structure optimization. The satisfaction index increased from 0.71 to 0.86, indicating that the dynamic target mapping layer can effectively allocate the multi-objective weights of task dimension, experience dimension and resource dimension, making the interactive response closer to the user's needs.
[0177] The policy variability (standard deviation) of M3 is 0.093, which is much lower than that of M1 and M2, indicating that the counterfactual coupling layer plays a role in suppressing high-variance gradients and stabilizing policy updates during training. By weighting the gating signal, the system only reinforces learning on high-confidence samples, avoiding learning oscillations caused by low-value samples.
[0178] With the same learning rate and sample size, the M3 model converges in approximately 910 epochs, shortening the training cycle by about 42% compared to M1. This is attributed to the "dynamic target mapping layer" of this invention, which adjusts the gradient direction of the value function in real time, enabling the actor subnetwork to quickly find the optimal policy direction in the multi-target space.
[0179] Although a counterfactual generation module and a dynamic mapping layer are introduced, the latency only increases by 7.4ms compared to the traditional model (M1), which is still within the acceptable range for real-time response (<60ms). Therefore, the present invention achieves a balance between interpretability, performance, and real-time performance.
[0180] In the gray-scale deployment of the intelligent customer service platform, after processing 500 rounds of real user interactions, the complaint rate decreased by 38.5% and the average number of interaction rounds decreased by 21.7%, indicating that it can achieve efficient decision-making and context alignment in real business. At the same time, the GPU utilization rate during the model training process was about 62%, and no significant resource bottlenecks were found, demonstrating engineering-level feasibility.
[0181] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. An AI interaction system based on a large language model, characterized in that, Includes the following modules: A multimodal input processing unit is used to collect multimodal input information from users and perform feature extraction and semantic encoding on the input information to obtain the current interaction state vector; The target parsing and weight calculation unit is used to determine a multi-dimensional set of interactive targets based on the interaction state vector, task type and user feedback information, and to calculate the weight coefficient vector of each target. An actor subnetwork is used to receive the interaction state vector and output the corresponding action strategy; The comment subnetwork is used to receive the interaction state vector and the action policy, and output the current state-action value function; The dynamic target mapping layer is used to receive the weight coefficient vector, generate a mapping matrix for adjusting the value function relationship between the actor subnetwork and the comment subnetwork, and perform real-time transformation on the state-action value function output by the comment subnetwork according to the mapping matrix; The counterfactual trajectory generation unit is used to call the large language model to generate candidate counterfactual trajectories similar to the current interaction state while the system executes the action strategy; The counterfactual coupling layer is used to input the real interaction trajectory and the candidate counterfactual trajectory, calculate the reward difference between the two, and generate a gating weight signal based on the reward difference. The parameter joint update unit is used to jointly update the parameters of the actor subnetwork and the commentary subnetwork based on the value function corrected by the dynamic target mapping layer and the gated weight signal generated by the counterfactual coupling layer.
2. The AI interaction system based on a large language model according to claim 1, characterized in that, Inter-module communication is achieved through the following methods: Collect multimodal input information from users, and perform feature extraction and semantic encoding on the input information to obtain the current interaction state vector; Based on the current interaction status, task type, and user feedback, determine the set of multi-dimensional interaction goals and calculate the weight coefficient vector of each goal; The interaction state vector is input into the actor subnetwork, and the corresponding action strategy is output. Input the interaction state vector and action strategy into the comment subnetwork, and output the current state-action value function; The weight coefficient vectors of each target are input into the dynamic target mapping layer to generate a mapping matrix for adjusting the value function relationship between the actor subnetwork and the comment subnetwork. The state-action value function output by the comment subnetwork is then transformed in real time based on the mapping matrix. While executing the action strategy, a large language model is used to generate candidate counterfactual trajectories similar to the current interaction state. The counterfactual trajectories include candidate states, actions, and reward estimates. The real interaction trajectory and the counterfactual trajectory are input into the counterfactual coupling layer, the reward difference between the two is calculated, and a gating weight signal is generated based on the reward difference. Based on the gradient feedback of the value function corrected by the dynamic target mapping layer and the gating weight signal adjusted by the counterfactual coupling layer, the parameters of the actor subnetwork and the commentary subnetwork are jointly updated.
3. The AI interaction system based on a large language model according to claim 2, characterized in that, The steps for generating the interaction state vector include: Modality recognition and normalization preprocessing are performed on speech, text, image, and structured data in the multimodal input information, respectively; The speech signal is input into the acoustic feature extraction subunit to extract spectral features and intonation parameters. The text is input into the text feature extraction subunit, which extracts word vectors and syntactic dependency features; The image is input into the visual coding subunit, and the visual embedding vector is extracted. Structured data is input into the feature mapping subunit to generate numerical feature vectors; The extracted features are semantically aligned and concatenated using a fusion encoder to obtain the current interaction state vector.
4. The AI interaction system based on a large language model according to claim 2, characterized in that, The steps for calculating the weight coefficient vector of each objective include: User feedback information is extracted from the multimodal input collected during the previous or current round of interaction, and the feedback information is subjected to semantic recognition and standardization processing. Based on the interaction state vector and task context information, identify the task type corresponding to the current interaction; Based on task type and standardized user feedback information, a multi-dimensional set of interactive goals is generated, including task dimension, experience dimension and resource dimension. Feature correlation calculations are performed on the interaction state vector and the multidimensional interaction target set respectively to obtain the correlation score of each target dimension; The initial weights of each target dimension are calculated based on the relevance scores, and the weight coefficient vector is obtained through weighted normalization.
5. The AI interaction system based on a large language model according to claim 2, characterized in that, The action strategy corresponding to the output includes: The interaction state vector is input into the input layer of the actor subnetwork; Normalization and feature mapping are performed on the input state vector to obtain a standardized state representation; Based on the standardized state representation, the action probability distribution is calculated through the hidden layer; Based on the action probability distribution, a set of candidate actions is obtained by sampling. Perform constraint judgment and filtering on the candidate action set, and remove actions that do not meet the current task conditions; The selected actions are then passed through the output layer of the actor subnetwork to generate corresponding action strategies.
6. The AI interaction system based on a large language model according to claim 2, characterized in that, The steps of the output state-action value function include: The interaction state vector is combined with the action strategy output by the actor subnetwork to form a state-action pair; The state-action pair execution features are concatenated and normalized to obtain a joint feature representation; The joint feature representation is input into the input layer of the comment sub-network to calculate the corresponding state-action value; The state-action value function output is obtained through multi-layer nonlinear mapping, and the current value function estimation result is generated in the output layer.
7. The AI interaction system based on a large language model according to claim 2, characterized in that, The steps for generating the mapping matrix and performing real-time transformation include: The weight coefficient vectors of each target are concatenated with the interaction state vector to form the target condition input; Perform linear mapping and normalization on the target condition input to generate a mapping weight matrix; The state-action value function output by the comment subnetwork is subjected to matrix operations using the mapping weight matrix to obtain the value function modified by the target conditions. During each interactive update cycle, the mapping weight matrix is recalculated based on the latest weight coefficient vector, and the mapping result of the previous cycle is replaced.
8. The AI interaction system based on a large language model according to claim 2, characterized in that, The steps for generating the counterfactual trajectory include: During the execution of the action strategy, the current interaction state, input content, and system response are recorded; The interaction state vector and the system response content are input into the large language model; By calculating the similarity between the current state and the current state in the semantic space using a large language model, historical state samples with similarity higher than a preset threshold are selected. Several candidate states are generated based on the historical state samples obtained through screening. For each candidate state, the large language model is invoked to infer and generate the corresponding candidate action and reward estimate; The candidate states, candidate actions, and estimated returns are combined to form a set of candidate counterfactual trajectories.
9. The AI interaction system based on a large language model according to claim 2, characterized in that, The steps for generating the gated weight signal include: Extract the state, action, and reward information from the actual interaction trajectory and the candidate counterfactual trajectory respectively; Semantic embedding mapping is performed on the states and actions of the two types of trajectories to obtain the corresponding embedding vectors; Calculate the distance metric between the true trajectory and the counterfactual trajectory in the semantic space to obtain the trajectory semantic difference; The reward difference is calculated based on the difference between the trajectory semantic difference and the corresponding reward value. The reward difference value is input into the gating function to generate a gating weight signal; The gating weight signal is then used to adjust the gradient backpropagation strength from the comment subnetwork to the actor subnetwork.
10. An AI interaction system based on a large language model according to claim 2, characterized in that, The joint update includes: The value function corrected by the dynamic target mapping layer and the gated weight signal output by the counterfactual coupling layer are input to the parameter update module; Calculate the value function loss of the comment subnetwork, and update the parameters of the comment subnetwork based on the loss; Calculate the policy gradient of the actor subnetwork and adjust the gradient according to the gating weight signal; The weighted policy gradient is used to update the parameters of the actor subnetwork; After completing one round of parameter updates, the target weight coefficient vector is re-evaluated, and the next round of interactive training begins.