Intelligent agent construction method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of natural language processing, and provides an agent construction method and device, electronic equipment and a storage medium, and the method comprises the steps: generating a response result through an agent model for a training sample, and calculating the sample mastery degree of the agent model for the training sample according to an award score for the response result; according to the sample mastering degree, determining a target training mode for the training sample from a supervised learning mode and a reinforcement learning mode; determining a target loss value based on the target training mode; and updating the parameters of the agent model by using the target loss value to obtain a target agent. According to the method, the sample mastery degree of the agent model is calculated in real time according to the training samples, and the target training mode is adaptively switched between the supervised learning mode and the reinforcement learning mode, so that the problem of training imbalance caused by data long-tail distribution is effectively solved; and flow splitting and performance bottleneck caused by simple staged training are avoided.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, electronic device, and storage medium for constructing an intelligent agent. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent agents based on large language models (LLM) are being used more and more widely in the field of financial customer service.

[0003] Existing methods for building intelligent agents typically employ supervised fine-tuning (SFT) or reinforcement learning (RL) for training. However, SFT relies on high-quality labeled data, has limited generalization ability, and performs poorly on long-tail problems; pure RL training suffers from cold-start difficulties, training instability, and blind exploration. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for constructing intelligent agents, in order to overcome the deficiencies existing in the prior art.

[0005] This invention provides a method for constructing an intelligent agent, comprising the following steps: For the training samples, the agent model generates response results, and the sample mastery of the agent model over the training samples is calculated based on the reward score for the response results. Based on the mastery of the samples, a target training mode for the training samples is determined from supervised learning mode and reinforcement learning mode. The target loss value is determined based on the target training mode; wherein, when the target training mode is the supervised learning mode, the target loss value is determined according to the label of the training sample; when the target training mode is the reinforcement learning mode, the target loss value is determined according to the reward score. The parameters of the agent model are updated using the target loss value to obtain the target agent.

[0006] According to a method for constructing an intelligent agent provided by the present invention, the step of generating response results using an intelligent agent model for training samples, and calculating the sample mastery of the intelligent agent model over the training samples based on the reward score for the response results, includes: The intelligent agent model is controlled to perform multiple inferences and generation operations on the training samples to obtain multiple response results, and the reward score for each response result is determined. The proportion of response results whose reward scores exceed a preset score threshold is statistically analyzed, and this proportion is determined as the sample mastery of the agent model over the training samples.

[0007] According to a method for constructing an intelligent agent provided by the present invention, determining the reward score for each response result includes: Obtain the format correctness score, semantic accuracy score, and thought chain density penalty for each response result; The format correctness score, the semantic accuracy score, and the thought chain density penalty term are weighted and summed to obtain the reward score for the corresponding response result; The thought chain density penalty term is used to apply a negative penalty to the corresponding reward score when the length of the corresponding response result exceeds a preset length threshold.

[0008] According to a method for constructing an intelligent agent provided by the present invention, the step of determining the thought chain density penalty term includes: Determine the actual generated length of the thought chain portion in the response result; Calculate the length difference between the actual generated length and the preset length threshold; The thought chain density penalty term is calculated based on the length difference.

[0009] According to a method for constructing an intelligent agent provided by the present invention, determining a target training mode for the training samples from supervised learning modes and reinforcement learning modes based on the sample mastery includes: If the sample mastery is less than or equal to the mastery threshold, the supervised learning mode is used as the target training mode. If the sample mastery is greater than the mastery threshold, the reinforcement learning mode is used as the target training mode.

[0010] According to a method for constructing an agent provided by the present invention, when the target training mode is the supervised learning mode, before determining the target loss value based on the target training mode, the method further includes: The input portion of the training sample is constructed, and the input portion includes user intent classification labels, short thought chain guidance, and tool call instructions; Construct the label part of the training sample, which includes the thought chain analysis content of the user's intent and the response generation content; The thought chain analysis content is constrained within a preset first character limit, the tool call command is constrained to a preset data exchange format, and the response generation content is constrained within a preset second character limit.

[0011] According to a method for constructing an agent provided by the present invention, after updating the parameters of the agent model using the target loss value to obtain the target agent, the method further includes: Receive natural language requests from user input; The target intelligent agent is used for streaming reasoning to generate thought chain content containing intent categories; When the end marker of the intent category of the thought chain content output is detected, an external tool call request corresponding to the intent category is asynchronously triggered.

[0012] According to the agent construction method provided by the present invention, before generating response results using an agent model for training samples, the method further includes: Obtain an initial large language model and an initial dataset, wherein the initial dataset contains instruction compliance data with intent classification labels and standard data format instructions; The initial large language model is subjected to supervised fine-tuning using the initial dataset to obtain the agent model.

[0013] According to a method for constructing an agent provided by the present invention, determining the target loss value based on the target training mode includes: When the target training mode is the supervised learning mode, the basic supervised loss is calculated based on the standard labels of the training samples, and the product of the basic supervised loss and the fine-tuning weight parameters is determined as the target loss value. When the target training mode is the reinforcement learning mode, the basic reinforcement loss is calculated based on the reward score, and the product of the basic reinforcement loss and the reinforcement weight parameter is determined as the target loss value.

[0014] According to a method for constructing an agent provided by the present invention, the step of calculating the basic reinforcement loss based on the reward score includes: Obtain the reward baseline calculated for the training samples, where the reward baseline represents the average reward level of the agent model in the current state; The difference between the reward score and the reward baseline is calculated to obtain the advantage function value; The basic reinforcement loss is calculated by multiplying the log probability gradient of the response result generated by the agent model with the advantage function value.

[0015] The present invention also provides an intelligent agent construction device, comprising the following modules: The calculation module is used to generate response results using the agent model for the training samples, and to calculate the sample mastery of the agent model for the training samples based on the reward score for the response results. The selection module is used to determine the target training mode for the training samples from supervised learning modes and reinforcement learning modes based on the sample mastery. A determination module is used to determine a target loss value based on the target training mode; wherein, when the target training mode is the supervised learning mode, the target loss value is determined according to the label of the training sample; when the target training mode is the reinforcement learning mode, the target loss value is determined according to the reward score. The update module is used to update the parameters of the agent model using the target loss value to obtain the target agent.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the intelligent agent construction method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent agent construction method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the intelligent agent construction method as described above.

[0019] The intelligent agent construction method, apparatus, electronic device, and storage medium provided by this invention calculate the sample mastery of the intelligent agent model in real time for training samples, and adaptively switch between supervised learning mode and reinforcement learning mode accordingly. Then, based on the determined target training mode, a target loss value is calculated to update the model parameters. Because it can teach according to the model's mastery of different samples—using labels for supervised correction of difficult samples that are not yet mastered, and using reward scores for reinforcement exploration of easy samples that are already mastered—it effectively solves the training imbalance problem caused by the long-tail distribution of data in complex scenarios, avoids the process fragmentation and performance bottlenecks caused by simple staged training, and ultimately constructs a target intelligent agent with both high accuracy and strong robustness. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1This is a flowchart illustrating the intelligent agent construction method provided by the present invention.

[0022] Figure 2 This is a schematic diagram of the target intelligent agent interaction link framework provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the intelligent agent construction device provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Traditional human customer service centers, operating on a capital-intensive model, face multiple challenges, including high labor costs, long training cycles, limited service hours, and difficulties in standardizing service quality. With the widespread adoption of mobile internet and the shift of customer habits online, the demand for 24 / 7 instant service has surged, exponentially increasing the pressure on traditional models. Against this backdrop, the first generation of intelligent customer service, based on keyword matching and simple natural language processing, emerged. Their core value lies in automating the processing of massive amounts of repetitive, standardized inquiries, freeing human agents from tedious basic tasks and significantly reducing operating costs. However, this stage of intelligent customer service is essentially still a passive question-and-answer machine, with a stiff interactive experience, unable to understand complex contexts, and lacking the ability to handle multi-turn, cross-business scenario complex tasks, resulting in limited improvement in customer satisfaction.

[0027] With the rapid development of artificial intelligence technology, especially the emergence of LLM (Limited Learning Model), intelligent customer service has undergone a transformation. Currently, LLM-based agent technology is gradually becoming mainstream. To build high-performance agents, the industry typically adopts two main training paradigms: one is a pre-arranged workflow-based approach, and the other is an interaction chain consisting of one or more agents. While the former is effective, it is tied to a specific LLM version and has poor scalability; the latter usually uses SFT (Self-Functional Theory) for vertical scenario transfer or further combines it with RL (Research and Development) for optimization. SFT can effectively keep the model consistent with human demonstrations, but it faces difficulties in data labeling and limited generalization ability when facing long-term tasks in dynamic environments; end-to-end RL training methods can improve robustness, but they suffer from high costs and convergence difficulties.

[0028] While LLM enhances the intent understanding and generation capabilities of customer service systems, it still faces numerous challenges in specific scenarios such as finance. Current training methods often employ a phased strategy, such as performing SFT before RL, or using only a single mode. This approach is prone to overfitting for simple and clear intents; while for complex and varied logical reasoning tasks, the model often struggles to converge due to a lack of effective exploration mechanisms. Furthermore, to ensure the accuracy of responses, the model often generates lengthy thought chains, leading to increased inference latency and failing to meet the extremely high requirements of financial customer service for immediate response. The traditional pipeline-based workflow, namely intent recognition-slot extraction-state management-response generation, is not only lengthy and prone to cascading errors, but also suffers from information loss between modules, resulting in a poor user experience and making it difficult to truly achieve an efficient closed loop of "perception-planning-action".

[0029] To address this issue, this invention provides a method for constructing an intelligent agent, aiming to dynamically and adaptively switch between supervised learning and reinforcement learning modes by introducing a sample mastery assessment mechanism. Specifically, for training samples, this invention utilizes the intelligent agent model to generate response results and calculate reward scores, thereby assessing the model's mastery of the sample. Based on the level of mastery, it intelligently selects between supervised learning using labels for correction and reinforcement learning using reward scores for exploration, and calculates the target loss value to update the model parameters accordingly. This method achieves differentiated processing of samples of varying difficulty during the same training process, ensuring rapid construction of basic capabilities while improving the model's generalization ability and response speed in complex scenarios. This results in the construction of a financial intelligent customer service agent capable of accurately understanding user intent, possessing efficient reasoning capabilities, and responding quickly, effectively reducing traditional steps and improving end-to-end interaction performance.

[0030] It should be noted that the intelligent agent construction method provided by this invention can be applied to financial intelligent customer service scenarios to achieve accurate intent understanding and business processing, as well as to medical consultation scenarios to assist doctors in analyzing conditions and recommending treatment plans, and also to industrial inspection scenarios to identify faults and dispatch maintenance tools through intelligent agents. This invention does not specifically limit its application in these areas. To facilitate the explanation of the technical solution of this invention, the following embodiments are all illustrated using the application in a financial intelligent customer service scenario as an example.

[0031] in, Figure 1 This is a flowchart illustrating the intelligent agent construction method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0032] Step 110: For the training samples, use the agent model to generate response results, and calculate the agent model's sample mastery of the training samples based on the reward score for the response results.

[0033] Here, training samples refer to data pairs used to train the intelligent agent model. They typically contain an input part and a desired output part, i.e., labels. In the scenario of financial intelligent customer service, the input part of the training samples can be a natural language query from the user about financial services, such as "Please help me check my credit card statement from last month and analyze the largest expense."

[0034] An intelligent agent model refers to an AI model built on LLM that has the ability to understand intent, reason and plan, and invoke tools. As the core brain of intelligent customer service, this model is responsible for receiving user input and generating corresponding output.

[0035] During training, for each training sample, it is first input into the current agent model. The agent model generates a corresponding response based on its own parameter distribution. This response can be a natural language reply, a reasoning process containing a specific thought process, or a command to invoke an external tool. For example, for the input of querying a credit card bill, the response might include the thought process of "first calling the bill query API to obtain data, and then traversing the data to find the largest expense item" and the corresponding API call code.

[0036] To evaluate the quality of the responses generated by the model, this embodiment introduces a reward scoring mechanism. The reward score is a numerical metric used to quantify the quality of the response. As an optional embodiment, the reward score can be calculated based on multiple dimensions, such as whether the response format conforms to specifications, whether the semantics accurately answer the user's question, and whether the reasoning process is concise and efficient. A higher score indicates a better quality response.

[0037] Furthermore, based on the reward score for this response, the agent model's sample mastery of the training sample is calculated. Sample mastery characterizes the degree to which the agent model has learned or is competent with the training sample.

[0038] As an optional implementation, sample mastery can be determined by comparing the reward score with a preset threshold. If the reward score is higher than the threshold, the model is considered to have a good mastery of the sample; otherwise, it is considered to have a poor mastery.

[0039] Furthermore, considering the potential randomness of a single generation, multiple sampling can be used to more accurately assess mastery. For example, the agent model can generate K different responses to the same training sample, and the reward score for each response can be calculated. Then, the proportion of these K generation responses with reward scores exceeding a preset score threshold can be counted, and this proportion can be used as the sample mastery.

[0040] Step 120: Based on the sample mastery, determine the target training mode for the training samples from supervised learning mode and reinforcement learning mode.

[0041] Specifically, supervised learning refers to a training method that uses pre-labeled standard answers (i.e., labels) from training samples to guide model updates. In this mode, the model aims to mimic the standard answer as closely as possible, typically achieved by minimizing the cross-entropy loss between the predicted result and the standard answer. This mode is similar to a teacher's hands-on instruction, suitable for rapid correction when the model has not yet mastered the knowledge.

[0042] Reinforcement learning refers to a training method that uses reward signals to guide model updates. In this mode, the model explores different generation paths and adjusts its strategy based on the reward scores obtained, with the goal of maximizing expected rewards. This mode is similar to a student exploring problems on their own and is suitable for models that have already achieved a certain level of proficiency and need to further improve their generalization ability and find better solutions.

[0043] The target training mode refers to the training method ultimately adopted for the specific training samples. The core idea of ​​this embodiment is "personalized instruction." Since financial data often exhibits a long-tail distribution, it contains both many simple common problems and complex long-tail problems. If the same mode is applied to all samples equally, it can easily lead to overfitting of simple samples or underfitting of complex samples.

[0044] Therefore, this embodiment makes dynamic decisions based on the sample mastery level calculated in step S110. As an optional embodiment, a mastery level threshold can be set. When the sample mastery level is less than or equal to this threshold, it indicates that the model has not yet mastered the sample and its corresponding knowledge points, and is in a state of ignorance. At this time, if the model is forced to execute reinforcement learning mode, it is easy to lead to blind trial and error or even training divergence. Therefore, the target training mode is determined to be supervised learning mode, using standard answers for strong supervision to help the model quickly establish basic cognition.

[0045] When the sample mastery exceeds this threshold, it indicates that the model has gained a certain understanding of the sample and is in a state of proficiency or familiarity. At this point, continuing supervised learning will yield diminishing returns. Therefore, the target training mode is determined to be reinforcement learning, which encourages the model to explore based on existing knowledge and attempt to generate better results than the standard answer, such as results with more refined logic and faster response speed, thereby improving the model's robustness and inference efficiency.

[0046] The adaptive switching mechanism provided in this embodiment effectively solves the problems of process fragmentation and difficulty in dealing with the imbalance of long-tail data distribution in traditional phased training (SFT first, then RL).

[0047] Step 130: Determine the target loss value based on the target training mode; wherein, when the target training mode is a supervised learning mode, the target loss value is determined based on the label of the training sample; when the target training mode is a reinforcement learning mode, the target loss value is determined based on the reward score.

[0048] Specifically, the target loss value is the specific numerical value used for backpropagation to update the model parameters. When the target training mode is supervised learning, the target loss value mainly consists of the supervised loss. Specifically, the supervised loss is determined by comparing the model-generated response with preset labels (i.e., standard reference answers) in the training samples. Optionally, the maximum likelihood estimation (MLE) method can be used to calculate the cross-entropy loss.

[0049] When the target training mode is reinforcement learning, the target loss is primarily composed of the reinforcement learning loss. Specifically, the reinforcement learning loss is calculated based on the reward score obtained from the response results generated by the model. This reward score can be calculated using a policy gradient algorithm, representing the advantage of the current response's reward score relative to the average baseline.

[0050] Furthermore, to describe the entire joint training process within a unified mathematical framework and achieve a smooth fusion of supervised and reinforcement learning signals, a joint optimization objective function can be constructed. Although in specific implementation logic, a dominant mode may be chosen for a single sample, from an expectation perspective, the target loss value can be characterized as a weighted combination of the two.

[0051] As an optional embodiment, the formula for calculating the joint optimization objective function is as follows: ; in, This represents the joint optimization objective function. The parameters representing the agent model, This indicates the training dataset Samples in Seeking expectations, This represents the weights corresponding to the supervised learning model, i.e., fine-tuning the weight parameters. This represents the weights corresponding to the reinforcement learning mode, i.e., the reinforcement weight parameters. Indicates the sample The adaptive gating weights are determined by the sample mastery. This indicates that there is a monitoring loss in the foundation. This represents the basic reinforcement loss. Using this formula, the model can dynamically adjust its focus on the SFT and RL losses based on the level of understanding of each sample.

[0052] Step 140: Update the parameters of the agent model using the target loss value to obtain the target agent.

[0053] Specifically, updating refers to backpropagating the gradient based on the calculated target loss value and adjusting the weights and biases of the neural network in the agent model. By repeatedly performing steps 110 to 140 above, the model undergoes iterative training on massive training samples. For each sample, the model can automatically select the training mode most suitable for its current mastery level. As training progresses, the model's mastery of most samples gradually improves, and the training mode naturally transitions from primarily supervised learning to primarily reinforcement learning.

[0054] Finally, training ends when the preset termination conditions are met, such as reaching the maximum number of iterations or the loss value converging. The resulting agent model with updated parameters is the target agent. This target agent possesses the ability to accurately understand complex user intentions, quickly plan task paths, and accurately invoke tools in financial intelligent customer service scenarios, realizing a complete construction process from basic capability building to autonomous performance optimization.

[0055] As an optional embodiment, the corresponding gradient update formula is as follows: ; in, This represents the total gradient of the joint optimization objective function. Indicates the current training batch Samples in Perform summation. There is a corresponding supervisory signal. This represents the dynamic weights of the supervised learning model. Indicates that in a given sample Below, the model generates standard label sequences. The logarithmic probability gradient, this part of the gradient guides the model to mimic the standard answer. Corresponding reinforcement learning signals, This represents the dynamic weights of the reinforcement learning pattern. Indicates to The average of the sampling results is calculated. Indicates the value of the dominant function. Indicates that in a given sample The model generates a sampling sequence. The log probability gradient guides the model to increase the probability of generating high-dominance sequences and decrease the probability of generating low-dominance sequences.

[0056] By performing the gradient updates described above, the agent model is continuously optimized in the parameter space, ultimately resulting in a target agent with high accuracy, low latency, and strong generalization ability.

[0057] The agent construction method provided in this embodiment calculates the sample mastery of the agent model in real time for training samples, and adaptively switches between supervised learning and reinforcement learning modes accordingly. Then, based on the determined target training mode, it calculates the target loss value to update the model parameters. Because it can tailor instruction to the model's mastery of different samples—using labels for supervised correction of difficult samples that are not yet mastered, and using reward scoring for reinforcement exploration of easy samples that are already mastered—it effectively solves the training imbalance problem caused by the long-tail distribution of data in financial scenarios, avoids the process fragmentation and performance bottlenecks caused by simple phased training, and ultimately constructs a target agent with both high accuracy and strong robustness.

[0058] It should be noted that the execution entity in this embodiment can be any electronic device capable of training and building an intelligent agent model, such as a server cluster, workstation, high-performance personal computer, cloud computing node, etc. In this embodiment, the intelligent agent model is typically built based on a deep learning architecture and can be applied to fields such as natural language processing, multimodal interaction, and automated task execution. The intelligent agent model can be, for example, a large language model or a large multimodal model based on the Transformer architecture.

[0059] Taking the application of an intelligent agent model to intelligent financial customer service as an example, the training samples mentioned above can be natural language commands from users regarding tasks such as account inquiries, financial advice, and business processing. Based on the method of the above embodiment, the training mode is adaptively determined and parameters are updated. The model performs intent understanding and task planning on the input natural language commands. Based on the inference results, the model generates corresponding response content or API call instructions (such as JSON format transfer requests), thereby achieving accurate business responses.

[0060] Taking the application of an intelligent agent model to medical assisted diagnosis as an example, the training samples mentioned above can be patient descriptions of their condition, examination reports, and other information. Based on the method described in the above embodiment, the training mode is adaptively determined and parameters are updated. The input patient information is analyzed and diagnostic reasoned. Based on the reasoning results, the model generates a patient analysis report or treatment suggestions, such as recommended examination items and medication recommendations, thereby assisting doctors in improving diagnostic efficiency.

[0061] Taking the application of an intelligent agent model to industrial fault inspection as an example, the training samples mentioned above can be equipment operation logs, sensor data, or on-site inspection records. Based on the method of the above embodiments, the training mode is adaptively determined and parameters are updated. Anomaly detection and fault location are performed on the input equipment status data. The model generates maintenance work orders or scheduling instructions based on the inference results, such as triggering automatic maintenance programs or dispatching maintenance personnel, thereby ensuring the safe and stable operation of industrial production.

[0062] When implementing step S110 above, considering that a single inference generation result may be affected by the randomness of the model, it is difficult to comprehensively and objectively reflect the model's true mastery level of the training sample. For example, the model may guess the answer correctly by chance in a generation, or it may generate a low-quality response due to accidental factors. In order to reduce this evaluation variance and improve the robustness and accuracy of the sample mastery calculation, this embodiment adopts the idea of ​​Monte Carlo sampling, estimating the model's competence probability through multiple samplings.

[0063] Specifically, for the training samples, the agent model generates response results, and based on the reward score for the response results, the agent model's mastery of the training samples is calculated, including: The control agent model performs multiple inferences to generate multiple response results based on the training samples, and determines the reward score for each response result; The proportion of responses with reward scores exceeding a preset score threshold is statistically analyzed, and this proportion is determined as the agent model's mastery of the training samples.

[0064] Here, multiple inference generation refers to having the agent model repeat the inference process K times for the same training sample while keeping the input unchanged.

[0065] As an alternative implementation, randomness can be introduced by setting the model's sampling parameters, thus ensuring that the generated response results may differ each time. These K generated results constitute multiple sets of response results. .

[0066] For example, for the same request to "query bill", the model might generate a response with detailed steps on the first attempt. The second response may have generated a slightly simplified version of the steps. Subsequently, for each generated response result... Calculate the corresponding reward score. .

[0067] After obtaining the reward scores for each response, in order to statistically measure the model's competence with the sample, it is necessary to perform aggregate analysis on these scores. Specifically, the proportion of responses with reward scores exceeding a preset score threshold is calculated, and this proportion is determined as the agent model's mastery of the training samples.

[0068] Here, the preset scoring threshold is a baseline score used to determine whether a generated response is "qualified" or "successful." If the reward score of a generated response is greater than this threshold, the generation is considered to have passed the quality check. The ratio refers to the proportion of responses that pass the quality check out of the total number of generation attempts (K).

[0069] Alternatively, the sample mastery level can be calculated using the following formula: ; in, Indicates the agent model for the first training samples The degree of sample knowledge Indicates the number of samples. Indicates the first The response result generated by the second sampling Indicates that for the first The reward score for the response results generated by the next sampling. This indicates the preset scoring threshold. express When the condition is met, the value is 1; otherwise, the value is 0.

[0070] For example, if 8 out of 10 results generated by the model for a sample exceed the threshold, the sample mastery is 0.8, indicating that the model has a good grasp of that sample; conversely, if only 1 exceeds the threshold, the mastery is 0.1, indicating that the model has a poor grasp of that sample. This statistically based evaluation method can more realistically reflect the model's capability boundaries at the current training stage.

[0071] The method provided in this embodiment achieves an unbiased estimation of the model's mastery of the samples by controlling the proportion of high-scoring response results generated by the intelligent agent model through multiple inference operations on the training samples. Because it employs a multiple sampling and statistical approach, it effectively smooths out the random noise generated by the model, avoids evaluation bias caused by single-generation results, and makes the adaptive selection of the training mode more accurate and reliable.

[0072] Considering that intelligent customer service scenarios in the financial sector require not only accurate answers but also rapid responses, most related technologies focus only on the accuracy of responses, neglecting the problem of redundant information in the model's reasoning process. That is, models often tend to generate lengthy thought processes to ensure logical correctness, but this directly leads to increased reasoning latency and a reduced user experience. To guide the model to shorten the reasoning path as much as possible while ensuring accuracy, this embodiment of the invention constructs a composite reward function based on information entropy density.

[0073] Specifically, the reward score for each response result is determined, including: Obtain the format correctness score, semantic accuracy score, and thought chain density penalty for each response result; The reward score for the corresponding response result is obtained by weighting and summing the scores for format correctness, semantic accuracy, and thought chain density. Among them, the thought chain density penalty term is used to apply a negative penalty to the corresponding reward score when the length of the corresponding response result exceeds a preset length threshold.

[0074] Here, the format correctness score measures whether the response conforms to predefined structured specifications. For example, when involving tool calls, the model output needs to conform to a strict JSON schema format for subsequent system parsing. As an optional implementation, this score can be calculated through regular expression matching or JSON parser validation. A positive reward is given for successful parsing; a negative penalty is given for failed parsing.

[0075] Semantic accuracy scoring measures whether the response accurately answers the user's question and whether the logic is coherent. As an alternative implementation, a pre-trained reward model can be used to score the response, or it can be obtained by calculating the semantic similarity between the response and the standard answer.

[0076] The thought chain density penalty term quantifies the redundancy of thought chains in the response result and converts it into a negative reward value. As an optional implementation, this penalty term is negatively correlated with the generation length of the response result. That is, given the same task completion, the longer the generation length, the greater the penalty, and thus the lower the reward value.

[0077] After obtaining the above three indicators, in order to obtain a comprehensive evaluation result, they need to be integrated. That is, the format correctness score, semantic accuracy score, and thought chain density penalty item are weighted and summed to obtain the reward score of the corresponding response result.

[0078] Here, weighted summation refers to assigning different weight coefficients to each indicator based on its importance, and then adding them together. As an optional implementation, reward scoring... The calculation formula is as follows: ; in, This represents the input of the training samples. This represents the response result generated by the model; The score indicates the correctness of the format. Indicates semantic accuracy score, This indicates a penalty term for the density of the thought chain; , , These represent the weight coefficients of the corresponding items.

[0079] It's important to note that the thought chain density penalty term is used to impose a negative penalty on the corresponding reward score when the length of the corresponding response exceeds a preset length threshold. This means that if the thought chain generated by the model is too long, containing a large amount of low-entropy nonsense, This will become a large negative value, thus significantly lowering the overall reward score. This mechanism forces the agent model to find the shortest reasoning path during reinforcement learning, thus mathematically guaranteeing low latency.

[0080] This embodiment determines the reward score by comprehensively considering format correctness, semantic accuracy, and thought chain density penalty terms, thus achieving a multi-dimensional evaluation of the response results. By introducing a length-related thought chain density penalty term, it effectively suppresses the generation of lengthy thought chains, guiding the model to learn semantic compression while maintaining accuracy, ultimately improving the inference efficiency and response speed of the agent model.

[0081] In obtaining the thought chain density penalty term, it was considered that if only a linear length penalty was used, i.e., deducting a fixed number of points for each additional word, the model might sacrifice necessary reasoning logic in pursuit of extremely short lengths, leading to a decrease in accuracy. To find the optimal balance between preserving reasoning logic and maximizing the compression ratio, this embodiment designed a non-linear penalty mechanism. Slightly excessive lengths are penalized with lower penalties to preserve expressive flexibility, while significantly excessive lengths are penalized with exponential penalties to force semantic compression.

[0082] Specifically, the steps for determining the thought chain density penalty term include: Determine the actual generated length of the thought chain portion in the response result; Calculate the length difference between the actual generated length and the preset length threshold; The density penalty term of the thought chain is calculated based on the length difference.

[0083] Specifically, the "thought chain" section refers to the text fragment in the response result used to demonstrate the reasoning process, typically located before the final conclusion. The actual generated length refers to the number of tokens contained in this text fragment. As an optional embodiment, the response result can be segmented using a tokenizer, and the actual generated length can be obtained by counting the number of tokens in the thought chain section.

[0084] After obtaining the actual generated length, in order to assess whether it exceeds the standard and to what extent, it is necessary to compare it with the ideal standard, that is, to calculate the length difference between the actual generated length and the preset length threshold.

[0085] Here, the preset length threshold refers to the upper limit of the micro-thought chain length under ideal conditions. For example, in financial intent classification tasks, it is usually set to around 30 tokens. The length difference is the result of subtracting the preset length threshold from the actual generated length. The smaller the length difference, the closer the actual generated length is to or less than the preset length threshold. In this case, the simplicity of the thought chain meets expectations, and no excessive penalty is needed. Conversely, the larger the length difference, the more the actual generated length significantly exceeds the preset length threshold. In this case, the thought chain contains a lot of redundant content, and a heavier penalty needs to be imposed.

[0086] After obtaining the length difference, in order to achieve the design intent of light penalties for minor overruns and heavy penalties for serious overruns, it is necessary to calculate the thought chain density penalty term based on the length difference.

[0087] As an optional implementation, the thought chain density penalty term It can be calculated based on the following formula: ; in, Indicates the penalty coefficient. Indicates the actual generated length. This indicates a preset length threshold. For a linear rectified function, when The penalty term is proportional to the square of the proportion by which the actual generated length exceeds the preset length threshold. When the actual generated length is slightly greater than the preset length threshold (e.g., the actual generated length is 32 tokens and the preset length threshold is 30 tokens), the penalty value is small, allowing the model to retain some expressive flexibility. However, when the actual generated length is significantly greater than the preset length threshold, the penalty value increases exponentially, thus strongly forcing the model to perform semantic compression.

[0088] This embodiment calculates the difference between the actual generated length and a preset length threshold, and then uses the square of this difference to calculate a thought chain density penalty term, thus achieving a non-linear constraint on the inference length. Because it employs a squared penalty instead of a non-linear penalty, it tolerates slight overlength while severely punishing serious redundancy, effectively guiding the model to automatically learn concise reasoning abilities during training, balancing logical integrity and response latency.

[0089] Considering that financial data often exhibits a significant long-tail distribution, meaning it contains a large number of easily learned common intentions as well as a large number of complex intentions that are difficult to grasp, applying a one-size-fits-all training strategy to all samples can easily lead to overfitting on simple samples and underfitting on complex samples. To achieve personalized learning, i.e., using the most suitable training strategy for samples with different levels of mastery, this invention proposes an adaptive switching mechanism based on a mastery threshold.

[0090] Specifically, based on the level of mastery of the samples, a target training mode for the training samples is determined from supervised learning modes and reinforcement learning modes, including: When the sample mastery is less than or equal to the mastery threshold, the supervised learning mode will be used as the target training mode. When the sample mastery is greater than the mastery threshold, the reinforcement learning mode is used as the target training mode.

[0091] Specifically, when the calculated sample mastery is less than or equal to the mastery threshold, it indicates that the model's current ability to process the training sample is weak, and most of the generated responses fail the quality test. At this point, the model is in a state of ignorance or confusion. If the model is forced to explore on its own through reinforcement learning, the excessively large exploration space and lack of effective guidance may lead to low training efficiency or even model collapse.

[0092] In this context, supervised learning is chosen as the target training mode. By directly utilizing the standard labels in the training samples to calculate the loss and update the parameters, the fully supervised signal provides powerful guidance and correction, helping the model quickly establish basic cognitive features and instruction-following capabilities for the sample.

[0093] When the calculated mastery of a sample exceeds the mastery threshold, it indicates that the model has achieved a high level of competence with that training sample, and the generated responses largely meet the requirements. At this point, the model is in a state of proficiency or mastery. If supervised learning continues without further supervision, the model's performance improvement will become limited and it will be difficult to exceed the quality ceiling of the training data itself.

[0094] In this context, reinforcement learning is chosen as the target training mode. This encourages the model to build upon its existing capabilities by exploring policies to generate more diverse inference paths, such as more concise expressions and better tool call sequences, and further optimize the policy parameter space based on the obtained reward scores. This enables the model to self-reinforce and optimize performance on high-confidence samples, thereby significantly improving the model's generalization ability and inference efficiency.

[0095] As an alternative implementation, this switching logic can achieve a smooth transition using adaptive gating weights. For example, the adaptive gating weights can be defined as a Sigmoid function mapping of sample mastery, mapping the sample mastery to a gating space of [0, 1]. To avoid training oscillations, a temperature parameter and a mastery threshold are introduced: ; in, Indicates sample Adaptive gating weights, Indicates the sample The degree of sample mastery Indicates the threshold of mastery. This represents the temperature parameter. When... hour, 0, the model is in a state of ignorance. When hour, 1. The model is in a state of mastery.

[0096] The method provided in this embodiment achieves automatic switching of training modes by comparing the sample mastery with a preset mastery threshold. Because it can dynamically select correction or exploration strategies based on the difficulty of the samples, it effectively solves the problem of training with long-tailed data.

[0097] Given that the target training mode is supervised learning, and considering that large language models, if trained solely on raw natural language dialogue data, often learn verbose and divergent expression habits from human conversations, making it difficult to meet the stringent requirements of accurate intent understanding and efficient tool invocation for intelligent financial customer service, this embodiment constrains the structure of the training samples to ensure that the model establishes a standardized thinking pattern and concise expression habits from the beginning of training.

[0098] Specifically, when the target training mode is supervised learning, before determining the target loss value based on the target training mode, the following steps are also included: The input part of the training samples is constructed, which includes user intent classification labels, short thought chain guidance, and tool call instructions; The label part of the training samples is constructed, which includes the mind chain analysis of user intent and the response generation content; Among them, the content of the thought chain analysis is constrained to a preset first number of characters, the tool call command is constrained to a preset data exchange format, and the content of the response generation is constrained to a preset second number of characters.

[0099] Specifically, the input section is a sequence of prompts input to the agent model. User intent classification labels refer to predefined business intent categories; including them as part of the input explicitly informs the model of the focus of the current task. Short thought chain guidance is a prompt used to guide the model to briefly consider before generating a response. Tool invocation instructions are placed before the input section in this embodiment, as part of the context, simulating a scenario where the agent has already planned or executed a tool invocation while performing a task, requiring the model to generate subsequent analysis based on this.

[0100] As an optional implementation, the data structure of the input section can be designed as "[Intent Label]#[Short Thought Chain Guidance]#[Tool Invocation Command]". Here, "#" is a separator, and the tool invocation command is constrained to a preset data exchange format.

[0101] After constructing the input section, a corresponding label section needs to be built to guide the model in generating the required output. The label section includes the user's thought process analysis, tool invocation instructions, and the generated response. Here, the label section represents the ideal output that the model should generate.

[0102] The thought chain analysis is the process by which the model combines the tool call results and user intent from the input to perform logical reasoning. The response generation is the final natural language answer presented to the user.

[0103] To ensure the efficiency of model generation, this embodiment imposes strict length constraints on the above content. Specifically, the thought chain analysis content is limited to a preset first character range, which can be set to no more than 30 tokens. The model is required to use extremely concise language to "provide a brief explanation of the user's current query in conjunction with the background of multiple rounds, and then output the reason for the judgment," forcing the model to learn to instantly understand the user's intent, rather than providing lengthy analysis.

[0104] The generated response is constrained to a preset second character limit, which can be set to approximately 30-50 tokens. The paradigm is "tool return result + user's real needs + key points of the response summarized with effective knowledge," avoiding verbose and polite language generated by the model and directly addressing the user's core needs.

[0105] This embodiment establishes rules for the model during the supervised learning phase by constructing training samples with specific structures and strict length constraints. Due to the strong compression of thought chains and response content, and the formatting constraints on tool instructions, the model already possesses the potential to generate short, concise, and fast high-quality responses before subsequent reinforcement learning, reducing inference latency.

[0106] When the target intelligent agent is trained and deployed to a real-world online financial customer service environment, considering the extremely high real-time requirements of users, following traditional serial processing logic, such as waiting for the model to fully generate all thought chains and instructions before calling external tools, would cause unnecessary waiting for users. To further reduce response time and achieve second-level service, this embodiment proposes an asynchronous triggering mechanism based on streaming inference.

[0107] Specifically, after updating the parameters of the agent model using the target loss value to obtain the target agent, the process also includes: Receive natural language requests from user input; Utilize the target intelligent agent for streaming reasoning to generate thought chain content containing intent categories; When the end marker of the intent category for the content output of the thought chain is detected, an asynchronous external tool call request corresponding to the intent category is triggered.

[0108] Specifically, a user-input natural language request refers to the text sent by the user on the client regarding business processing or information inquiry. For example, a natural language request could be "I want to pay back the money I spent on furniture last month in 12 installments".

[0109] Considering that traditional generation methods require waiting for all content to be generated before subsequent processing can proceed, resulting in high response latency, this embodiment, upon receiving a natural language request, utilizes the target agent to perform streaming reasoning to generate a thought chain content containing intent categories. Here, streaming reasoning means that the target agent outputs each generated token immediately, rather than waiting for the entire sequence to be generated. The thought chain content is the internal reasoning process generated by the target agent based on the natural language request. According to the constraints of the aforementioned training phase, the first half of the thought chain content quickly determines and outputs the user's intent category. For example, the target agent begins to generate word by word "[Thinking] The user wants to apply for installment payments...#installment application#...".

[0110] If the end marker of the intent category in the thought chain content output is detected, it indicates that the target agent has clearly identified the user's intent category, and the subsequent generated content is likely to depend on or be related to the business data corresponding to that intent. Considering that subsequent external tool calls are usually time-consuming operations, it would waste valuable time if the model were to be fully generated before making the call. Therefore, this embodiment asynchronously triggers the external tool call request corresponding to the intent category.

[0111] The intent category end marker refers to a special symbol or structural feature used to indicate the completion of intent judgment. For example, in the example above, when the model outputs "#installment application", a clear intent label is detected. (This example is not very intuitive, as it does not clearly illustrate which is the end marker.) Once the end marker is detected, there is no need to wait for the model to continue generating subsequent parameter analysis or response content. Instead, the corresponding external tool call request is immediately triggered asynchronously in the background, such as preloading the installment rate calculation API or querying the user's installment eligibility API.

[0112] This embodiment triggers an external tool call request immediately upon detecting the end marker of the intent category, without waiting for the subsequent text content to be fully generated. This achieves parallelization of model inference and business processing (API call), effectively utilizing the time window for model text generation. For example, while the model is still generating the sentence "Inquiring about installment plans for you...", the API data in the background may have already been returned. By the time the model has finished generating and is ready to output the specific plan, the data is already ready, thus eliminating the user-perceived API call time consumption.

[0113] This embodiment achieves parallel processing of inference and execution by listening to the generation of intent categories during the streaming inference process and asynchronously triggering external tool calls in the first instance. Since time-consuming business interfaces can be started without waiting for the model to finish generating completely, end-to-end system latency is significantly reduced, providing users with a near-zero-wait interactive experience.

[0114] Considering that directly using an untuned general-purpose language model for joint training might lead to extremely low exploration efficiency in the early stages of training, or even garbled characters or format errors, since the model has not yet mastered the intent classification rules and tool call formats specific to the domain, reinforcement learning cannot obtain effective reward signals, thus causing difficulties in model convergence. To address this cold start problem and ensure that the model possesses basic instruction following and intent understanding capabilities before entering the joint training phase, this embodiment introduces a pre-training fine-tuning step.

[0115] Specifically, before generating response results using the agent model on the training samples, the following steps are also included: Obtain the initial large language model and the initial dataset, which contains instruction compliance data with intent classification labels and standard data format instructions; The initial large language model is fine-tuned in a supervised manner using the initial dataset to obtain the agent model.

[0116] Specifically, the initial large language model refers to a basic model pre-trained on a large-scale general corpus. It possesses general language understanding and generation capabilities but lacks specific knowledge for financial customer service scenarios. The initial dataset is a high-quality, small-scale dataset specifically prepared for the cold start phase.

[0117] Instruction-compliant data refers to data organized in an "instruction-input-output" format. In this embodiment, this data is specially constructed to include intent classification labels and standard data format instructions.

[0118] As an alternative implementation, the initial dataset can be generated and cleaned manually or using a more powerful closed-source model to ensure high data quality.

[0119] After preparing the model and data, the initial large language model is subjected to supervised fine-tuning using the initial dataset to obtain the agent model. Here, supervised fine-tuning refers to fine-tuning the initial large language model using the initial dataset with all parameters or parameter-efficient methods (such as LoRA).

[0120] Through this process, the model can quickly learn the intent classification system in financial scenarios and master how to generate tool call instructions that conform to the standard format.

[0121] The model after fine-tuning is the agent model. At this point, the agent model has the ability to follow instructions and output in a formatted manner, and can generate response results with basically correct format.

[0122] This embodiment introduces a cold start phase before joint training, using a dataset containing explicit intent labels and standard format instructions to perform supervised fine-tuning of the base model, achieving rapid alignment of model capabilities. Since the model has already mastered basic instruction compliance capabilities before entering the complex training process, it effectively avoids training divergence caused by blind exploration during the reinforcement learning phase, significantly improving the overall training stability and convergence speed.

[0123] Considering that the loss values ​​of supervised learning and reinforcement learning may differ in actual joint training, and the degree of emphasis placed on each may vary at different training stages, this embodiment of the invention introduces weighted control coefficients to flexibly adjust the influence of the two modes on model parameter updates and prevent one mode from dominating the gradient direction.

[0124] Specifically, determining the target loss value based on the target training pattern includes: When the target training mode is supervised learning mode, the basic supervised loss is calculated based on the standard labels of the training samples, and the product of the basic supervised loss and the fine-tuned weight parameters is determined as the target loss value. When the target training mode is reinforcement learning, the basic reinforcement loss is calculated based on the reward score, and the product of the basic reinforcement loss and the reinforcement weight parameter is determined as the target loss value.

[0125] Specifically, the basic supervised loss is typically calculated using the cross-entropy loss function, which calculates the difference between the predicted probability distribution generated by the model and the standard labels. The fine-tuning weights are preset hyperparameters used to control the proportion of supervised signals in the total loss. As an optional implementation, the target loss value... The calculation formula can be: , ; in, The parameters representing the agent model, This indicates fine-tuning of the weighting coefficients. This indicates that there is supervised loss on the baseline. In the early stages of training, the agent model's basic capabilities are relatively weak, so a larger loss can be set. The value is used to enhance the guiding role of supervised signals and accelerate model convergence.

[0126] When the target training mode is reinforcement learning, the basic reinforcement loss is calculated based on the reward score, and the product of the basic reinforcement loss and the reinforcement weight coefficients is determined as the target loss value. Here, the basic reinforcement loss is usually calculated based on the policy gradient algorithm, the core of which is to use the reward score to evaluate the merits of the current policy and adjust the model parameters accordingly to maximize the expected reward. The reinforcement weight coefficients are preset hyperparameters used to control the proportion of reinforcement learning signals in the total loss. As an optional embodiment, the formula for calculating the target loss value can be: in, This represents the reinforcement weighting coefficient. This represents the basic reinforcement loss. In the later stages of training, once the model has mastered the basic capabilities, this loss can be appropriately increased. This value is used to encourage the model to explore further and optimize itself.

[0127] The method provided in this embodiment achieves weighted control of the loss value under different training modes by introducing fine-tuning weight coefficients and strengthening weight coefficients. Since the weights can be flexibly adjusted according to the training stage and task requirements, the strength of imitation learning and self-exploration is balanced, ensuring that the model can continuously improve its performance ceiling while achieving stable convergence.

[0128] Considering that directly using absolute reward values ​​for optimization may lead to excessive training variance and make it difficult to distinguish between marginal improvements on a high-quality baseline and performance fluctuations on a low-quality baseline, for example, for a highly complex sample, even if the model's generation quality only reaches the baseline level, its absolute reward value may be higher than the reward value of a low-quality response generated on a simple sample. To more accurately assess the relative advantage of the current response result relative to the model's expected reward level, and thus more effectively guide gradient updates, this embodiment calculates the basic reinforcement loss based on reward scores, including: Obtain the reward baseline calculated for the training samples. The reward baseline represents the average reward level of the agent model in the current state. The difference between the reward score and the reward baseline is calculated to obtain the advantage function value; The basic reinforcement loss is calculated by multiplying the log probability gradient of the response generated by the agent model with the advantage function value.

[0129] Specifically, the reward baseline is used to measure the model's average performance on that training sample. As an optional implementation, the reward baseline can be obtained by calculating the average reward score of multiple response results generated from multiple samplings. The reward baseline... It can be calculated based on the following formula: ; in, Indicates the number of samples. For the first The reward score corresponding to the response result generated by the next sampling.

[0130] After obtaining the reward baseline, in order to evaluate whether the current response result is "better than average" or "worse than average," it needs to be compared with the baseline. That is, the difference between the reward score and the reward baseline is calculated to obtain the dominance function value. (The meanings of "better than average" and "worse than average" are unclear; please use the features provided in this embodiment to explain.) The dominance function value is used to quantify the degree of dominance of the response result relative to the average level. As an optional implementation, the dominance function value... The calculation formula is as follows: ; From the above formula, it can be seen that if A value of 0 indicates that the response is better than average and should be encouraged, meaning its generation probability should be increased; if... A value of 0 indicates that the response is worse than average and should be suppressed, i.e., its generation probability should be reduced.

[0131] After obtaining the advantage function value, in order to update the model parameters through gradient descent, it is necessary to calculate the final loss function by combining the model's prediction probability. In this embodiment, the basic reinforcement loss is calculated by multiplying the log probability gradient of the response result generated by the agent model with the advantage function value.

[0132] In this context, the logarithmic probability gradient of the agent model's generated response represents the direction in which changes in model parameters affect the generation probability. The basic reinforcement loss is typically defined as the direction that increases the probability of actions with a greater advantage. Within the framework of minimizing the loss, it is usually negative.

[0133] As an optional embodiment, the formula for calculating the basic reinforcement loss is as follows: ; As can be seen from the above formula, for a response with a positive dominance value, its log probability is increased; for a response with a negative dominance value, its log probability is decreased.

[0134] This embodiment achieves policy optimization based on relative advantage by introducing a reward baseline and calculating the advantage function value, making the reinforcement learning process more stable, avoiding training oscillations caused by fluctuations in the absolute size of the reward value, and effectively improving the stability of model convergence and final performance.

[0135] Figure 2 This is a schematic diagram of the target intelligent agent interaction link framework provided by the present invention, such as... Figure 2 As shown, the target intelligent agent constructed in this invention is a single-agent category, mainly integrating sub-capabilities such as user intent understanding, scenario knowledge retrieval, and tool invocation. Since the types of business intents in the financial field are large, potentially exceeding hundreds, this invention, from an optimization perspective, embeds user intent understanding capabilities into the target intelligent agent. This enables it to understand complex and ever-changing user intents and output the user intent type and tool invocation protocol.

[0136] Specifically, the overall process of this interaction link framework is as follows: First, the system receives natural language requests from users, which may contain complex business processing needs or information query intents. Then, it performs preliminary information retrieval by generating configuration nodes through retrieval enhancements and attempts to recall the intent.

[0137] Next, the natural language request enters the target agent. The target agent first performs user intent understanding, using its powerful reasoning ability to quickly determine the user's true intent. During this process, the target agent outputs its judgment result and decides whether the intent has been correctly identified.

[0138] If the user's intent is not matched, it indicates that the user's question may be a general knowledge inquiry and does not involve specific business operations. In this case, the process flows to the knowledge Q&A module, which generates a corresponding customer service response based on the knowledge base and directly provides feedback to the user, completing the interaction.

[0139] If the intent is matched, it indicates that the user needs to perform a specific task. At this point, the process enters the task processing branch. The target AI will generate the corresponding tool invocation command based on the intent and interact with the tool invocation layer.

[0140] Subsequently, the target agent, in conjunction with the results returned by the tool invocation layer, processes them again through the response generation module. To avoid excessively long outputs and address the scaling issue, the target agent performs limited generation within the scene, resulting in interactive responses that are accurate, comprehensive, concise, and free of redundancy.

[0141] Finally, the interactive response is output to the user. After the overall response is completed, the next interaction process can begin, awaiting the user's next request. This process achieves an end-to-end closed loop from user input to final response, which not only improves response speed but also ensures the professionalism and accuracy of the response.

[0142] The intelligent agent construction apparatus provided by the present invention is described below. The intelligent agent construction apparatus described below and the intelligent agent construction method described above can be referred to in correspondence.

[0143] Based on any of the above embodiments Figure 3 This is a schematic diagram of the intelligent agent construction device provided by the present invention, as shown below. Figure 3 As shown, the device includes: The calculation module 310 is used to generate response results using the agent model for the training samples, and to calculate the sample mastery of the agent model over the training samples based on the reward score for the response results. Selection module 320 is used to determine the target training mode for the training samples from supervised learning mode and reinforcement learning mode based on the sample mastery. The determination module 330 is used to determine the target loss value based on the target training mode; wherein, when the target training mode is a supervised learning mode, the target loss value is determined according to the label of the training sample; when the target training mode is a reinforcement learning mode, the target loss value is determined according to the reward score. The update module 340 is used to update the parameters of the agent model using the target loss value to obtain the target agent.

[0144] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the intelligent agent construction method.

[0145] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the agent construction method provided by the above methods.

[0147] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the agent construction methods provided by the above methods.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an intelligent agent, characterized in that, include: For the training samples, the agent model generates response results, and the sample mastery of the agent model over the training samples is calculated based on the reward score for the response results. Based on the mastery of the samples, a target training mode for the training samples is determined from supervised learning mode and reinforcement learning mode. The target loss value is determined based on the target training mode; wherein, when the target training mode is the supervised learning mode, the target loss value is determined according to the label of the training sample; when the target training mode is the reinforcement learning mode, the target loss value is determined according to the reward score. The parameters of the agent model are updated using the target loss value to obtain the target agent.

2. The intelligent agent construction method according to claim 1, characterized in that, The process of generating response results using an agent model for training samples, and calculating the agent model's sample mastery of the training samples based on the reward score for the response results, includes: The intelligent agent model is controlled to perform multiple inferences and generation operations on the training samples to obtain multiple response results, and the reward score for each response result is determined. The proportion of response results whose reward scores exceed a preset score threshold is statistically analyzed, and this proportion is determined as the sample mastery of the agent model over the training samples.

3. The intelligent agent construction method according to claim 2, characterized in that, The determination of reward scores for each response result includes: Obtain the format correctness score, semantic accuracy score, and thought chain density penalty for each response result; The format correctness score, the semantic accuracy score, and the thought chain density penalty term are weighted and summed to obtain the reward score for the corresponding response result; The thought chain density penalty term is used to apply a negative penalty to the corresponding reward score when the length of the corresponding response result exceeds a preset length threshold.

4. The method for constructing an intelligent agent according to claim 3, characterized in that, The steps for determining the thought chain density penalty term include: Determine the actual generated length of the thought chain portion in the response result; Calculate the length difference between the actual generated length and the preset length threshold; The thought chain density penalty term is calculated based on the length difference.

5. The method for constructing an intelligent agent according to any one of claims 1 to 4, characterized in that, The step of determining the target training mode for the training samples from supervised learning modes and reinforcement learning modes based on the sample mastery includes: If the sample mastery is less than or equal to the mastery threshold, the supervised learning mode is used as the target training mode. If the sample mastery is greater than the mastery threshold, the reinforcement learning mode is used as the target training mode.

6. The method for constructing an intelligent agent according to any one of claims 1 to 4, characterized in that, When the target training mode is the supervised learning mode, before determining the target loss value based on the target training mode, the method further includes: The input portion of the training sample is constructed, and the input portion includes user intent classification labels, short thought chain guidance, and tool call instructions; Construct the label part of the training sample, which includes the thought chain analysis content of the user's intent and the response generation content; The thought chain analysis content is constrained within a preset first character limit, the tool call command is constrained to a preset data exchange format, and the response generation content is constrained within a preset second character limit.

7. The method for constructing an intelligent agent according to any one of claims 1 to 4, characterized in that, After updating the parameters of the agent model using the target loss value to obtain the target agent, the process further includes: Receive natural language requests from user input; The target intelligent agent is used for streaming reasoning to generate thought chain content containing intent categories; When the end marker of the intent category of the thought chain content output is detected, an external tool call request corresponding to the intent category is asynchronously triggered.

8. The method for constructing an intelligent agent according to any one of claims 1 to 4, characterized in that, Before generating response results using the agent model on the training samples, the following steps are also included: Obtain an initial large language model and an initial dataset, wherein the initial dataset contains instruction compliance data with intent classification labels and standard data format instructions; The initial large language model is subjected to supervised fine-tuning using the initial dataset to obtain the agent model.

9. The method for constructing an intelligent agent according to any one of claims 1 to 4, characterized in that, Determining the target loss value based on the target training mode includes: When the target training mode is the supervised learning mode, the basic supervised loss is calculated based on the standard labels of the training samples, and the product of the basic supervised loss and the fine-tuning weight parameters is determined as the target loss value. When the target training mode is the reinforcement learning mode, the basic reinforcement loss is calculated based on the reward score, and the product of the basic reinforcement loss and the reinforcement weight parameter is determined as the target loss value.

10. The method for constructing an intelligent agent according to claim 9, characterized in that, The calculation of the basic enhancement loss based on the reward score includes: Obtain the reward baseline calculated for the training samples, where the reward baseline represents the average reward level of the agent model in the current state; The difference between the reward score and the reward baseline is calculated to obtain the advantage function value; The basic reinforcement loss is calculated by multiplying the log probability gradient of the response result generated by the agent model with the advantage function value.

11. An intelligent agent construction device, characterized in that, include: The calculation module is used to generate response results using the agent model for the training samples, and to calculate the sample mastery of the agent model for the training samples based on the reward score for the response results. The selection module is used to determine the target training mode for the training samples from supervised learning modes and reinforcement learning modes based on the sample mastery. A determination module is used to determine a target loss value based on the target training mode; wherein, when the target training mode is the supervised learning mode, the target loss value is determined according to the label of the training sample; when the target training mode is the reinforcement learning mode, the target loss value is determined according to the reward score. The update module is used to update the parameters of the agent model using the target loss value to obtain the target agent.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the agent construction method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the agent construction method as described in any one of claims 1 to 10.

Images