Intelligent agent optimization method, electronic equipment and computer program product

Abstract

The invention provides an intelligent agent optimization method, electronic equipment and a computer program product.The method comprises the steps that firstly, optimization auxiliary information is obtained, the optimization auxiliary information comprises one or more of evaluation information and log information, and the evaluation information comprises an evaluation mode used for evaluating an intelligent agent; the log information comprises records of abnormal responses of the agents in the running process, then optimization suggestions for the agents are generated through the analysis model and displayed, input of the analysis model comprises optimization auxiliary information and description information used for describing the agents, and finally the optimization suggestions are selected in response to the received selection result of the optimization suggestions. And controlling the optimization model to optimize the intelligent agent according to the selected optimization suggestion represented by the selection result.

Description

Technical Field

[0001] This disclosure specifically relates to methods for optimizing intelligent agents, electronic devices, readable storage media, and computer program products. Background Technology

[0002] Intelligent agents are widely used in various business activities, such as providing users with dialogue services, data analysis services, and environmental awareness services. When an intelligent agent deviates or malfunctions during the service process, it is necessary to locate and optimize the abnormal points of the agent.

[0003] Currently, anomaly localization is mainly performed manually by technical personnel, who then manually optimize the code to fix the agent's malfunctions. This optimization method is inefficient and requires a high level of background knowledge and understanding of the agent from the technical personnel, demanding a deep understanding of the agent's technical aspects. Summary of the Invention

[0004] This disclosure provides methods for optimizing intelligent agents, electronic devices, readable storage media, and computer program products.

[0005] The first aspect of this disclosure proposes a method for optimizing an intelligent agent, comprising: acquiring optimization assistance information, the optimization assistance information including one or more of evaluation information and log information, the evaluation information including an evaluation method for evaluating the intelligent agent, and the log information including records of abnormal responses of the intelligent agent during operation; generating and displaying optimization suggestions for the intelligent agent through an analysis model, the input of the analysis model including the optimization assistance information and descriptive information for describing the intelligent agent; and, in response to receiving a selection result for the optimization suggestion, controlling the optimization model to optimize the intelligent agent according to the selected optimization suggestion indicated by the selection result.

[0006] According to some embodiments of this disclosure, before generating and displaying optimization suggestions for the agent through the analysis model, the method further includes: if the obtained optimization assistance information includes the log information, then controlling the agent to run N times according to the content described in the log information to obtain N new log information, where N≥1; and if the records of the abnormal response included in the N new log information satisfy the stopping condition, then stopping the optimization process.

[0007] According to some embodiments of this disclosure, the log information includes multi-round interaction information, and the agent is configured to output according to the output content of the agent recorded in the log information in a specified interaction round, wherein the specified interaction round includes at least a portion of the interaction rounds in which the agent does not have an abnormal response.

[0008] According to some embodiments of this disclosure, if the abnormal responses recorded in the log information are distributed across multiple interaction rounds, then the action of running N times is performed for each interaction round with abnormal responses.

[0009] According to some embodiments of this disclosure, N>1, the stopping condition includes: the proportion of the number of log messages containing the abnormal response record in the N new log messages is not higher than a proportion threshold.

[0010] According to some embodiments of this disclosure, the descriptive information includes one or more of the agent's code and prompt words.

[0011] According to some embodiments of this disclosure, if the optimization assistance information includes the evaluation information, then the optimization suggestions generated by the analysis model include: optimization suggestions obtained by evaluating at least one of the agent's structural framework and prompt words based on the evaluation method.

[0012] According to some embodiments of this disclosure, the evaluation information includes multiple evaluation methods, which correspond to multiple evaluation dimensions, and some or all of the optimization suggestions generated by the analysis model correspond to the multiple evaluation dimensions.

[0013] According to some embodiments of this disclosure, if the optimization assistance information includes the log information, then the optimization suggestions generated by the analysis model include: optimization suggestions obtained by evaluating at least one of the agent's structural framework and prompt words based on semantic analysis of the log information.

[0014] According to some embodiments of this disclosure, if the optimization assistance information includes the log information, then after controlling the optimization model to optimize the agent, the method further includes: performing a regression test on the agent optimized by the optimization model using a first sample set, obtaining a first test result, and displaying it.

[0015] According to some embodiments of this disclosure, after obtaining and displaying the first test result, the method further includes: in response to receiving an indication message, determining whether to perform a new test; and if it is determined to perform a new test, performing a new regression test on the agent optimized by the optimization model using a second sample set and obtaining a second test result; otherwise, re-executing the step of generating and displaying optimization suggestions for the agent through the analysis model, wherein the second sample set contains more samples than the first sample set.

[0016] According to some embodiments of this disclosure, before generating and displaying optimization suggestions for the agent through an analysis model, the method further includes: in response to receiving first input information, determining at least a portion of optional optimization methods as optimization methods to be executed, the optional optimization methods including prompt word optimization and agent framework optimization; generating and displaying optimization suggestions for the agent through an analysis model includes: generating corresponding optimization suggestions for the agent according to the optimization methods to be executed through the analysis model, and displaying the generated corresponding optimization suggestions.

[0017] According to some embodiments of this disclosure, there are multiple optimizable agents and the multiple agents work together, and the optimization suggestions generated by the analysis model correspond to the multiple agents.

[0018] According to some embodiments of this disclosure, if the optimization method to be performed by the analysis model includes agent framework optimization, then the optimization suggestions on the structural framework of the agent generated by the analysis model are used to represent one or more of the following optimization contents: the way in which the number of agents is changed, and the way in which the execution logic of multiple agents working together is changed.

[0019] According to some embodiments of this disclosure, if the optimization method to be performed by the analysis model includes prompt word optimization, then the optimization suggestions about prompt words generated by the analysis model for different agents are used to represent one or more of the addition, deletion, and replacement of the prompt words.

[0020] According to some embodiments of this disclosure, the analysis model generates multiple optimization suggestions. After receiving the selection result of the latest displayed optimization suggestion, the next optimization suggestion is displayed until the selection result of all optimization suggestions is determined.

[0021] According to some embodiments of this disclosure, in response to receiving a selection result for the optimization suggestion, controlling the optimization model to optimize the agent according to the selected optimization suggestion represented by the selection result includes: in response to receiving a selection result for the optimization suggestion, generating and displaying content to be modified based on the selected optimization suggestion represented by the selection result; in response to receiving second input information, determining whether to perform new optimization; and if new optimization is performed, discarding the content to be modified, generating and displaying new optimization suggestions for the agent through the analysis model, otherwise controlling the optimization model to optimize the agent according to the content to be modified.

[0022] A second aspect of this disclosure provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform the method described in any of the above embodiments.

[0023] A third aspect of this disclosure provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the method described in any of the above embodiments.

[0024] The fourth aspect of this disclosure provides a computer program product comprising a computer program that, when executed by a processor, is used to implement the method described in any of the above embodiments. Attached Figure Description

[0025] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0026] Figure 1 The diagram illustrates an application scenario of the intelligent agent optimization method according to some embodiments of the present disclosure.

[0027] Figures 2-6 A schematic diagram of the overall process of an intelligent agent optimization method M100 according to some embodiments of the present disclosure is shown.

[0028] Figure 7 A flowchart illustrating some embodiments of this disclosure for optimizing an agent using evaluation criteria is shown.

[0029] Figure 8 A flowchart illustrating some embodiments of this disclosure for optimizing an agent through nonconforming cases is shown.

[0030] Figure 9 This is a schematic block diagram of the structure of an intelligent agent optimization device according to one embodiment of the present disclosure.

[0031] Figure 10 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure. Detailed Implementation

[0032] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0033] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0035] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0036] When intelligent agents are applied in actual business practices, there may be some aspects that need optimization. For example, the answers provided to users during interaction may be incorrect, or the computational logic during data analysis may be flawed, and so on.

[0037] Currently, the common method for optimizing intelligent agents is manual tuning or manual repair, which involves operations and maintenance personnel or technicians optimizing and modifying the code in the codebase. This optimization method has the following three problems.

[0038] 1. Optimization efficiency issues: Manual code optimization is inefficient, as it requires searching and modifying code in the code repository, which is time-consuming and labor-intensive.

[0039] 2. Technical barriers: It requires a high level of knowledge from operations and maintenance personnel. They need to understand the framework and business knowledge of the intelligent agent and other relevant background knowledge. For those who lack relevant background knowledge, the optimization process will be more complicated and difficult.

[0040] 3. Knowledge limitations: The reasoning model of the intelligent agent can demonstrate the ability to surpass human thinking in some scenarios. Manual optimization may fall into local optima due to the limitations of the personnel themselves, making it difficult to achieve the expected optimization results.

[0041] Therefore, this disclosure proposes an optimization method for intelligent agents.

[0042] Figure 1 A schematic diagram illustrating an application scenario of the intelligent agent optimization method according to some embodiments of this disclosure is shown. This application scenario may include a user terminal 10 and a server terminal 20.

[0043] User terminal 10 can communicate with server terminal 20 to send and receive data or commands. In this disclosure, both user terminal 10 and server terminal 20 include at least one processor and at least one memory.

[0044] For example, the user terminal 10 can be a terminal device held by a user, who can be the operation and maintenance personnel or related technical personnel of the intelligent agent. The server terminal 20 can be a service platform or system provided by the enterprise to the operation and maintenance personnel, or it can be a third-party service platform or system. The user terminal 10 can deploy the code library of the intelligent agent. The user terminal 10 interacts with the server terminal 20 to upload files of the intelligent agent and issue optimization instructions to the server terminal 20. The server terminal 20 can generate modification instructions through a Large Language Model (LLM) and modify the corresponding data and code in the user terminal 10, thereby optimizing the intelligent agent.

[0045] exist Figure 1 The shape and structure of the user terminal 10 and the server terminal 20 shown should not be construed as a limitation on the scope of protection of this disclosure. In this disclosure, "terminal device" can be different types of electronic devices, such as mobile phones, tablets, laptops, or desktop computers.

[0046] Figure 2 A schematic diagram of the overall flow of an intelligent agent optimization method M100 according to some embodiments of this disclosure is shown. For example... Figure 2 The method shown includes steps S110, S130, and S150. This method can be executed by an electronic device such as a computer.

[0047] S110, Obtain optimization assistance information, which includes one or more of evaluation information and log information. The evaluation information includes the evaluation method used to evaluate the agent, and the log information includes the record of abnormal responses of the agent during operation.

[0048] Optimization auxiliary information (or simply auxiliary information) is used to provide the model with the information needed for the optimization process when using the model to optimize the agent. The optimization auxiliary information provides the model with an optimization benchmark or specific case, and the model generates optimization suggestions based on the optimization benchmark or specific case.

[0049] The aforementioned auxiliary information may consist solely of evaluation information. Evaluation information provides a method for evaluating the agent, essentially offering evaluation criteria. These criteria may include algorithms with functions, allowing the model to obtain evaluation results for the agent through function operations, and subsequently generate corresponding optimization suggestions. Evaluation information may contain multiple dimensions of evaluation criteria; for example, evaluating the anthropomorphism of the agent's responses to the user based on tone, or assessing the strong relevance of the agent's responses to the user's question based on semantics, and so on. The specific types of evaluation criteria included in the evaluation criteria can be set according to the agent's function and purpose.

[0050] The aforementioned auxiliary information may also include only log information. Log information provides historical operational records of the agent, such as conversation records between the agent and a user over a period of time. Log information may contain records of the agent's abnormal responses. For example, in a multi-turn dialogue within a conversation record, if the agent's response in one turn is irrelevant to the user's question, insufficiently detailed, or contains other types of defects, then the agent's response in that turn is considered an abnormal response. Similarly, in a computation record, if the computational logic used in one computation is incorrect, the computational result is biased, or there are other types of defects, then the computation performed by the agent is considered an abnormal response. In other words, the specific cases contained in the log information are bad cases, which allows the model to analyze bad cases to infer areas for improvement and generate corresponding optimization suggestions.

[0051] Understandably, auxiliary information can also include both evaluation information and log information. For example, it can analyze non-compliance cases to identify areas for improvement and obtain evaluation results through evaluation criteria, thereby generating optimization suggestions.

[0052] For example, the optimization method M100 provided in this disclosure can be implemented through an optimization platform or system deployed on the user end or the backend server. The optimization platform or system is configured with an interactive interface. After opening the interface, the user (the agent's operations and maintenance personnel or technicians) can input the storage path of the required files. For example, at the beginning of the agent optimization process, the user inputs the storage path of the evaluation standard file containing evaluation information or the storage path of the case file containing unqualified cases. The case file can be a JSON file. The optimization platform or system obtains the evaluation standard file or badcase file through the received file storage path.

[0053] S130: The analysis model generates and displays optimization suggestions for the agent. The inputs to the analysis model include optimization assistance information and descriptive information for describing the agent.

[0054] The analysis model can employ a Large Language Model (LLM), such as a Generative Pre-trained Model (GPT). The descriptive information describes at least one of the agent's operational logic, working principles, functions, and uses. For example, the descriptive information may include content from technical documents such as the agent's design documents, functional specifications, and API (interface) documents, used to describe the agent's architecture, data structures, algorithms, business logic, inputs and outputs, and interface calls.

[0055] The descriptive information can be obtained in step S110. For example, the interactive interface described above can also provide a control for inputting descriptive information, allowing the user to input the storage address of the technical document. Through step S110, auxiliary information and descriptive information are obtained and input into the analysis model in step S130.

[0056] The cue words of the analysis model can be used to instruct the model to find ways to fix, improve, or optimize based on the input information and output them, thus enabling the analysis model to automatically analyze and generate optimization suggestions instead of manual intervention. The optimization suggestions output by the analysis model will be displayed in the interactive interface for the user to select. The optimization suggestions can be presented as reflective content based on the reasoning of the large model. For example, an optimization suggestion could be "When the task requires an output of 'none,' the lack of template specifications may lead to free interpretation and non-standard responses (such as 'not found yet'), affecting the downstream agent's parsing."

[0057] S150, in response to receiving the selection result of the optimization suggestion, controls the optimization model to optimize the agent according to the selected optimization suggestion represented by the selection result.

[0058] The optimization suggestions output by the analysis model can be displayed as text in the interactive interface. Each suggestion can correspond to an option control, such as a checkbox. After reviewing the suggestions, if the user agrees, they can select the corresponding checkbox. If they disagree, they do not need to select the checkbox. After reviewing all suggestions and making their selections, the user can click a button to indicate the end of the selection process. This triggers the optimization platform or system to obtain the user's selection results. The platform or system then generates a modification plan based on the selected suggestions. The optimization model then modifies the corresponding content of the agent according to the modification plan, such as changing the agent's prompts, functional modules, and operational logic. In this way, human experts can accept, provide feedback on, and correct candidate strategies during the optimization process, without requiring manual code modification or complex operations, thus improving optimization efficiency and accuracy.

[0059] The optimization model can be the same as the analysis model, meaning the same model is used to provide optimization suggestions and modify agent content. Alternatively, the optimization model can be different from the analysis model. For example, the optimization model can be a large language model combined with an agent framework (such as OpenHands) and configured with code modification capabilities. The optimization model uses the agent framework to autonomously parse multi-agent code files, performing pseudocode, key function abstraction, and logic analysis, and supports automated modification of agent prompts and functional code files.

[0060] The intelligent agent optimization method proposed in this disclosure utilizes a large model to replace manual problem analysis, code searching, and code modification, improving the optimization efficiency of the intelligent agent. Furthermore, it eliminates the need for operators to possess extensive background knowledge or intelligent agent framework knowledge, resulting in a lower technical threshold. Simultaneously, leveraging the reasoning capabilities of the large model avoids the limitations of manual tuning that sometimes hinder optimal optimization. This allows for rapid and intelligent modification and optimization of the intelligent agent. Moreover, an effective human-machine collaboration mechanism is constructed during the optimization process, fully integrating the prior knowledge of domain experts while ensuring consistency between the optimization direction and business needs, enabling the optimization effect to reach the expected level. The final system solution meets the requirements in terms of usability and operability, better realizing the practical application value and technological promotion potential of intelligent agents in complex business scenarios.

[0061] Figure 3 A schematic flowchart of the optimization method M100 for an intelligent agent according to other embodiments of this disclosure is shown. (See also...) Figure 3The agent optimization method M100 may further include steps S121 and S122. Steps S121 and S122 are executed before step S130.

[0062] S121, if the obtained optimization assistance information includes log information, then the control agent runs N times according to the content described in the log information to obtain N new log information, where N≥1.

[0063] S122, If the abnormal response records contained in N new log messages meet the stopping condition, then stop the optimization process.

[0064] When the optimization assistance information obtained in step S110 includes case information of unqualified cases, it indicates that the agent encountered a problem during a service provision process. If the problem is intermittent, no optimization is needed; if the problem is not intermittent, optimization is required. Steps S121 and S122 determine whether the problem is intermittent or not.

[0065] Step S121 is used to reproduce (reenact) the non-compliant cases. Following the interaction process in the non-compliant case, corresponding inputs are provided to the agent, and the agent's output in response to these inputs is received. A reproduction is completed after all inputs from the interacting parties (including the agent and the user) in the non-compliant case have been completed and all outputs from the agent have been obtained. One or more reproductions are required, and in each reproduction, it is determined whether there is an abnormal response of the same type as the abnormal response in the non-compliant case in the agent's output.

[0066] The stopping condition can be determined based on the number or proportion of successful reproductions. Assuming two reproductions were performed, and the agent's response in the 5th round of the session was abnormal in both reproductions, and the agent's response in the 5th round of the session was also abnormal in the non-compliant case, then both reproductions of the abnormal response were considered successful, the stopping condition was not met, and optimization is required. If the agent's response was not abnormal in either reproduction, then the reproduction failed, indicating that the abnormality in the non-compliant case was intermittent, the stopping condition was met, and no optimization is needed, i.e., subsequent steps S130 and S150 do not need to be executed.

[0067] Steps S121 and S122 can determine whether the abnormal response of the agent is sporadic or non-spontaneous, thus saving the need to optimize sporadic anomalies.

[0068] Records of abnormal responses included in the log information can be identified through their own identifiers. In other words, the above-mentioned erroneous or remedial responses can be automatically labeled by the system or manually labeled, and dialogues containing abnormal content can be identified from multi-turn conversations through these identifiers.

[0069] Log information can include multi-turn interaction information, such as multi-turn dialogue information between an agent and a user. The agent can be configured to output according to the agent's output recorded in the log information during specified interaction turns. These specified interaction turns can include at least a portion of the interaction turns in which the agent did not exhibit any abnormal responses.

[0070] For example, an invalid case might involve five rounds of dialogue. In the fifth round, the user asks three questions, but the agent's response in that round lacks an answer to one of those questions. During reproduction, the simulated user provides the agent with the same input as in the invalid case in each round. This could be rounds 1-4. The agent's responses in rounds 1-4 could be the same as the agent's output in the invalid case. In the fifth round, the agent needs to respond to the simulated user's questions based on its own capabilities, rather than simply using the output recorded in the invalid case. This allows us to see if the agent's response in the reproduction matches the invalid case. This approach enables a more realistic reproduction of the invalid case scenario, making the assessment of whether the invalid case is accidental more accurate.

[0071] If the abnormal responses recorded in the log information are distributed across multiple interaction rounds, then the action can be run N times for each interaction round with abnormal responses. For example, a failed case might involve 10 rounds of dialogue, where the agent's responses failed to satisfy the user in two rounds: rounds 4 and 8. In this case, these two rounds need to be reproduced separately. Specifically, the abnormal response in round 4 can be reproduced N times, where the agent's responses in rounds 1-3 and 5-10 can be the same as the responses in the corresponding rounds of the failed case. Then, the abnormal response in round 8 can be reproduced N times, where the agent's responses in rounds 1-7 and 9-10 can be the same as the responses in the corresponding rounds of the failed case.

[0072] The value of N can be N>1, meaning multiple reproductions will be performed. The stopping condition can include: the percentage of log messages containing abnormal response records among N new log messages does not exceed a certain threshold. For example, if 5 reproductions were performed, with only 1 successful reproduction, the reproduction rate is 20%. If the threshold is 30%, the reproduction rate is not higher than the threshold, the stopping condition is met, and the optimization process will stop, without executing steps S130 and S150. This allows for quick and easy determination of whether non-compliant cases are sporadic.

[0073] In step S130, the description information may specifically include one or more of the agent's code and prompt words. For example, the description information may include code in the agent's code file, as well as prompt words in the code file. The optimization method provided in this disclosure may primarily optimize the agent's code and prompt words.

[0074] If the optimization auxiliary information includes evaluation information, the optimization suggestions generated by the analysis model may include: optimization suggestions obtained by evaluating at least one of the agent's structural framework and prompt words based on the evaluation method.

[0075] Evaluation information (testing criteria) is primarily applicable to evaluating agents not yet in use, and mainly for general optimization following standard procedures. If the auxiliary information only includes evaluation information, reproduction is unnecessary since no cases exist. After inputting the evaluation information, agent code, and prompts into the analysis model, the model can evaluate the agent's structural framework and prompts according to the evaluation methods in the evaluation information. Optimization suggestions derived from evaluating the structural framework could include, for example, "The speed of storing and retrieving historical information needs to reach a certain value." Optimization suggestions derived from evaluating prompts could include, for example, "When current dialogue information is insufficient, there is a 'cannot be judged' rule for professional judgment, but the prompt does not clearly express this situation, which may lead to invalid conclusions mixed in the output."

[0076] Evaluation information can represent multiple evaluation methods. These multiple evaluation methods can correspond to multiple evaluation dimensions. Some or all of the optimization suggestions generated by the analysis model can correspond to these multiple evaluation dimensions. For example, the speed of storing and retrieving historical information can correspond to the information retrieval dimension. By setting functions for testing and calculation, the evaluation result under the information retrieval dimension can be determined to indicate whether the speed is fast or slow. Through user-defined evaluation functions (accuracy, latency, etc.), the evaluation system becomes flexible and scalable, facilitating multi-dimensional evaluation of intelligent agents.

[0077] If the optimization auxiliary information includes log information, the optimization suggestions generated by the analysis model include: optimization suggestions obtained by evaluating at least one of the agent's structural framework and prompt words based on semantic analysis of the log information.

[0078] Log information (non-compliant cases) is primarily applicable to agents or agent frameworks already in use, and mainly for targeted optimization based on confirmed abnormal responses from agents. After inputting the log information, agent code, and prompts into the analysis model, the model can analyze the semantics of the conversations within the non-compliant cases in the log information to understand the topic and specific content of the conversations, thereby evaluating the agent's structural framework and prompts. Optimization suggestions derived from evaluating the structural framework could include, for example, "The agent does not support image input and cannot perform multimodal interaction," etc. Optimization suggestions derived from evaluating prompts could include, for example, "When the user sends mixed content (such as 'Okay...'), the rules for detecting meaningless characters are unclear, which may lead to misjudgment as valid information," etc.

[0079] Figure 4 A schematic flowchart of the optimization method M100 for an intelligent agent according to other embodiments of this disclosure is shown. (See also...) Figure 4 The optimization method M100 may further include step S160. Step S160 is performed when the optimization auxiliary information includes log information, and is performed after step S150.

[0080] S160, Perform regression testing on the optimized agent of the optimized model using the first sample set, obtain the first test result and display it.

[0081] The first sample set can be a simple evaluation set containing a small number of samples (e.g., fewer than 10 samples) to automatically perform regression testing after optimization. Regression testing is used to verify whether optimizations to the agent (such as fixing defects, adding new prompts, optimizing performance, etc.) have introduced new errors or caused existing functions to malfunction.

[0082] A rapid evaluation is performed using the first sample set to generate the first test result, which is essentially a regression report, allowing for a quick assessment of whether the optimization has achieved its objectives. The regression testing process can involve inputting the questions from the sample set into the agent, conducting multiple rounds of dialogue, inputting the dialogue information into the analysis model, and comparing it with the dialogue information of the agent before optimization to obtain the test results. The dialogue information of the agent before optimization can include the aforementioned log information (bad cases), log information obtained from N repetitions, and dialogue information obtained from testing the agent on the sample set before optimization.

[0083] The regression test results can represent the trend of the agent's performance under function evaluation before and after optimization. If the trend is positive under multiple evaluation functions corresponding to different evaluation dimensions, it means that the optimized agent performs better than the unoptimized agent under these evaluation dimensions, indicating that the optimization is effective. If the trend is flat under these evaluation functions, it means that the optimized agent performs roughly the same as the unoptimized agent under these evaluation dimensions, indicating that the optimization is ineffective. If the trend is negative under these evaluation functions, it means that the optimized agent performs worse than the unoptimized agent under these evaluation dimensions, indicating that the optimization needs to be reversed and re-optimized.

[0084] Continue reading Figure 4 The optimization method M100 may also include steps S170 and S180. Steps S170 and S180 are used to perform a full regression test.

[0085] S170, in response to receiving the instruction information, determines whether to perform a new test.

[0086] S180: If a new test is determined, a new regression test is performed on the agent optimized by the model using the second sample set, and the second test result is obtained. Otherwise, the steps described above, which involve generating and displaying optimization suggestions for the agent through model analysis, are repeated, i.e., the process jumps to step S130. The second sample set contains more samples than the first sample set.

[0087] After performing a simplified regression test in step S160 and obtaining the first test result, if the first test result indicates a good optimization effect, the user can issue an instruction to trigger a full-scale test of the optimized agent in more comprehensive scenarios to view its performance. The full-scale test uses a second sample set belonging to the full-scale evaluation set, with a significantly larger sample size than the first sample set. The resulting second test results are more referential and representative, providing a more accurate overall assessment of the agent's performance. If the first test result indicates a poor optimization effect, the process can directly proceed to step S130 to restart generating optimization suggestions for the agent using the analysis model.

[0088] Figure 5 A schematic flowchart of the optimization method M100 for an intelligent agent according to other embodiments of this disclosure is shown. (See also...) Figure 4The optimization method M100 may also include step S123. Step S123 is executed before step S130. And if the auxiliary information includes log information, that is, if steps S121 and S122 exist, then step S123 is executed after step S122.

[0089] S123, in response to receiving the first input information, determine at least some of the optional optimization methods as optimization methods to be executed. The optional optimization methods include prompt word optimization and agent framework optimization.

[0090] Before starting the optimization phase, the optimization platform or system can display two optional optimization methods on the interactive interface: "Prompt Optimization" and "Architecture Optimization," corresponding to prompt word optimization and agent framework optimization, respectively. If the user selects "Prompt Optimization," the optimization phase will only optimize the prompt word. If the user selects "Architecture Optimization," the optimization phase will only optimize the agent's structural framework. Users can also select both optional optimization methods, in which case the optimization phase will optimize both the prompt word and the structural framework.

[0091] Accordingly, in step S130, optimization suggestions for the agent are generated and displayed through the analysis model. Specifically, this may include: generating corresponding optimization suggestions for the agent according to the optimization method to be executed through the analysis model, and displaying the generated corresponding optimization suggestions.

[0092] Regardless of whether the auxiliary information includes evaluation information or log information, the user can select the desired optimization method in step S123. Therefore, there are four specific optimization scenarios.

[0093] Scenario 1: If the auxiliary information only includes evaluation information and the user selects to optimize only the prompt words, then there is no need for a reproduction stage. Furthermore, the analysis model evaluates the agent's prompt words based on multiple evaluation methods, thereby obtaining optimization suggestions for the prompt words.

[0094] Scenario 2: If the auxiliary information only includes evaluation information and the user selects to optimize only the agent framework, then there is no need for a reproduction phase. Furthermore, the analysis model evaluates the agent structure framework based on multiple evaluation methods, thereby obtaining optimization suggestions for the structure framework.

[0095] Scenario 3: If the auxiliary information only includes log information and the user selects to optimize only the prompt words, then after completing the reproduction phase, the analysis model performs semantic analysis on the log information, evaluates the agent's prompt words based on the semantic analysis results, and obtains optimization suggestions for the prompt words.

[0096] Scenario 4: If the auxiliary information only includes log information and the user selects to optimize only the agent framework, then after completing the reproduction phase, the analysis model performs semantic analysis on the log information, evaluates the agent structure framework based on the semantic analysis results, and obtains optimization suggestions for the structure framework.

[0097] It is understandable that there are other optimization scenarios, such as when a user selects both prompt word optimization and frame optimization. These scenarios are combinations of the four scenarios mentioned above and will not be elaborated upon here.

[0098] There can be multiple optimizable agents, and these agents can work collaboratively. That is, the agent optimization method provided in this disclosure can optimize not only a single agent but also the agents within a multi-agent system (MAS). A multi-agent system is a distributed system composed of multiple autonomous agents, each capable of autonomously perceiving the environment, learning, and making decisions to achieve its own goals or collaboratively accomplish the overall goal.

[0099] Accordingly, the optimization suggestions generated by the analysis model can correspond to multiple agents. For example, a multi-agent system includes dialogue agents, emotion agents, attention agents, and reflection agents. The analysis model can generate optimization suggestions for each of these agents and display them sequentially on the interactive interface for the user to select from, according to the agent type. It is understood that the descriptive information provided to the analysis model in step S130 includes agent code files, which allow the analysis model to determine that the optimization targets are multiple agents in the multi-agent system.

[0100] Therefore, this disclosure enables a multi-dimensional automatic evolution mechanism, supporting autonomous evolution of prompts for each agent, interaction methods between agents, and functional definitions. It achieves automated optimization and intelligent human-machine collaboration, enabling efficient autonomous evolution of multiple agents. At the prompt optimization level, compared to independently optimizing prompt templates for a single agent, the optimization method provided in this disclosure establishes a global optimization mechanism for collaborative prompts among multiple agents. At the architecture optimization level, the deep integration of the optimization method provided in this disclosure with the inference model allows the synergistic effect of model capabilities and system architecture to be fully utilized.

[0101] If the optimization methods to be performed by the analysis model include agent framework optimization, then the optimization suggestions on the agent's structural framework generated by the analysis model can be used to represent one or more of the following optimization contents: the way to change the number of agents, the way to change the execution logic of multiple agents working together.

[0102] The optimization method to be executed is the one selected by the user from the available optimization methods. If the user selects "Architecture optimization" in step S123, then for a multi-agent system, the optimization suggestions provided by the analysis model may include: adding agents, deleting existing agents, or changing the functions of existing agents. Optimization suggestions may also include logic for inter-agent collaboration. For example, for a user's question, which was originally processed first by agent A1 and then by agent A2, the optimization suggestion could be that agent A2 processes the question first, and then agent A1 processes it.

[0103] If the optimization methods to be executed by the analysis model include prompt word optimization, the analysis model can generate optimization suggestions for prompt words separately for different agents. The optimization suggestions for prompt words generated by the analysis model for agents can be used to indicate one or more of the following: adding, deleting, or replacing prompt words. For example, a prompt word optimization suggestion might include: "The mandatory constraint requires the process to terminate and ask a question when the word appears, but the specific logic for the question is not defined, which may lead to the question content being disconnected from the current topic." If the user selects this optimization suggestion, the optimization model will add new content to the existing prompt words, thereby defining the specific logic for the question and avoiding topic disconnect.

[0104] The analysis model can generate multiple optimization suggestions. After receiving the selection result for the latest displayed optimization suggestion, the next suggestion can be shown, until the selection result for all optimization suggestions is determined. In other words, each optimization suggestion can be displayed sequentially, awaiting user selection. The user indicates selection or non-selection by inputting 'y' and 'n'. Only after the user inputs the character indicating selection or non-selection will the interface display the next optimization suggestion. This allows users to focus on each individual optimization suggestion and make a reasonable choice.

[0105] Figure 6 A schematic flowchart of the optimization method M100 for an intelligent agent according to other embodiments of this disclosure is shown. (See also...) Figure 6 Step S150 may specifically include steps S151, S152 and S153.

[0106] S151, in response to receiving the selection result of the optimization suggestion, generate the content to be modified and display it based on the selected optimization suggestion indicated by the selection result.

[0107] For prompt word optimization, the content to be modified can be the new prompt word text after modification according to the optimization suggestions. For framework optimization, the content to be modified can be some specific changes involving the code level. Step S151 is only used to provide users with specific optimization measures and does not actually implement the optimization.

[0108] S152, in response to receiving the second input information, determines whether to perform new optimization.

[0109] S153, if new optimization is to be performed, the content to be modified is discarded, and new optimization suggestions for the agent are generated and displayed through the analysis model; otherwise, the optimization model is controlled to optimize the agent according to the content to be modified.

[0110] After the specific optimization measures are displayed on the interactive interface, users can choose whether to actually implement the optimizations and modifications.

[0111] If a user is not satisfied with the optimization measures after reviewing them, they can input relevant information to instruct the optimization platform or system to perform new optimizations, which means starting an optimization iteration. In this case, the current optimization suggestions and content to be modified will be discarded, and the user interface will restart the optimization phase. The user will then select an optimization method, and the analysis model will provide new optimization suggestions.

[0112] If the user is satisfied with the optimization measures, they can instruct the optimization platform or system to proceed with the next steps by inputting the corresponding information. At this point, the optimization model will begin to modify the prompt words and agent code according to the content to be modified.

[0113] Steps S151-S153 provide the user with an opportunity to back out. The user can abandon this optimization and start a new round of optimization to avoid the performance and effectiveness of the agent deteriorating due to incorrect optimization.

[0114] Figure 7 A flowchart illustrating the optimization of an agent using evaluation criteria according to some embodiments of this disclosure is shown. See also... Figure 7 The optimization process mainly includes the initial stage and the optimization stage.

[0115] In the initial stage, users enter the storage path of the agent code file and the storage path of the evaluation standard file in the interactive interface.

[0116] Then, the optimization phase begins. The user chooses either prompt optimization or frame optimization. The analysis model then generates optimization suggestions for either the prompt or the frame based on the user's chosen optimization method and the evaluation criteria. These suggestions, along with the corresponding content to be modified, are displayed on the interactive interface. If the user deems the content to be modified unreasonable, they can choose to abandon the current optimization. The optimization system will then discard the generated optimization suggestions and content to be modified and begin a new optimization iteration, displaying the optimization method again on the interactive interface for the user to choose from. If the user deems the content to be modified reasonable, the optimization model is instructed to modify the prompt or frame for the agent. After the modification is completed, the optimization ends.

[0117] Figure 8 A flowchart illustrating the optimization of an agent using nonconforming cases in some embodiments of this disclosure is shown. See also... Figure 8 The optimization process mainly includes the initial stage, the reproduction stage, the optimization stage, and the regression stage.

[0118] In the initial stage, the user enters the storage path of the agent code file and the storage path of the badcase file in the interactive interface.

[0119] Then, the optimization phase begins. The user chooses either prompt optimization or frame optimization. The analysis model then generates optimization suggestions for either the prompt or the frame based on the conversation between the agent and the user in the bad case, according to the user's selected optimization method. These suggestions, along with the corresponding content to be modified, are displayed on the interactive interface. If the user deems the content to be modified unreasonable, they can choose to abandon the current optimization. The optimization system will then discard the generated optimization suggestions and content to be modified and begin a new optimization iteration, displaying the optimization method again on the interactive interface for the user to choose from. If the user deems the content to be modified reasonable, the optimization model is instructed to modify the agent's prompt or frame.

[0120] The process then moves to the regression phase. First, a simplified regression test is performed using a simplified evaluation set, and the results are displayed. If a positive trend is observed across multiple evaluation dimensions, and no new bad cases emerge, full stability verification is performed using the full evaluation set, at which point the optimization process ends. If a negative trend is observed across multiple evaluation dimensions, and new bad cases appear, the process returns to the beginning of the optimization phase, starting a new round of optimization to form a complete evolutionary loop of "analysis-optimization-verification."

[0121] Based on any of the above embodiments, this disclosure also provides an optimization device for intelligent agents. Figure 9 This is a schematic block diagram of the structure of an optimization device for an intelligent agent according to one embodiment of this disclosure. Figure 9 As shown, the optimization device includes: an auxiliary information acquisition module 110, an optimization suggestion generation module 120 (i.e., the analysis model described above), and an agent optimization module 130 (i.e., the optimization model described above).

[0122] The auxiliary information acquisition module 110 is used to acquire optimization auxiliary information, which includes one or more of evaluation information and log information. The evaluation information includes the evaluation method used to evaluate the agent, and the log information includes the record of abnormal responses of the agent during operation.

[0123] The optimization suggestion generation module 120 is used to generate and display optimization suggestions for the agent through the analysis model. The input of the analysis model includes optimization assistance information and descriptive information for describing the agent.

[0124] The agent optimization module 130 is used to control the optimization model to optimize the agent according to the selected optimization suggestion as indicated by the received optimization suggestion in response to the selection result.

[0125] The optimization device for the aforementioned intelligent agent can be in the form of computer software, and each module of the optimization device can be implemented through computer software modules. The specific implementation process of the functions and roles of each module in the aforementioned device is detailed in the corresponding steps of the above method, and will not be repeated here.

[0126] The execution subject of the intelligent agent optimization method in the specific embodiments of this disclosure can be an electronic device such as a computer.

[0127] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the intelligent agent optimization method of any of the embodiments described above.

[0128] Figure 10 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0129] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 is also connected to a screen 1500 for displaying an interactive interface.

[0130] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0131] The processor 1200 can be a central processing unit (CPU). The processor 1200 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0132] The memory 1300 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions of the computer program in the embodiments of this disclosure. The processor 1200 implements the agent optimization method by running the non-transitory software programs, instructions, and modules stored in the memory 1300.

[0133] The memory 1300 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function. The data storage area may store data created by the processor 1200, such as evaluation reports, optimization suggestions, regression reports, etc. Furthermore, the memory 1300 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1300 may optionally include memory remotely located relative to the processor 1200, and these remote memories may be connected to the processor 1200 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0134] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0135] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this disclosure are performed wholly or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable device.

[0136] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0137] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0141] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0142] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0143] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for optimizing an intelligent agent, characterized in that, include: Obtain optimization assistance information, which includes one or more of evaluation information and log information. The evaluation information includes an evaluation method for evaluating the agent, and the log information includes records of abnormal responses of the agent during operation. The analysis model generates and displays optimization suggestions for the agent. The input of the analysis model includes the optimization assistance information and descriptive information for describing the agent. as well as In response to receiving the selection result of the optimization suggestion, the optimization model is controlled to optimize the agent according to the selected optimization suggestion represented by the selection result.

2. The method for optimizing an intelligent agent according to claim 1, characterized in that, Before generating and displaying optimization suggestions for the agent through the analysis model, the method further includes: If the obtained optimization assistance information includes the log information, then control the agent to run N times according to the content described in the log information, obtaining N new log information, where N≥1; and If the abnormal response records contained in the N new log messages meet the stopping conditions, then the optimization process is stopped.

3. The method for optimizing an intelligent agent according to claim 1, characterized in that, If the optimization assistance information includes the evaluation information, then the optimization suggestions generated by the analysis model include: optimization suggestions obtained by evaluating at least one of the agent's structural framework and prompt words based on the evaluation method.

4. The method for optimizing an intelligent agent according to claim 1, characterized in that, If the optimization assistance information includes the log information, then the optimization suggestions generated by the analysis model include: optimization suggestions obtained by evaluating at least one of the agent's structural framework and prompt words based on semantic analysis of the log information.

5. The method for optimizing an intelligent agent according to claim 1 or 4, characterized in that, If the optimization assistance information includes the log information, then after controlling the optimization model to optimize the agent, the method further includes: The agent optimized by the optimization model is subjected to regression testing using the first sample set to obtain and display the first test results.

6. The method for optimizing an intelligent agent according to claim 1, characterized in that, Before generating and displaying optimization suggestions for the agent through the analysis model, the method further includes: in response to receiving first input information, determining at least a portion of the optional optimization methods as optimization methods to be executed, the optional optimization methods including prompt word optimization and agent framework optimization; The step of generating and displaying optimization suggestions for the agent through the analysis model includes: generating corresponding optimization suggestions for the agent according to the optimization method to be executed through the analysis model, and displaying the generated corresponding optimization suggestions.

7. The method for optimizing an intelligent agent according to claim 1 or 6, characterized in that, There are multiple optimizable agents, and these agents work collaboratively. The optimization suggestions generated by the analysis model correspond to these multiple agents.

8. The method for optimizing an intelligent agent according to claim 1, characterized in that, In response to receiving a selection result for the optimization suggestion, the optimization model is controlled to optimize the agent according to the selected optimization suggestion represented by the selection result, including: In response to receiving the selection result of the optimization suggestion, based on the selected optimization suggestion represented by the selection result, the content to be modified is generated and displayed; In response to receiving the second input information, determine whether to perform a new optimization; and If new optimizations are to be performed, the content to be modified is discarded, and new optimization suggestions for the agent are generated and displayed through the analysis model; otherwise, the optimization model is controlled to optimize the agent according to the content to be modified.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 8.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, is used to implement the method of any one of claims 1 to 8.

Images