Intelligent agent modeling method and device, electronic equipment and storage medium

Abstract

The invention discloses an intelligent agent modeling method and device, electronic equipment and a storage medium, and relates to the technical field of intelligent agents and other artificial intelligence. According to the specific implementation scheme, an input request is obtained; obtaining a plurality of tool calling tracks generated by the intelligent agent based on the input request, wherein each tool calling track is used for identifying track information of at least two tools called by the intelligent agent in the process of responding to the input request; for each tool calling track, obtaining a target response result corresponding to the intelligent agent responding to the input request based on the corresponding tool calling track and an intermediate response result of calling an internal tool in the response process; and performing multi-target modeling on the intelligent agent based on the plurality of tool calling tracks, the target response result of each tool calling track and the intermediate response result of calling the internal tool.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as intelligent agents, and particularly to a modeling method, apparatus, electronic device and storage medium for intelligent agents. Background Technology

[0002] An intelligent agent is a core concept in the field of artificial intelligence, referring to a system capable of perceiving its environment, making decisions, and taking actions to achieve specific goals. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactivity. An intelligent agent perceives changes in the environment, such as data input, makes judgments and decisions based on its learned knowledge and algorithms, and then executes actions to influence the environment or achieve predetermined goals.

[0003] The brain of an intelligent agent can be implemented based on a Large Language Model (LLM). Intelligent agents typically require specific modeling before they can be deployed. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for modeling intelligent agents.

[0005] According to one aspect of this disclosure, a method for modeling an intelligent agent is provided, comprising:

[0006] Get input request;

[0007] Obtain several tool call trajectories generated by the agent based on the input request, each of the tool call trajectories being used to identify the trajectory information of at least two tools called by the agent in response to the input request;

[0008] For each tool call trajectory, obtain the target response result of the agent responding to the input request based on the corresponding tool call trajectory, as well as the intermediate response result of calling internal tools during the response process;

[0009] Based on the aforementioned tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, the agent is modeled in multiple objectives.

[0010] According to another aspect of this disclosure, a modeling apparatus for an intelligent agent is provided, comprising:

[0011] The input acquisition module is used to acquire input requests;

[0012] The trajectory acquisition module is used to acquire several tool call trajectories generated by the agent based on the input request. Each tool call trajectory is used to identify the trajectory information of at least two tools called by the agent in response to the input request.

[0013] The response acquisition module is used to acquire, for each tool call trajectory, the target response result of the agent responding to the input request based on the corresponding tool call trajectory, as well as the intermediate response result of calling internal tools during the response process;

[0014] The modeling module is used to perform multi-objective modeling of the agent based on the plurality of tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools.

[0015] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0019] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described above and any possible implementation thereof.

[0020] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0021] The technology disclosed herein can effectively improve the modeling accuracy and efficiency of intelligent agents.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Detailed Implementation

[0023] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0024] Figure 1This is a schematic diagram based on the first embodiment of the present disclosure;

[0025] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0026] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0027] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0028] Figure 5 This is a block diagram of an electronic device used to implement the methods of the embodiments of this disclosure. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] Obviously, the described embodiments are only some, not all, of the embodiments disclosed herein. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0031] It should be noted that the terminal devices involved in the embodiments of this disclosure may include, but are not limited to, smart devices such as mobile phones, personal digital assistants (PDAs), wireless handheld devices, and tablet computers; the display devices may include, but are not limited to, personal computers, televisions, and other devices with display functions.

[0032] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0033] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; as shown Figure 1 As shown, this embodiment provides a method for modeling an intelligent agent, which may specifically include the following steps:

[0034] S101, Obtain input request;

[0035] In this embodiment, the input requests from the agent can adopt different formats depending on the application scenario. For example, input requests can include text-based input requests or speech-based input requests. On the agent side, after receiving a speech-based input request, it can perform Automatic Speech Recognition (ASR). After obtaining a text-based input request, it can then perform corresponding processing. In practical applications, other formats of input requests can also be used in other application scenarios, which will not be elaborated on here.

[0036] S102. Obtain several tool call trajectories generated by the agent based on the input request. Each tool call trajectory is used to identify the trajectory information of at least two tools called by the agent in the process of responding to the input request.

[0037] In this embodiment, to improve the accuracy of the modeling agent during modeling, for each input request, after the input request is input to the agent, the agent can generate several tool call trajectories based on the input request. The specific number of trajectories can be set according to needs and experience, for example, it can be set to an integer greater than 1, such as 6, 8, or 16 trajectories.

[0038] In this embodiment, each tool call trajectory may include a sequence of identifiers of at least two tools called during the response process, arranged in the order of call. Optionally, it may also include call information for each tool. The at least two tools called in this embodiment may include external tools and internal tools of the agent. Internal tools may refer to tools with functions integrated into the agent itself.

[0039] S103. For each tool call trajectory, obtain the target response result of the agent responding to the input request based on the corresponding tool call trajectory, as well as the intermediate response result of calling internal tools during the response process;

[0040] For each tool call trajectory, the target response result obtained when the agent responds based on that trajectory can be acquired. Additionally, in this embodiment, it is also necessary to acquire the intermediate response results corresponding to the calls to internal tools within the tool call trajectory. If a tool call trajectory involves multiple steps of calling internal tools, the intermediate response result for each call to an internal tool needs to be acquired separately.

[0041] S104. Based on several tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, multi-objective modeling is performed on the agent.

[0042] In this embodiment, based on the obtained call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, the agent is modeled in multiple objectives. This enables the agent to not only optimize the accuracy of tool call trajectory generation but also the accuracy of internal tool calls, thereby effectively improving the accuracy of the agent's planning and response.

[0043] The agent modeling method in this embodiment performs multi-objective modeling of the agent by using several tool call trajectories based on input requests, as well as the target response results of each tool call trajectory and the intermediate response results of calling internal tools. This can effectively improve the modeling accuracy and efficiency of the agent.

[0044] Figure 2 This is a schematic diagram based on the second embodiment of this disclosure; the modeling method of the intelligent agent in this embodiment, in the above... Figure 1 Based on the technical solutions of the illustrated embodiments, the technology of this disclosure will be further described in more detail, such as... Figure 2 As shown, the modeling method for the intelligent agent in this embodiment may specifically include the following steps:

[0045] S201, Obtain input request;

[0046] S202. Obtain several tool call trajectories generated by the intelligent agent based on the input request;

[0047] Specifically, based on an input request, the agent can use a Large Language Model (LLM) to plan several tool call trajectories. The LLM used by the agent can be considered as an integrated version of the LLM used by the agent.

[0048] S203. For each tool call trajectory, obtain the target response result of the agent responding to the input request based on the corresponding tool call trajectory, as well as the intermediate response result of calling internal tools during the response process;

[0049] For example, consider a tool call path including a1→s1→a2→s2→s3→……→Sn. Here, tool identifiers a1 and a2 are external tools, while tool identifiers s1, s2, s3, and sn are internal tools. In this embodiment, external tools can refer to tools that can run without calling the current LLM, such as a mathematical calculator or a Python execution environment.

[0050] The internal tools in this embodiment may refer to tools with certain functions inherent in the LLM itself within the intelligent body, such as at least one of summarizing, reflecting, code generation, and deduction.

[0051] In this embodiment, for the tool call trajectory, it is necessary to obtain the target response result of the agent based on the tool call trajectory, as well as the intermediate response results when the internal tools s1, s2, s3 and sn are called respectively.

[0052] For each tool call trajectory, the target response result and all intermediate response results of internal tool calls are obtained accurately and comprehensively in the manner described above.

[0053] The specific implementation methods of steps S201-S203 can also refer to the above. Figure 1 The steps S101-S103 of the illustrated embodiment are described.

[0054] S204. Based on several tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, and referring to the performance parameters required by the agent, reinforcement learning (RL) is used to perform multi-objective modeling of the LLM of the integrated agent.

[0055] The performance parameters required by the agent in this embodiment are used to identify the performance requirements of the agent in different scenarios. In this embodiment, the performance parameters can be used to identify the performance of the agent's internal tools, or they can be considered as the performance of the LLM's internal tools within the agent.

[0056] For example, in this embodiment, the performance parameters required by the agent may include at least one of summarizing, reflecting, code generation, and deduction. For instance, in some application scenarios, the agent's summarizing ability needs to be emphasized; in this case, summarizing should be included as a required performance parameter. In other application scenarios, the agent's deduction ability needs to be used frequently; in this case, deduction should be included as a required performance parameter. Still other application scenarios require the agent to have a certain level of reflecting ability; in this case, reflecting should be included as a required performance parameter. In still other application scenarios, the agent needs to have very strong code generation ability; in this case, code generation should be included as a required performance parameter. In practical applications, the required performance parameters of the agent can be determined according to the needs of the application scenario. Specifically, the required performance parameters of the agent may include at least one of the above-mentioned performance parameters. In practical applications, more performance parameters can be set according to the requirements of the application scenario of the agent; these will not be listed in detail here.

[0057] In the modeling of this embodiment, based on the several tool call trajectories obtained above, the target response results of each tool call trajectory, and the intermediate response results of the internal tool calls, the LLM of the agent is modeled in a multi-objective manner using the RL approach. This allows the LLM to not only optimize the accuracy of the planned tool call trajectories, but also optimize the accuracy of the internal tool calls, thereby comprehensively improving the accuracy of the agent.

[0058] In addition, in this embodiment, the performance parameters required by the agent can be referenced during the modeling process, so that LLM can also meet the performance requirements of the agent application scenario and comprehensively improve the performance of the agent.

[0059] Alternatively, in one embodiment of this disclosure, step S104 can also employ other methods to perform multi-objective modeling of the agent. For example, based on several tool call trajectories, other evaluation tools can be used to evaluate the optimal tool call trajectory, and the optimal tool call trajectory can be used to perform RL modeling of the LLM; alternatively, the target response results and internal tool call intermediate response results of each tool call trajectory can be combined with the target response results and internal tool call intermediate response results evaluated by other evaluation tools to achieve the optimal result; and then, based on the combined optimal target response results and internal tool call intermediate response results, RL modeling of the LLM can be performed. In practical applications, other methods can also be used to implement multi-objective modeling in step S104, which will not be elaborated here.

[0060] Further, optionally, in one embodiment of this disclosure, step S204 may include the following steps:

[0061] (1) Based on the tool call trajectory, the target response result of the tool call trajectory and the intermediate response result of the internal tool call, and with reference to the performance parameters required by the agent, obtain the first reward value and the second reward value of each tool call trajectory;

[0062] The first reward value is used to characterize the reward corresponding to the tool call trajectory; the second reward value is used to characterize the reward corresponding to the internal tool call in the tool call trajectory.

[0063] In other words, in this embodiment, each tool call trajectory can be evaluated from two dimensions. On the one hand, the reward is evaluated from the dimension of the tool call trajectory itself to assess its effectiveness; on the other hand, the reward is evaluated from the dimension of internal tool calls to assess the effectiveness of the internal tool calls involved in the tool call trajectory.

[0064] Optionally, in one embodiment of this disclosure, the first reward value of each tool call trajectory can be evaluated using a preset evaluation tool or a pre-trained evaluation model, based on each tool call trajectory and the target response result of each tool call trajectory.

[0065] The first reward value of the tool call trajectory is used to comprehensively represent the reward brought by the tool call trajectory. It includes not only the reward of the tool call trajectory itself, but also the reward brought by the target response result based on the tool call trajectory.

[0066] For example, the first reward value of the tool call trajectory can be represented by the following formula (1):

[0067] R t =R1+R final (1)

[0068] Among them, R t R represents the first reward value of the tool call trajectory, and R1 represents the reward value of the tool call trajectory itself; R final This represents the reward value corresponding to the target response result of the tool's call trajectory.

[0069] Furthermore, the evaluation of the tool call trajectory itself can be carried out from multiple dimensions such as format, execution correctness, number of tool calls, and rationality of tool calls. Therefore, the first reward value of the tool call trajectory can be represented by the following formula (2):

[0070] R t =R format + R tool +R CallTime +R rationality +R final (2)

[0071] Among them, R t R represents the first reward value of the tool's call trajectory. format The reward value, R, represents the format of the tool call trajectory evaluation. tool R represents the reward value for evaluating the correctness of the tool call trajectory execution. CallTime The reward value representing the number of tool calls evaluated in the tool call trajectory; R rationality R represents the reward value for evaluating the reasonableness of tool call trajectory; final This represents the reward value corresponding to the target response result of the tool's call trajectory.

[0072] Similarly, the evaluation of each reward value also employs a preset evaluation tool or a pre-trained evaluation model. The evaluation of the format dimension of the tool call trajectory includes assessing the overall format of the tool call trajectory and the call format of each tool within it. The evaluation of the execution correctness of the tool call trajectory includes assessing the correctness of the overall execution result. The evaluation of the number of tool calls within the tool call trajectory is used to evaluate the reasonableness of the number of tool calls included in the trajectory. A single tool call trajectory should not have too many or too few tool calls; too many may increase response latency, while too few may lead to poor response accuracy. During the evaluation process, the more reasonable the number of tool calls, the higher the corresponding reward value; the more unreasonable the number of tool calls, the lower the corresponding reward value. The reasonableness of tool calls is used to evaluate whether the calls to each tool within the tool call trajectory are reasonable. For a tool call trajectory, the higher the reward value is, the more reasonable each tool call is. The more unreasonable tool calls a tool call trajectory involves, the lower the reward value is.

[0073] Optionally, in practical applications, reward values ​​for evaluating tool call trajectories in other dimensions can be added, which will not be listed here.

[0074] Using the above method, the tool call trajectory can be comprehensively and accurately evaluated effectively.

[0075] Optionally, in one embodiment of this disclosure, the second reward value of each tool call trajectory can be evaluated based on the target response result of each tool call trajectory and the intermediate response result of calling internal tools, using a preset evaluation tool or a pre-trained evaluation model, and referring to the performance parameters required by the agent.

[0076] The second reward value of the tool call trajectory is used to comprehensively characterize the rewards brought by the internal tool calls in the trajectory. This includes not only the rewards of each internal tool call itself, but also the rewards derived from the target response results based on the tool call trajectory. Further optionally, in this embodiment, the rewards for internal tool calls may also include rewards for the performance parameters required by the agent.

[0077] For example, the second reward value of the tool call trajectory can be represented by the following formula (3):

[0078] R t = R final +R2+R3(3)

[0079] Here, R2 represents the reward value of the internal tool. If a tool call trajectory includes multiple internal tool calls, the reward value of each internal tool call needs to be accumulated here. R3 represents the reward value corresponding to the performance parameters required by the agent. Specifically, the performance parameters of each internal tool call in the tool call trajectory are evaluated using an evaluation tool or evaluation model to obtain the corresponding reward value. Correspondingly, R... t It also requires accumulating the reward value for each performance parameter of the internal tool call at each step in the tool call trajectory.

[0080] Using the above method, we can comprehensively and accurately evaluate the internal tools in the tool call trajectory.

[0081] (2) Construct the target loss function based on the first and second reward values ​​of each tool call trajectory;

[0082] Following the above method, the first reward value and the second reward value corresponding to the internal tool of each tool call trajectory can be obtained. In this embodiment, a target loss function can be constructed together based on the first and second reward values ​​of each tool call trajectory in several tool call trajectories.

[0083] For example, in a specific implementation, the following steps may be included:

[0084] (a) Construct a first loss function based on the first reward value of each tool call trajectory;

[0085] Specifically, the first loss function is constructed based on the first reward value of each tool call trajectory in the tool call trajectory dimension.

[0086] For example, in the specific construction, the first reward average and first reward standard deviation can be calculated based on the first reward value of each tool call trajectory. Then, based on the first reward value, first reward average, and first reward standard deviation of each tool call trajectory, a first loss function is constructed. Specifically, for each tool call trajectory, the difference between the first reward value and the first reward average of that tool call trajectory can be divided by the first reward standard deviation, which can be used as a reference intermediate value for that tool call trajectory, such as the relative advantage value of that tool call trajectory. Then, based on the relative advantage value of the tool call trajectory, the first loss function is constructed using certain rules, which allows the LLM model to focus on learning tool call trajectories with higher reward values. The tool call trajectory with higher reward value corresponds to the tool call trajectory with higher accuracy. Therefore, it can also be considered that this first loss function allows the LLM model to focus on learning tool call trajectories with higher accuracy in the tool call trajectory dimension.

[0087] Specifically, the construction principle of the first loss function can be referred to the construction of the loss function in Group Relative Policy Optimization (GRPO), which will not be repeated here.

[0088] (b) Construct a second loss function based on the second reward value of each tool call trajectory;

[0089] Specifically, the second loss function is constructed based on the second reward value of each tool call trajectory in the internal tool call dimension;

[0090] For example, in specific construction, the corresponding average and standard deviation of the second reward can be calculated based on the second reward value of each tool call trajectory. Then, based on the second reward value of each tool call trajectory, as well as the average and standard deviation of the second reward, a second loss function can be constructed. The construction of the first loss function described above can be referenced. The first loss function is constructed from the tool call trajectory dimension, while the second loss function is constructed from the internal tool call dimension; the underlying principle is the same. The purpose of constructing the second loss function is to enable the LLM model to focus on learning the response results of internal tools with higher reward values. Alternatively, it can be considered that this second loss function enables the LLM model to focus on learning intermediate response results with higher accuracy in the internal tool call dimension. Similarly, the construction principle of the second loss function can refer to the construction of the loss function in GRPO, which will not be elaborated here.

[0091] (c) Construct the target loss function based on the first loss function and the second loss function.

[0092] Specifically, the target loss function can be obtained by weighted summation of the first and second loss functions. The specific weight ratios can be set according to requirements.

[0093] By adopting the above method, the first loss function and the second loss function can be constructed accurately and effectively. Based on the first loss function and the second loss function, the target loss function can be obtained efficiently and accurately.

[0094] (3) Adjust the parameters of the large language model based on the target loss function.

[0095] In this embodiment, the target loss function obtained through the above method can include information on the tool call trajectory dimension and the internal tool call dimension. Then, by using the target loss function and adjusting the LLM parameters according to the gradient descent method, target optimization in two dimensions can be achieved, that is, multi-objective modeling of LLM can be realized.

[0096] Based on the above analysis, it can be seen that the multi-objective modeling of this embodiment is based on several tool call trajectories and the target response results of each tool call trajectory. Referring to the performance parameters required by the agent, a reinforcement learning (RL) approach is used to model the LLM of the integrated agent in the tool call trajectory dimension. Furthermore, based on the target response results of each tool call trajectory and the intermediate response results of internal tool calls during the response process, and referring to the performance parameters required by the agent, an RL approach is used to model the LLM of the integrated agent in the internal tool call dimension. Ultimately, this allows the agent to not only improve the accuracy of the planned tool call trajectories but also effectively improve the accuracy of the agent's response to each internal tool call, thereby further improving the agent's overall accuracy. The agent modeling method of this embodiment, through the above-described approach and the RL approach, not only optimizes the LLM of the agent in planning tool call trajectories but also optimizes the LLM's skills in calling internal tools, effectively improving the agent's performance, modeling accuracy, and modeling efficiency. Moreover, the modeling method is simple, convenient, and highly practical.

[0097] Furthermore, the modeling method for the intelligent agent in this embodiment also takes into account the performance parameters required by the intelligent agent, which can further optimize the skills of internal tool invocation and improve the modeling accuracy and efficiency of the intelligent agent.

[0098] Figure 3 This is a schematic diagram based on the third embodiment of this disclosure; as shown Figure 3 As shown, this embodiment provides a modeling device 300 for an intelligent agent, including:

[0099] The input acquisition module 301 is used to acquire input requests;

[0100] The trajectory acquisition module 302 is used to acquire several tool call trajectories generated by the agent based on the input request. Each tool call trajectory is used to identify the trajectory information of at least two tools called by the agent in the process of responding to the input request.

[0101] The response acquisition module 303 is used to acquire, for each tool call trajectory, the target response result of the agent responding to the input request based on the corresponding tool call trajectory, as well as the intermediate response result of calling internal tools during the response process;

[0102] The modeling module 304 is used to perform multi-objective modeling of the agent based on the plurality of tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools.

[0103] The intelligent agent modeling device 300 in this embodiment achieves the same implementation principle and technical effect as the above-mentioned related method embodiments by using the above-mentioned modules. For details, please refer to the description of the above-mentioned related method embodiments, which will not be repeated here.

[0104] Figure 4 This is a schematic diagram based on the fourth embodiment of the present disclosure; as shown Figure 4 As shown, the modeling device 400 for the intelligent agent in this embodiment, in the above-described... Figure 3 Based on the technical solutions of the illustrated embodiments, the technical solutions of this disclosure will be further described in more detail. For example... Figure 4 As shown, the modeling device 400 for intelligent agents in this embodiment includes... Figure 3 The modules with the same name and function shown are: input acquisition module 401, trajectory acquisition module 402, response acquisition module 403, and modeling module 404.

[0105] In this embodiment, the modeling module 404 is used for:

[0106] Based on the aforementioned tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, and with reference to the performance parameters required by the agent, a reinforcement learning approach is used to perform multi-objective modeling on the large language model integrating the agent.

[0107] Further optional, such as Figure 4 As shown, in one embodiment of this disclosure, the modeling module 404 may include:

[0108] The reward acquisition unit 4041 is used to acquire a first reward value and a second reward value for each tool call trajectory based on the tool call trajectory in the plurality of tool call trajectories, the target response result of each tool call trajectory and the intermediate response result of calling internal tools, and with reference to the performance parameters required by the agent. The first reward value is used to characterize the reward corresponding to the tool call trajectory; the second reward value is used to characterize the reward corresponding to the internal tool call in the tool call trajectory.

[0109] Construction unit 4042 is used to construct a target loss function based on the first reward value and the second reward value of each tool call trajectory;

[0110] The adjustment unit 4043 is used to adjust the parameters of the large language model based on the target loss function.

[0111] Further optionally, in one embodiment of this disclosure, the reward acquisition unit 4041 is used for:

[0112] Based on the tool call trajectories and the target response results of each tool call trajectory, the first reward value of each tool call trajectory is evaluated using a preset evaluation tool or a pre-trained evaluation model.

[0113] Based on the target response results of each tool call trajectory and the intermediate response results of calling internal tools, the second reward value of each tool call trajectory is evaluated using a preset evaluation tool or a pre-trained evaluation model, and with reference to the performance parameters required by the agent.

[0114] Further optionally, in one embodiment of this disclosure, the construction unit 4042 is used for:

[0115] Based on the first reward value of each tool call trajectory, a first loss function is constructed;

[0116] A second loss function is constructed based on the second reward value of each tool call trajectory.

[0117] The target loss function is constructed based on the first loss function and the second loss function.

[0118] Further optionally, in one embodiment of this disclosure, the construction unit 4042 is used for:

[0119] Based on the first reward value of each tool call trajectory, calculate the corresponding first reward average value and first reward standard deviation;

[0120] The first loss function is constructed based on the first reward value of each tool call trajectory, the average value of the first reward, and the standard deviation of the average value of the first reward.

[0121] Further optionally, in one embodiment of this disclosure, the construction unit 4042 is used for:

[0122] Based on the second reward value of each tool call trajectory, calculate the corresponding average second reward and standard deviation of the second reward.

[0123] The second loss function is constructed based on the second reward value of each tool call trajectory, the second reward average value, and the second reward standard deviation.

[0124] Further, optionally, in one embodiment of this disclosure, the performance parameters required by the agent include at least one of summarizing, reflecting, code generation, and deduction.

[0125] The intelligent agent modeling device 400 in this embodiment achieves the same implementation principle and technical effect as the above-mentioned related method embodiments by using the above-mentioned modules. For details, please refer to the description of the above-mentioned related method embodiments, which will not be repeated here.

[0126] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0127] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0128] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0129] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0130] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the methods of this disclosure. For example, in some embodiments, the methods of this disclosure may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the methods of this disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the methods of this disclosure by any other suitable means (e.g., by means of firmware).

[0131] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0133] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0135] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0136] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0138] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for modeling an intelligent agent, comprising: Get input request; Obtain several tool call trajectories generated by the agent based on the input request, each of the tool call trajectories being used to identify the trajectory information of at least two tools called by the agent in response to the input request; For each tool call trajectory, obtain the target response result of the agent responding to the input request based on the corresponding tool call trajectory, as well as the intermediate response result of calling internal tools during the response process; Based on the aforementioned tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, the agent is modeled in multiple objectives.

2. The method according to claim 1, wherein, Based on the aforementioned tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, multi-objective modeling is performed on the intelligent entity, including: Based on the aforementioned tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, and with reference to the performance parameters required by the agent, a reinforcement learning approach is used to perform multi-objective modeling on the large language model integrating the agent.

3. The method according to claim 2, wherein, Based on the aforementioned tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools, and referring to the performance parameters required by the agent, a reinforcement learning approach is used to perform multi-objective modeling on the large language model integrating the agent, including: Based on each of the tool call trajectories in the plurality of tool call trajectories, the target response result of each tool call trajectory and the intermediate response result of calling internal tools, and with reference to the performance parameters required by the agent, a first reward value and a second reward value are obtained for each tool call trajectory. The first reward value is used to characterize the reward corresponding to the tool call trajectory; the second reward value is used to characterize the reward corresponding to the internal tool call in the tool call trajectory. Based on the first reward value and the second reward value of each tool call trajectory, a target loss function is constructed; Based on the target loss function, the parameters of the large language model are adjusted.

4. The method according to claim 3, wherein, Based on each of the aforementioned tool call trajectories, the target response result of each tool call trajectory, and the intermediate response result of calling internal tools, and with reference to the performance parameters required by the agent, a first reward value and a second reward value are obtained for each tool call trajectory, including: Based on the tool call trajectories and the target response results of each tool call trajectory, the first reward value of each tool call trajectory is evaluated using a preset evaluation tool or a pre-trained evaluation model. Based on the target response results of each tool call trajectory and the intermediate response results of calling internal tools, the second reward value of each tool call trajectory is evaluated using a preset evaluation tool or a pre-trained evaluation model, and with reference to the performance parameters required by the agent.

5. The method according to claim 3, wherein, Based on the first reward value and the second reward value of each tool call trajectory, a target loss function is constructed, including: Based on the first reward value of each tool call trajectory, a first loss function is constructed; A second loss function is constructed based on the second reward value of each tool call trajectory. The target loss function is constructed based on the first loss function and the second loss function.

6. The method according to claim 5, wherein, Based on the first reward value of each tool call trajectory, a first loss function is constructed, including: Based on the first reward value of each tool call trajectory, calculate the corresponding first reward average value and first reward standard deviation; The first loss function is constructed based on the first reward value of each tool call trajectory, the average value of the first reward, and the standard deviation of the average value of the first reward.

7. The method according to claim 5, wherein, Based on the second reward value of each tool call trajectory, a second loss function is constructed, including: Based on the second reward value of each tool call trajectory, calculate the corresponding average second reward and standard deviation of the second reward. The second loss function is constructed based on the second reward value of each tool call trajectory, the second reward average value, and the second reward standard deviation.

8. The method according to any one of claims 2-7, wherein, The performance parameters required for the agent include at least one of summarization, reflection, code generation, and deduction.

9. A modeling apparatus for an intelligent agent, comprising: The input acquisition module is used to acquire input requests; The trajectory acquisition module is used to acquire several tool call trajectories generated by the agent based on the input request. Each tool call trajectory is used to identify the trajectory information of at least two tools called by the agent in response to the input request. The response acquisition module is used to acquire, for each tool call trajectory, the target response result of the agent responding to the input request based on the corresponding tool call trajectory, as well as the intermediate response result of calling internal tools during the response process; The modeling module is used to perform multi-objective modeling of the agent based on the plurality of tool call trajectories, the target response results of each tool call trajectory, and the intermediate response results of calling internal tools.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images