Information processing system, information processing method and program

Abstract

To provide an information processing system etc. which learn so as to effectively improve performance of an agent with respect to a task requiring a plurality of steps.SOLUTION: An information processing system comprises: an agent which outputs a row of a plurality of actions which include the action which outputs a command to an external tool responding to a user query; a feedback acquisition unit which acquires final feedback with respect to the outputted row of the plurality of actions; and a parameter update unit which updates a parameter of a model of the agent by using the feedback.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] In recent years, the learning and application of tool-assisted agents using language models has become widespread. These agents aim to solve complex tasks that cannot be solved by language models alone or that require some kind of external intervention. Traditional methods mainly use distillation learning from the behavioral history of a teacher agent (e.g., GPT-4) (e.g., FireAct) or methods that use textual feedback to change the agent's behavior (e.g., LATS). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Chen, Baian, et al. "Fireact: Toward language agent fine-tuning." arXiv preprint arXiv:2310.05915 (2023). [Non-patent document 2] Zhou, Andy, et al. "Language agent tree search unifies reasoning acting and planning in language models." arXiv preprint arXiv:2310.04406 (2023). [Non-patent document 3] Petrov, Aleksandar, Philip HS Torr, and Adel Bibi. "When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations." arXiv preprint arXiv:2310.19698 (2023). [Non-patent document 4] Qiao, Shuofei, et al. "Making Language Models Better Tool Learners with Execution Feedback." arXiv preprint arXiv:2305.13068(2023). Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method of distillation learning from the behavioral history of a teacher agent may suffer from a decline in performance due to errors in a high-performance teacher agent, or because the behavioral history of the teacher agent may become training data that ignores the existence of a performance difference (for example, amount of knowledge) between the agent being trained and the teacher agent. Also, the method of changing agent behavior using text-format feedback shown in Non-Patent Document 2 has the problem that, as shown in Non-Patent Document 3, text-format feedback is inferior in its ability to change the behavior of a language model compared to methods that update the parameters of an agent model.

[0005] Furthermore, in order to solve these problems, TRICE, shown in the above-mentioned Non-Patent Document 4, employs a method in which distillation learning is performed from the behavioral history of a teacher agent, and then the parameters of the agent being trained are updated using the agent's own behavior and feedback from the outside world in response to that behavior, thereby further improving performance.

[0006] The TRICE technology described above primarily assumes tasks that can be completed in one step and only accepts feedback corresponding to one of the agent's actions. However, in reality, there are many complex tasks that require multiple steps, and there is a need for learning that is suitable for the multiple actions of an agent performing such complex tasks.

[0007] The present disclosure has been made to solve such problems, and aims to provide an information processing device and the like that learns to effectively improve the performance of an agent for tasks that require multiple steps. [Means for solving the problem]

[0008] The information processing system according to the present disclosure includes an agent that outputs a sequence of multiple actions including an action that outputs a command to an external tool in response to a user query, a feedback acquisition unit that acquires final feedback for the output sequence of multiple actions, and a parameter update unit that updates parameters of a model of the agent using the final feedback.

[0009] An information processing method executed by a computer according to the present disclosure includes an agent outputting a sequence of multiple actions including an action that outputs a command to an external tool in response to a user query, obtaining final feedback for the output sequence of multiple actions, and updating parameters of a model of the agent using the final feedback.

[0010] The program according to the present disclosure causes a computer to execute the following processes: a process of outputting a sequence of multiple actions by an agent, the sequence including an action that outputs a command to an external tool in response to a user query; a process of obtaining final feedback for the output sequence of multiple actions; and a process of updating parameters of a model of the agent using the final feedback. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide an information processing system and the like that learns to effectively improve the performance of an agent for a task that requires multiple steps. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an agent learning system according to some embodiments of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing a configuration of an information processing system 1 according to some embodiments of the present disclosure. [Figure 3] FIG. 3 is a flowchart illustrating an information processing method according to the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating a configuration example of an information processing device according to another embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments. In addition, the following description and drawings have been simplified as appropriate for clarity of explanation.

[0014] Figure 1 shows an agent learning system according to some embodiments. A use case assumed in the present embodiments will be explained using a specific example. The agent model learning system of the present embodiments efficiently learns a model of an agent that converses with a user.

[0015] This information processing system 1 may include an information processing device (server computer) 100 having an agent model and its learning function, and a user terminal 20 such as a smartphone or personal computer owned by a general user 2 that is communicatively connected to the information processing device 100.

[0016] A user uses a user terminal to complete a task consisting of multiple utterances, such as planning a trip or making a restaurant reservation, while interacting with an agent on a server. In particular, the task according to this embodiment may be a complex task that requires results (also called behavioral results) output from an external tool (e.g., a search engine, a calculator, an email program, etc.) and cannot be solved by a language model alone. Such a complex task may require knowledge that is more current than the time the language model was trained, or a task that requires intervention from outside the language model. During the conversation, the agent asks questions and makes suggestions to the user. The user sends a simple evaluation, such as "good" or "bad," for each utterance made by the agent via their smartphone. The simple evaluation data, such as "good" or "bad," may be converted into numerical data. The learning device on the server uses these evaluations to update the parameters of the agent model and improve its conversational ability.

[0017] Furthermore, the user can evaluate the agent's utterances during the task and generate intermediate feedback. This allows the agent's conversational ability to be improved efficiently, even for tasks that involve multiple utterances. This makes it possible to create a high-performance language model agent for tasks that require multiple use of external tools to solve problems, such as RPA (Robotic Process Automation).

[0018] In this way, this embodiment is a system that makes maximum use of feedback obtained from conversations with actual users and realizes effective learning of the agent model.

[0019] (Configuration of information processing system) The configuration of an information processing system 1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of an information processing system 1 according to some embodiments. The information processing system 1 includes one or more information processing devices 100. The information processing device 100 may be, for example, an information processing device such as a server computer. As shown in FIG. 2 , the information processing device 100 includes an agent 110, a feedback acquisition unit 120, and a parameter update unit 130.

[0020] The information processing system 1 may include a plurality of information processing devices. For example, the information processing system 1 may include an information processing device including an agent 110 (agent model), a user terminal including a feedback acquisition unit 120, and a learning device including a parameter update unit 130. In other words, the components of the information processing device may be distributed and processed by a plurality of computers connected to each other so as to be able to communicate with each other.

[0021] The agent 110 generates a string in response to a user query, thereby outputting a sequence of multiple actions, including an action that outputs a command to an external tool (i.e., a string that expresses how to use the external tool). The agent 110 uses an untrained agent model (i.e., a language model) to directly answer a given task (problem), and outputs a sequence of multiple actions. The agent model (i.e., the language model used in this specification) is not limited to traditional language models that handle only text information, but may also include, for example, a model that generates image or audio data and presents the image or audio data to a user.

[0022] The language model used in this specification is a model that learns the relationships between words in a sentence and generates related strings related to a target string from the target string. By using a language model that has been trained with sentences and paragraphs from various contexts, it is possible to generate related strings with appropriate content related to the target string.

[0023] For example, let us consider a case where a language model is used in question answering. The language model receives an input question such as "What kind of country is Japan?" as a target string. The language model generates a string such as "Japan is an island country in the Northern Hemisphere..." as an answer to the question. The learning method of the language model is not particularly limited, but as an example, the language model may be learned to output at least one sentence that includes an input character string.

[0024] To cite a specific example, a language model is a Generative Pre-trained Transformer (GPT), which outputs sentences containing an input string by predicting a string that is likely to follow the input string. Other examples of language models include T5 (Text-to-Text Transfer Transformer), BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly optimized BERT approach), and ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately).

[0025] Furthermore, the character strings generated by the language model are not limited to natural languages. For example, the language model may output an artificial language (such as program source code) for a character string input in a natural language. For example, the language model may accept an input question such as "How do I retrieve data containing a specific character string from a database?" as a target character string. The language model may output program source code for performing database processing. Alternatively, the language model may output a natural language corresponding to a character string input in an artificial language.

[0026] The language model may also input image data, video data, audio data, or data in other data formats.

[0027] Furthermore, the content generated by the language model is not limited to character strings. For example, the language model may generate image data, video data, audio data, or other data formats corresponding to the input character string.

[0028] The behavior of the agent 110 refers to the response from the agent model at each step (e.g., a response to the user and a command output for using an external tool). A sequence of multiple behaviors is a series of behaviors output in response to a user query. It refers to all of the agent's multiple behaviors until the task is completed (in FIG. 1, until a final response is obtained), regardless of whether the task in response to the user query is successful. The task in this embodiment may require knowledge that is more current than the knowledge at the time the language model was trained, or may require intervention outside the language model. Therefore, the sequence of multiple behaviors for solving the task includes at least one behavior that outputs a command expressing how to use the external tool. The sequence of multiple behaviors may include a first behavior that outputs a command expressing how to use the external tool, and a second behavior based on the action result obtained from the external tool as a result of outputting the first behavior. Note that the sequence of multiple behaviors may include one or more other behaviors before the first behavior or after the second behavior.

[0029] The external tool here refers to a tool external to the agent model, which can be operated by a command output from the agent 110. The external tool may be, for example, application software pre-installed in the information processing device. Examples of application software that can be an external tool include, but are not limited to, a browser, a search engine, email software, a calculator, and a control program for a built-in camera.

[0030] The agent 110 may store an action history including multiple actions and action results (ie, responses from external tools) in a storage unit of the information processing system.

[0031] The feedback acquisition unit 120 acquires feedback on a sequence of multiple actions of the agent 110. In this specification, feedback received after the completion of a task may also be called final feedback (in contrast to intermediate feedback, which will be described later). The feedback acquisition unit 120 may acquire the agent 110's behavioral history and user evaluation data for that behavioral history. The behavioral history refers to a sequence of one or more agent actions. The user's evaluation data may be provided directly to the information processing device by a human user. Alternatively, data mechanically generated by calculation processing within the information processing device may be used as the evaluation data. The feedback acquisition unit 120 acquires feedback when the task is completed. A more detailed method for receiving feedback will be described later.

[0032] The parameter update unit 130 uses the acquired feedback to update the parameters of the agent model of the agent 110. The parameter update unit 130 updates the parameters of the agent model using the behavior history and user evaluation data acquired by the feedback acquisition unit. A more detailed method for updating the parameters will be described later.

[0033] (Example 1 of how to accept feedback and update) The parameter update unit 130 uses user feedback on the behavioral history across multiple steps to update the parameters of the agent model. This process takes into account multiple past actions taken by the agent and feedback on those actions. The feedback is information indicating the degree to which the agent's actions contributed to achieving the task goal.

[0034] As an example, consider the case where a decoder-type language model is used as an agent. A decoder-type language model is a model that recursively generates a sentence that follows a given sentence. For example, an agent model expresses input from a user, a response from an external tool, and an agent's action as sentences, and then inputs these as sentences that are arranged in chronological order and combined into a single sentence into the model. In response, the agent model outputs the action for the next step as a continuation sentence.

[0035] Based on the final feedback obtained at the end of the task, the probability of the agent's output (e.g., a portion of a generated sentence) occurring at each step is adjusted. This adjustment of the probabilities allows the agent to learn more effective behaviors.

[0036] As a specific example, consider the case where a user inputs "Make an appointment with an internal medicine doctor in Tokyo" into the agent model of an information processing device. In the first step, the user inputs string A "Make an appointment with an internal medicine doctor in Tokyo" into the agent model, and the agent model responds by outputting a command string B "Web_Search("Tokyo Internal Medicine")" as an action. Then, by executing this command, a response string C "Tanaka Internal Medicine, Address: Tokyo, E-mail: tanaka@sample" is obtained from an external tool (for example, a search engine).

[0037] In the next step, character string D, which is a combination of character strings A, B, and C, is input to the agent model, and in response, the agent model outputs character string E, "Send_Mail("tanaka@sample", "I would like to make a reservation")" as an action (i.e., sends an email). After these steps are completed, the feedback acquisition unit 120 acquires final feedback according to the degree of success of the task as evaluation data.

[0038] Specifically, the feedback can be expressed, for example, as a numerical value greater than or equal to 0 and less than or equal to 1. In this case, for example, the closer the feedback value is to 1, the higher the likelihood of success for the task, and the closer the feedback value is to 0, the higher the likelihood of failure. Also, any number greater than or equal to 0 and less than or equal to 1 may be set as the threshold. In this case, the information processing device 100 may determine whether the task has succeeded or failed depending on whether the feedback exceeds the threshold. In this way, in this embodiment, since the feedback can be acquired as a numerical value, it is possible to perform parameter update to more appropriately change the behavior of the language model compared to feedback in text format.

[0039] The feedback may be obtained directly from the user via a user interface. For example, the information processing device 100 may display the degree of success of the task on the user interface and allow the user to select an answer through an operation. Alternatively, the information processing device 100 may obtain the degree of success of the task by estimating it from a facial image of the user captured by a camera. Furthermore, the information processing device 100 may obtain the degree of success of the task by estimating it from audio data of the user collected by a microphone. Furthermore, in another embodiment, the information processing device 100 may use data mechanically generated by some kind of arithmetic processing as the evaluation data.

[0040] Next, the parameter update unit 130 updates the parameters of the agent model using the obtained behavior history and evaluation data. To do this, the parameter update unit 130 acquires input / output pairs for the agent model at each step, i.e., (character string A, character string B) and (character string D, character string E), from the behavior history. Then, based on the evaluation data, the parameter update unit 130 adjusts the probability that the agent model will output a corresponding output for each input in these input / output pairs. For example, if the evaluation data is greater than a certain threshold (the task is successful), the parameter update unit 130 updates the parameters of the agent model using a gradient method or the like so that the probability of outputting character string B for character string A and character string E for character string D increases.

[0041] (Example 2 of how to accept feedback and update) The feedback acquisition unit 120 may acquire not only final feedback for the entire behavior history, but also intermediate feedback for some behaviors included in the behavior history.

[0042] For example, in the case of the specific example above, in addition to the final feedback according to the degree of success of the task described above, feedback may be given on the desirability of the action in step 1, i.e., the generation of character string B for character string A. In this case, the feedback acquisition unit 120 acquires, as evaluation data, intermediate feedback on the desirability of the action in step 1 described above, in addition to the final feedback according to the degree of success of the task in specific example 1.

[0043] In this case, the parameter update unit 130 also performs parameter updates that take into account intermediate feedback for some actions, in addition to the parameter updates in specific example 1. For example, in the case described above, if the desirability of the action in step 1 is greater than a certain threshold (i.e., the action is highly likely to be desirable), the parameters of the agent model are updated using a gradient method or the like so that the probability of generating character string B for character string A increases.

[0044] By updating parameters using this type of intermediate feedback, it is possible to achieve the following improvements in learning efficiency. For example, in the first step, outputting string B in response to string A is desirable for task completion. However, even if the task cannot be completed due to failures in steps after the first step, parameter updates can be performed to promote the behavior in the first step. In other words, by using intermediate feedback, it is possible to optimize parameters for intermediate actions in order to complete the final task, even if the agent's performance is low and the probability of achieving the final task is low.

[0045] (Method 1 for generating intermediate feedback) In the above specific example 2, a situation was described in which interim feedback was given from the outside (for example, a user), but the interim feedback may also be generated inside the feedback acquisition unit 120. For example, an evaluation model that outputs a score indicating how desirable the behavior of the evaluation target is for solving the task may be used. In the above case, a character string that combines character string A input by the user, which is the purpose of the task, character string B, which is the behavior of the agent model in response to the character string A, and an instruction statement that instructs the agent model to evaluate the desirability of the behavior of the agent model based on these and output the result as a numerical value, is input to a pre-trained language model. Then, the output character string obtained from the pre-trained language model is converted into a numerical value and can be used as interim feedback for the behavior in step 1.

[0046] (Method 2 for generating intermediate feedback) As another specific example of generating intermediate feedback within the feedback acquisition unit 120, a method will be described in which final feedback for the entire behavioral history obtained after task completion is used to estimate how desirable a state at a certain point in time in the behavioral history is for completing a task. In the above specific example, the state at a certain point in time in the behavioral history refers to the state at the time the first step is completed. That is, the state at a certain point in time in the behavioral history can refer to state S1 in which the agent outputs string B, which is a command string, in response to string A, which is a user query, and string C, which is information about Tanaka Internal Medicine Hospital, is obtained as a response from an external tool.

[0047] To estimate how desirable this state S1 is for solving the task of "making an appointment with an internal medicine doctor in Tokyo," we sample the behavioral histories and results of various different agents from the above state using a tree search method such as Monte-Carlo Tree Search. For details, see Zhou, Andy, et al. "Language agent tree search unifies reasoning, acting, and planning in language models," arXiv preprint arXiv:2310.04406 (2023).

[0048] Then, the feedback acquisition unit 120 acquires feedback indicating the degree of success of the task based on the sampled behavioral history and results in the manner described above. Subsequently, the feedback acquisition unit 120 can calculate the average of these acquired feedbacks as the value of state S1 based on the acquired feedback. For example, if the sample is successful 700 times out of 1000 times, a 70% success probability (i.e., a state value of 0.7) can be obtained. Then, the feedback acquisition unit 120 can acquire the value of state S1 described above as intermediate feedback for the action in the first step, i.e., outputting string B in response to string A.

[0049] (Flow of information processing method) FIG. 3 is a flowchart illustrating an information processing method according to the present disclosure. This information processing method is executed by a computer. Specifically, The agent 110 outputs a sequence of multiple actions, including an action that outputs a command to an external tool in response to a user query (S11). The feedback acquisition unit 120 acquires final feedback for the sequence of multiple actions that has been output (S12). The parameter update unit 130 updates the parameters of the agent model using the acquired feedback (S13).

[0050] The agent model can be optimized for each user through the above learning process. Alternatively, the agent model can be optimized for each user attribute (e.g., age, sex, occupation) through the above learning process. Also, a general-purpose agent model can be generated and the user attributes acquired to obtain a response that is appropriate for the user.

[0051] FIG. 4 is a block diagram showing an example configuration of the information processing device 100. Referring to FIG. 4, the information processing device 100 includes a network interface 1201, a processor 1202, and a memory 1203. The network interface 1201 is used to communicate with other network node devices constituting a communication system. The network interface 1201 may be used to perform wireless communication. For example, the network interface 1201 may be used to perform wireless LAN communication defined in the IEEE 802.11 series or mobile communication defined in 3GPP (3rd Generation Partnership Project) (registered trademark). Alternatively, the network interface 1201 may include, for example, a network interface card (NIC) conforming to the IEEE 802.3 series.

[0052] The processor 1202 reads and executes software (computer programs) from the memory 1203, thereby performing the processing of the information processing device 100 described using flowcharts or sequences in the above-mentioned embodiments. The processor 1202 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof.

[0053] The memory 1203 is configured by a combination of volatile memory and non-volatile memory. The memory 1203 may include storage located remotely from the processor 1202. In this case, the processor 1202 may access the memory 1203 via an I / O interface (not shown).

[0054] 4, the memory 1203 is used to store software modules. The processor 1202 reads and executes these software modules from the memory 1203, thereby performing the processing of the information processing device 100 described in the above embodiment.

[0055] As described with reference to FIG. 4, each of the processors included in the information processing device 100 executes one or more programs including a group of instructions for causing a computer to execute the algorithm described with reference to the drawings.

[0056] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0057] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0058] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0059] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. (Appendix 1) an agent that outputs a sequence of multiple actions, including an action that outputs a command to an external tool, in response to a user query; a feedback acquisition unit that acquires final feedback for the output sequence of the plurality of actions; a parameter update unit that updates parameters of the agent model using the final feedback; An information processing system comprising: (Appendix 2) The sequence of the plurality of actions includes: a first action of outputting a command representing how to use the external tool; and a second action based on the action result obtained from the external tool as a result of the first action being output. (Appendix 3) a feedback acquisition unit that acquires final feedback for the output sequence of the plurality of actions and intermediate feedback for some actions of the sequence of the plurality of actions, the final feedback and the intermediate feedback being numerical feedback; 3. The information processing system according to claim 1, further comprising: a parameter update unit that updates parameters of a model of the agent using the final feedback and the intermediate feedback. (Appendix 4) The information processing system according to any one of appendices 1 to 3, wherein the intermediate feedback is generated using an evaluation model that outputs a score indicating how desirable the behavior of the evaluation target is for solving the task. (Appendix 5) An information processing system according to any one of appendices 1 to 4, wherein a character string obtained by combining a first character string input by a user which is the object of a task, a second character string which is the action of the agent model in response to the first character string, and an instruction statement which instructs the agent model to evaluate the desirability of the action of the agent model based on the first character string and the second character string and output the evaluation as a numerical value is input to a pre-trained language model, and the obtained output character string is converted into a numerical value and used as intermediate feedback for the first action. (Appendix 6) 4. The information processing system of claim 3, wherein the intermediate feedback is generated by a tree search method that uses final feedback for the entire sequence of actions obtained after the task is completed to estimate how desirable a state value at a certain point in the sequence of actions is for completing the task. (Appendix 7) The information processing system according to any one of appendices 1 to 6, wherein the external tool is a software program that is external to the agent model and pre-installed in the information processing device. (Appendix 8) 4. The information processing system of claim 3, wherein the final feedback and the intermediate feedback are obtained from a user via a user interface and converted into numerical feedback. (Appendix 9) outputting, by the agent, a sequence of a plurality of actions including an action that outputs a command to an external tool in response to a user query; obtaining a final feedback for the sequence of the output actions; A computer-implemented information processing method that uses the final feedback to update parameters of a model of the agent. (Appendix 10) outputting, by the agent, a sequence of a plurality of actions including an action that outputs a command to an external tool in response to a user query; obtaining final feedback for the output sequence of the plurality of actions; and updating parameters of the agent model using the final feedback.

[0060] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 8 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 9 and 10 in the same dependency relationship as Supplementary Notes 2 to 8. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods. [Explanation of symbols]

[0061] 1. Information Processing Systems 2 users 20 User terminal 100 Information processing device 110 Agent 120 Feedback Income Department 130 Parameter Update Unit

Claims

1. an agent that outputs a sequence of multiple actions, including an action that outputs a command to an external tool, in response to a user query; a feedback acquisition unit that acquires final feedback for the output sequence of the plurality of actions; a parameter update unit that updates parameters of the agent model using the final feedback; An information processing system comprising:

2. The sequence of the plurality of actions includes: a first action of outputting a command representing a method of using the external tool; The information processing system according to claim 1 , further comprising: a second action based on an action result obtained from the external tool as a result of the first action being output.

3. a feedback acquisition unit that acquires final feedback for the output sequence of the plurality of actions and intermediate feedback for some actions of the sequence of the plurality of actions, the final feedback and the intermediate feedback being numerical feedback; 3. The information processing system according to claim 1, further comprising: a parameter update unit that updates parameters of the agent model using the final feedback and the intermediate feedback.

4. The information processing system according to claim 3 , wherein the intermediate feedback is generated using an evaluation model that outputs a score indicating how desirable the behavior of the evaluation target is for solving the task.

5. 3. The information processing system according to claim 1 or 2, wherein a character string obtained by combining a first character string input by a user which is the object of a task, a second character string which is the action of the agent model in response to the first character string, and an instruction statement which instructs the agent model to evaluate the desirability of the action of the agent model based on the first character string and the second character string and to output the evaluation as a numerical value is input to a pre-trained language model, and the obtained output character string is converted into a numerical value which is used as intermediate feedback for the first action.

6. 4. The information processing system of claim 3, wherein the intermediate feedback is generated by a tree search method that uses final feedback for the entire sequence of the plurality of actions obtained after the task is completed to estimate how desirable a state value at a certain point in the sequence of the plurality of actions is for completing the task.

7. 2. The information processing system according to claim 1, wherein the external tool is a software program that is external to the agent model and pre-installed in the information processing device.

8. The information processing system according to claim 3 , wherein the final feedback and the intermediate feedback are obtained in text form from the user via a user interface and converted into numerical feedback.

9. outputting, by the agent, a sequence of a plurality of actions including an action that outputs a command to an external tool in response to a user query; obtaining a final feedback for the sequence of the output actions; A computer-implemented information processing method that uses the final feedback to update parameters of a model of the agent.

10. outputting, by the agent, a sequence of a plurality of actions including an action that outputs a command to an external tool in response to a user query; obtaining final feedback for the output sequence of the plurality of actions; and updating parameters of the agent model using the final feedback.

Images