Trained models, information processing devices, information processing methods, and computer programs

Abstract

To significantly improve the overall performance of the system. [Solution] The system consists of multiple modules, each performing processing on input data and outputting output data, and achieves a specific objective through processing by multiple modules. The trained model has parameters learned by reinforcement learning. The trained model causes the computer to function to perform post-processing on all modules to retrospectively correct the output data from the modules.

Description

[Technical Field]

[0001] The technologies disclosed herein relate to trained models for post-processing that corrects the outputs of multiple modules constituting a system. [Background technology]

[0002] Task-oriented dialogue systems are known to support the completion of specific tasks (e.g., hotel reservations) in response to user requests. These systems achieve the requested task by engaging in one or more turns of natural language dialogue with the user.

[0003] Generally, task-oriented dialogue systems consist of multiple modules connected in series. For example, a task-oriented dialogue system has a pipeline structure in which a Natural Language Understanding module (hereinafter also called the "NLU module"), a Dialogue State Tracking module (hereinafter also called the "DST module"), a Policy module (hereinafter also called the "Policy module"), and a Natural Language Generation module (hereinafter also called the "NLG module") are connected in this order. In each turn of dialogue with the user, the task-oriented dialogue system outputs a single response statement after a series of processes by the above-mentioned multiple modules.

[0004] To improve the performance of task-oriented dialogue systems, one might consider optimizing each module that makes up the dialogue system using, for example, reinforcement learning. However, such optimization methods cannot be used when the parameters of each module that makes up the dialogue system cannot be directly accessed, such as when the dialogue system is implemented on an API basis.

[0005] To solve these problems, a method has been proposed that uses a pre-trained model for "post-processing" to retrospectively modify the output data (hereinafter simply referred to as "output") of each module constituting a task-oriented dialogue system. Hereafter, such a pre-trained model will also be referred to as a "post-processing network," "Post-processing Network," or "PPN." For example, Non-Patent Literature 1 proposes a technique for implementing three post-processing networks (hereinafter also referred to as "BinPPN") that perform post-processing for the NLU module, DST module, and Policy module, respectively, as multi-label binary classification models. Non-Patent Literature 2 also proposes a technique for implementing a post-processing network (hereinafter also referred to as "GenPPN") that performs post-processing of the system response statement output by the NLG module, as text conversion using a large-scale language model. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Atsumoto Ohashi, et al., "Optimization of a Task-Oriented Dialogue System by Reinforcement Learning Using a Post-Processing Network," Proceedings of the 28th Annual Meeting of the Association for Natural Language Processing, Association for Natural Language Processing, March 2022, pp. 375-379. [Non-Patent Document 2] Atsumoto Ohashi, et al., "Optimization of a Task-Oriented Dialogue System using a Generative Post-Processing Network," Proceedings of the 30th Annual Meeting of the Association for Natural Language Processing, Association for Natural Language Processing, March 2024, pp. 2427-2432. [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] The BinPPN and GenPPN described above can only be applied to some of the multiple modules that make up a dialogue system, individually. Furthermore, because these two post-processing networks have different model structures and learning algorithms, they cannot be used together to optimize the same dialogue system. While it is possible to perform post-processing on all modules constituting the dialogue system by combining individually optimized BinPPN and GenPPN, achieving optimization of the dialogue system as a whole is difficult. Therefore, there is a problem that the performance of the dialogue system as a whole cannot be sufficiently improved even when using these post-processing networks. This problem is not limited to task-oriented dialogue systems, but is common to systems that achieve a specific purpose through processing by multiple modules. [Means for solving the problem]

[0008] The technologies disclosed herein can be implemented, for example, in the following forms:

[0009] (1) The trained models disclosed herein are models having parameters learned by reinforcement learning. The trained models perform operations on a system. The system is composed of a plurality of modules, each performing operations on input data and outputting output data, and is a system that achieves a specific purpose through the operations performed by the plurality of modules. The trained models cause the computer to function to perform post-processing on all of the modules, thereby correcting the output data from the modules retrospectively. With this trained model, a single model can perform post-processing to correct the output data from all modules constituting the system retrospectively, thereby significantly improving the overall performance of the system.

[0010] (2) In the system described above in the trained model, the multiple modules may be connected in series. This configuration makes it possible to significantly improve the performance of a system having such a pipeline structure.

[0011] (3) In the trained model described above, the input data and the output data may be text sequences, and the post-processing may be a sequence conversion task. With this configuration, post-processing can be performed uniformly on the output of all modules constituting the system, and the performance of the system can be significantly improved.

[0012] (4) In the above-mentioned trained model, the reinforcement learning may be a module-level Markov decision process that uses the unit of post-processing by the module as the unit of time step in the Markov decision process. With this configuration, the training of the trained model can be performed stably, and the performance of the system can be sufficiently improved.

[0013] (5) In the above-mentioned trained model, imitation learning may be performed to learn the format of the input data and the output data before the reinforcement learning. With this configuration, the training of the trained model can be performed efficiently.

[0014] (6) In the above-mentioned trained model, the imitation learning may be performed by supervised fine-tuning. This configuration allows for efficient learning of the input data and output data formats.

[0015] (7) In the trained model described above, the imitation learning may be performed using a combination of the output of the module in a particular turn and the output of the module randomly sampled from another turn as demonstration data. This configuration allows for the efficient generation of demonstration data used for imitation learning.

[0016] (8) In the trained model described above, the other turn may be selected based on its similarity to the specific turn described above. This configuration makes it possible to efficiently generate high-quality demonstration data for use in imitation learning.

[0017] (9) In the trained model described above, the post-processing may include a process of copying the output data when it is not necessary to modify the output data from the module. This configuration allows for the explicit training of cases where it is not necessary to modify the output data from the module, while improving computational efficiency.

[0018] (10) In the trained model described above, the system may be a task-oriented dialogue system that, for the purpose described above, supports the achievement of a specific task in response to a user's request. With this configuration, a single model can perform post-processing to retrospectively modify the output data from all modules constituting the task-oriented dialogue system, thereby significantly improving the overall performance of the dialogue system.

[0019] (11) In the trained model described above, the system may include a language understanding module, a state tracking module, a policy module, and a language generation module as modules. With this configuration, post-processing can be performed to retrospectively modify the output data from all four modules that constitute the task-oriented dialogue system, thereby significantly improving the overall performance of the dialogue system.

[0020] (12) The information processing device disclosed herein comprises a model acquisition unit and a post-processing execution unit. The model acquisition unit acquires a trained model having parameters learned by reinforcement learning. The post-processing execution unit is composed of a plurality of modules, each of which performs processing on input data and outputs output data, and for a system that achieves a specific purpose through the processing by the plurality of modules, the post-processing unit uses the trained model to perform post-processing on all of the modules to retrospectively modify the output data from the modules. According to this information processing device, post-processing can be performed to retrospectively modify the output data from all modules constituting the system using a single model, thereby significantly improving the overall performance of the system.

[0021] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, in the form of an information processing device, an information processing method, a computer program that implements the method, a trained model, or a non-temporary recording medium that stores the computer program or the trained model. [Brief explanation of the drawing]

[0022] [Figure 1] A conceptual diagram illustrating the module output modification model. [Figure 2] An explanatory diagram showing an example of post-processing using a module output correction model. [Figure 3] An explanatory diagram showing an example of a method for generating demonstration data for post-processing used in imitation learning. [Figure 4] A conceptual diagram illustrating module-level MDP. [Figure 5] Diagram illustrating the general configuration of the information processing device. [Figure 6] A flowchart illustrating the processes performed by the information processing device. [Modes for carrying out the invention]

[0023] (Embodiment) (Regarding the module output modification model) First, we will describe the overview of the module output modification model MO introduced into the dialogue system DS in this embodiment. Figure 1 is a conceptual diagram illustrating the module output modification model MO. Figure 1 shows the dialogue system DS that interacts with user Us. In Figure 1, for convenience, two user Us and two dialogue systems DS are shown, but in reality, the two user Us are the same person, and the two dialogue systems DS are the same system.

[0024] The dialogue system DS in this embodiment is a task-oriented dialogue system that supports the achievement of a specific task (e.g., hotel reservation) in response to a request from the user Us. The dialogue system DS achieves the achievement of the requested task by engaging in one or more turns of dialogue with the user Us using natural language.

[0025] The DS dialogue system consists of four modules. Specifically, the DS dialogue system has a pipeline structure in which Module1 (NLU module), Module2 (DST module), Module3 (Policy module), and Module4 (NLG module) are sequentially linked. The NLU module estimates the intent of the utterance made by the user Us. The DST module updates the current dialogue state based on the estimated intent of the utterance. The dialogue state is various information about the user's intent up to the current point in the dialogue. The Policy module determines the next action of the DS dialogue system based on the updated dialogue state. The NLG module generates utterances based on the action output from the Policy module. The utterance made by the user Us is input to Module1, the output from Module1 is input to Module2, the output from Module2 is input to Module3, the output from Module3 is input to Module4, and the output from Module4 is considered the utterance from the DS dialogue system. The dialogue system DS outputs a single response statement after going through a series of processes by multiple modules during each turn of interaction with the user Us.

[0026] The Module Output Correction Model (MO) is a language model-based, trained model. It is a single model that performs post-processing to correct the output from all modules constituting the dialogue system DS. Hereafter, the Module Output Correction Model (MO) will also be referred to as the "Universal Post-processing Network" or "UniPPN". The input data (hereinafter simply referred to as "input") and output data (hereinafter simply referred to as "output") of the Module Output Correction Model (MO) are text sequences, and the post-processing performed by the Module Output Correction Model (MO) is a sequence conversion task.

[0027] As shown in Figure 1, when output out1 from Module1 is input to module output modification model MO, module output modification model MO performs post-processing to modify output out1, and the modified output out1 + This is output to the subsequent Module2. Similarly, when the output out2 from Module2 is input to the module output modification model MO, the module output modification model MO performs post-processing on the output out2 and outputs the modified output out2. + The output is directed to the subsequent Module3, and when the output out3 from Module3 is input to the module output modification model MO, the module output modification model MO performs post-processing on the output out3, and the modified output out3 + The output is directed to the subsequent Module4. When the output out4 from Module4 is input to the module output modification model MO, the module output modification model MO performs post-processing on the output out4 and corrects the output out4. + This is output as an utterance from the dialogue system DS to the user Us. Note that in post-processing by the module output modification model MO, the input does not necessarily need to be changed, and if no change is necessary, the input may be output as is. In other words, "Modify" in post-processing includes "Copy".

[0028] Figure 2 is an explanatory diagram illustrating an example of post-processing by the module output correction model MO. For example, the text sequence "Area:east,Price:cheep" is input to the module output correction model MO as output out2 from Module2. The module output correction model MO then performs post-processing to correct this output out2, resulting in the corrected output out2. + For example, it outputs the text sequence "Area:east,Price:expensive". Also, for example, the text sequence "There are 21.They are in the middle." is input to the module output correction model MO as output out4 from Module4. Then, the module output correction model MO performs post-processing to correct this output out4, and the corrected output out4 + It outputs the text sequence, "There are 21 hotels in the middle of town."

[0029] The parameters of the module output modification model MO are learned through reinforcement learning to maximize the overall task achievement capability of the dialogue system DS. Therefore, introducing the module output modification model MO into the dialogue system DS can significantly improve the overall performance of the dialogue system DS.

[0030] (Method for generating a module output modification model) Next, we will explain the method for generating (learning) the module output correction model MO described above. First, we will explain general methods for optimizing dialogue systems.

[0031] The problem of learning capabilities for multi-turn task-oriented dialogue is formulated as a Markov Decision Process (hereinafter also referred to as "MDP") and optimized by Reinforcement Learning (hereinafter also referred to as "RL"). MDP is defined by a tuple (S, A, P, R, γ). S and A represent all possible dialogue histories and system response texts, respectively. P(s’|s,a) is the transition model that defines the dialogue environment including the user. R(s,a) is the immediate reward function. γ is the discount factor. In each turn t, the policy F (i.e., the dialogue system) samples an action (i.e., the system response) a t ~F(a t |s t ). Until reaching the final state at turn T, the next state s’ t ~P(s’ t |s t ,a t ) and the immediate reward r t =R(s t ,a t ) are obtained. The goal of reinforcement learning is to train F to maximize the value function V, which is the expected value of the cumulative discounted reward, as shown in Equation (1) below.

Equation

[0032] In complex problems such as task-oriented dialogue, it is difficult to directly obtain the policy that maximizes Equation (1). One effective approach is a policy network F parameterized by θ θThis is a policy gradient-based approach that directly improves the process (Sutton, RS; McAllester, D.; Singh, S.; and Mansour, Y. 1999. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Proceedings of Advances in Neural Information Processing Systems, 1057-1063). According to the policy gradient theorem, the gradient is defined by equation (2) below: ψ t The specific definition of advantage varies depending on the implementation of the reinforcement learning algorithm, and may include the total rewards obtained over all turns or an estimated value of the advantage (Schulman, J.; Moritz, P.; Levine, S.; Jordan, M.; and Abbeel, P. 2015. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv preprint arXiv:1506.02438.).

number

[0033] Next, the method for generating the module output modification model MO of this embodiment will be described. In this embodiment, the optimization problem of F is formulated in a way that includes post-processing. The interactive system F has M modules (Module1~Module M It shall be modular, consisting of ). In each turn t, each module m This is the preceding module. (m-1) output out (t,m-1) Enter in (t,m) The result is obtained as follows, and the processing result is output. (t,m) ~Module m (in (t,m) The module outputs the dialogue history as additional input. tYou may also use each module. m Post-processing network PPN m The output is out (t,m) Correct the output. Corrected output out + (t,m) ~PPN m (s t ・in (t,m) ,out (t,m) ) is the subsequent module Module (m+1) This will be the input. In the optimization of the dialogue system F, the module Module m Instead, the post-processing network PPN m They are trained.

[0034] As described above, UniPPN, which is the module output modification model MO of this embodiment, is a single model that performs post-processing of the output from all M modules. That is, UniPPN is a single model that performs post-processing of the output from any module. m The output is out + (t,m) ~UniPPN(s t ・in (t,m) ,out (t,m) , prefix m ) should be corrected to prefix. m The module to be processed is Module m This is an indicator that identifies that. Each of them, (s t ・in (t,m) ,out (t,m) , prefix m ) and out + (t,m) The tokenized sequence x = (x1, ..., x k ) and y=(y1,...,y l The following conditional probability is modeled for ). In equation (3), π θθ is a pre-trained language model parameterized by θ. UniPPN can perform post-processing uniformly across all modules by treating post-processing not only for natural language, such as the output of the NLG module, but also for structured data, such as the output of the NLU and DST modules, as sequence transformation tasks.

number

[0035] (Imitation learning of post-processing) Pre-trained language models are generally trained on web text, and therefore may not have sufficient ability to modify the module's output in task-oriented dialogue systems. Therefore, in this embodiment, when generating UniPPN, supervised fine-tuning is used to modify the input (s t ・in (t,m) ,out (t,m) , prefix m ) and out + (t,m) The form of Model π θ Additional pre-training is performed to teach the system. In a typical RL paradigm, imitation learning (IL), which is performed prior to online RL, uses demonstration data consisting of the behavioral history of human-like experts. However, in the problem setting adopted in this embodiment, there is no demonstration data for post-processing of the output from each module constituting the dialogue system F. Therefore, such demonstration data for post-processing is automatically generated and used for supervised fine-tuning.

[0036] Figure 3 is an explanatory diagram illustrating an example of a method for generating demonstration data for post-processing used in imitation learning. In this method, the dialogue is sampled by repeatedly performing interactions between the dialogue system F and the environment P, and each module at each turn t is sampled. m Input / output history regarding h (t,m) =(st , in (t,m) , out (t,m) ) is generated, and finally H m = {h (t,m) ,..., h (|Hm|,m)} is obtained.

[0037] Next, a demonstration for the correction of the output out (t,m) is created. However, the correct label out (t,m) representing the correction of the output out + (t,m) cannot be automatically generated. Therefore, in this embodiment, under the assumption that the output out m of Module (t,m) is reasonable, the output out (t,m) is regarded as the target output after post - processing, and out - (t,m) randomly sampled from another turn u (which can be the same or different conversations) is used as the negative output to be post - processed. As a result, one demonstration instance d - (t,m) representing the correction from out (t,m) to out (t,m) = {(s t , in (t,m) , out - (t,m) , prefix m ), out (t,m)} is created. This pseudo - data creation process is applied to all samples of H m , and the final demonstration dataset D m = {d (1,m) ,..., d (|Hm|,m)} is obtained.

[0038] When creating the demonstration dataset D m , the following method may be used. That is, if a turn u completely unrelated to the context of turn t is sampled, noise will be introduced, and the model π will not learn to appropriately correct out - (t,m) , but out -(t,m) It may learn to ignore it. Therefore, to ensure that the error is reasonable, sample turns that have a similar context to turn t. t Of the entire history excluding h (t,m) Context s t Extract the top few turns with the highest cosine similarity to the vector representation, and from these extracted turns h u Randomly sample the data. To vectorize the context, a general-purpose embedding model such as E5 can be used (Wang, L.; Yang, N.; Huang, X.; Jiao, B.; Yang, L.; Jiang, D.; Majumder, R.; and Wei, F. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.).

[0039] Also, demonstration dataset D m In creating the output, the following methods may be used. That is, as mentioned above, it is not always necessary to change the output in post-processing, and output that does not cause problems may be copied as is. Therefore, in this embodiment, in order to indicate at the IL stage that changes are not always necessary, the original output out (t,m) This is input to model π. In this case, the target output is the special token "copy". Specifically, the demonstration in this case is d (t,m) ={(s t ・in (t,m) ,out (t,m) , prefix m This results in ),"copy"}. This approach allows us to improve computational efficiency while explicitly learning cases where no modification of the module's output is necessary. Whether each instance becomes a copy instance or not is randomly determined by the hyperparameter copy ratio α.

[0040] The final dataset D is the collection of post-processed data from all M modules. 1:M =[D1,;...;D M Using [ ], θ is updated based on the maximum likelihood estimation method. Below, the parameter optimized by such an IL step will be denoted as φ.

[0041] Next, the model is optimized using reinforcement learning (RL). In the RL phase, the UniPPNπ obtained in the IL step described above is used. φ The system is installed in the dialogue system F. The dialogue system F is then made to interact with the environment P over multiple turns, and based on these experiences, φ is updated using a policy gradient-based approach. In a typical task-oriented dialogue system using online RL, only one policy network (e.g., Policy module) operates per turn and is updated according to equation (2) above. In contrast, this embodiment uses a policy π that operates M times per turn and ultimately outputs the system response as action a. Each of the M actions should have a different gradient based on its respective characteristics, but equation (2) above treats them as having the same contribution. As a result, the reward calculation becomes coarse and learning becomes unstable. Therefore, in this embodiment, as shown in Figure 4, the standard MDP is extended and a module-level MDP (MDP-MO in Figure 4) is adopted, in which the unit of time step is not "a response from the dialogue system in one turn" but "post-processing of one module by the module output modification model". Specifically, the value function to be maximized and π φ The policy gradient is given by equations (4) and (5) below.

number

number

[0042] In the above formula, r (t,m) This is the module in turn t. mThis represents the immediate reward for post-processing of the output. A small negative fixed value is continuously assigned until the end of the interaction. x (t,m) and y (t,m) These are the input texts for UniPPN, respectively. t ・in (t,m) ,out (t,m) , prefix m ) and the tokenized sequence of output text. Equation (5) shows that the gradient can be calculated as the cumulative gradient of M steps.

[0043] ψ (t,m) Generalized advantage estimation can be used in its implementation (Schulman, J.; Moritz, P.; Levine, S.; Jordan, M.; and Abbeel, P. 2015. High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv preprint arXiv:1506.02438.). Specifically, another language model V parameterized by ψ as a critic network. ψ x estimated by (t,m) Value V ψ (x (t,m) Based on this, the advantage estimate is calculated according to equation (6).

number

number

[0044] In the above formula, δ (t,m) V represents the TD residual. The hyperparameter λ controls the trade-off between the actual long-term reward and the estimate. ψ Each module in each turn t mSince it estimates the state value, even with sparse reward settings spanning multiple turns of dialogue, it is possible to estimate the advantage of each module in detail according to its contribution. The advantage of this algorithm is that it eliminates the need for costly manual reward design for each module. ψ It is trained to minimize the mean squared error with respect to cumulative reward, π φ This is optimized using a clipped surrogate objective function via proximity policy optimization (PPO) (Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.).

[0045] (Configuration of information processing device) Next, the configuration of the information processing device 100, which generates the module output modification model MO described above, will be explained. Figure 5 is an explanatory diagram showing the schematic configuration of the information processing device 100. The information processing device 100 is composed of a computer (PC, server, etc.).

[0046] The information processing device 100 comprises a control unit 110, a storage unit 120, a display unit 130, an operation input unit 140, and an interface unit 150. These units are connected to each other via a bus 190 so that they can communicate with one another. The information processing device 100 may also include a speaker as an output means.

[0047] The display unit 130 of the information processing device 100 is configured, for example, as a liquid crystal display, and displays various images and information. The operation input unit 140 is configured, for example, as a keyboard, mouse, buttons, microphone, trackpad, etc., and receives operations and instructions from the administrator. The display unit 130 may also function as the operation input unit 140 by being equipped with a touch panel. The interface unit 150 is configured, for example, as a LAN interface or USB interface, and communicates with other devices via wired or wireless connection.

[0048] The storage unit 120 of the information processing device 100 is composed of, for example, ROM, RAM, a hard disk drive (HDD), and stores various programs and data, and is used as a workspace and temporary storage area for data when executing various programs. For example, the storage unit 120 stores an interactive processing program CP, which is a computer program for executing various processes described later. The interactive processing program CP is provided, for example, stored on a computer-readable recording medium (not shown) such as a CD-ROM, DVD-ROM, or USB memory, or is provided in a state that can be obtained from an external device (for example, a server on the cloud or other terminal device) via the interface unit 150, and is stored in the storage unit 120 in a state that can be operated on the information processing device 100. In addition, the storage unit 120 of the information processing device 100 stores a module output modification model MO.

[0049] The control unit 110 of the information processing device 100 is configured, for example, with a CPU, and controls the operation of the information processing device 100 by executing a computer program read from the storage unit 120. For example, the control unit 110 functions as an interactive processing unit 111 that executes various processes described later by reading and executing an interactive processing program CP from the storage unit 120. The interactive processing unit 111 includes an original model acquisition unit 112, a model acquisition unit 113, and a post-processing execution unit 114.

[0050] (Processing performed by an information processing device) Next, the processing performed by the information processing device 100 of this embodiment will be described. Figure 6 is a flowchart of the processing performed by the information processing device 100. In this embodiment, the information processing device 100 acquires a module output modification model MO and introduces the module output modification model MO to the dialogue system DS, and performs dialogue with the user Us via the dialogue system DS. The processing shown in Figure 6 is started, for example, when the user operates the operation input unit 140 of the information processing device 100 and inputs a start command.

[0051] First, the original model acquisition unit 112 (Figure 5) of the information processing device 100 acquires the original model MOo (S110). The original model MOo is a language model that serves as the backbone of the module output modification model MO.

[0052] Next, the model acquisition unit 113 (Figure 5) of the information processing device 100 performs imitation learning on the original model MOo (S120). As described above, imitation learning is additional pre-training that teaches the model the format of inputs and outputs, and is performed, for example, by supervised fine-tuning.

[0053] Next, the model acquisition unit 113 (Figure 5) of the information processing device 100 acquires a module output modified model MO by performing reinforcement learning on the original model MOo after imitation learning (S130). As described above, in reinforcement learning, the model parameters are optimized so as to maximize the task achievement ability of the entire dialogue system DS. In this embodiment, the module-level MDP described above is employed in reinforcement learning. The acquired module output modified model MO is stored in the storage unit 120.

[0054] The dialogue processing unit 111 (Figure 5) of the information processing device 100 introduces the module output correction model MO into the dialogue system DS and executes dialogue processing by the dialogue system DS (S140). At this time, the post-processing execution unit 114 uses the module output correction model MO to perform post-processing that corrects the output from all modules constituting the dialogue system DS retrospectively.

[0055] (Performance evaluation) We performed a performance evaluation on the aforementioned module output modification model MO (i.e., UniPPN). For the performance evaluation, we used the MultiWOZ dataset (Budzianowski, P.; Wen, T.-H.; Tseng, B.-H.; Casanueva, I.; Ultes, S.; Ramadan, O.; and Gasi´c, M. 2018. MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 5016-5026.), which contains multi-domain, task-oriented dialogues between customers and store clerks regarding travel information. We applied UniPPN to various dialogue systems developed for MultiWOZ and evaluated its task completion capability based on the task success rate.For user simulation, an agenda-based user simulator (Schatzmann, J.; Thomson, B.; Weilhammer, K.; Ye, H.; and Young, S. 2007. Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System. In Proceedings of Human Language Technologies 2007: The Conference) provided by ConvLab-2 (Zhu, Q.; Zhang, Z.; Fang, Y.; Li, X.; Takanobu, R.; Li, J.; Peng, B.; Gao, J.; Zhu, X.; and Huang, M. 2020. ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 142-149.), a task-oriented dialogue system evaluation toolkit, is used (Schatzmann, J.; Thomson, B.; Weilhammer, K.; Ye, H.; and Young, S. 2007. Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System. In Proceedings of Human Language Technologies 2007: The Conference). (From the North American Chapter of the Association for Computational Linguistics, pp. 149-152.)

[0056] Performance evaluations were conducted for both pipeline and end-to-end interaction systems. For the modules constituting the pipeline system, models were selected that were relatively recently proposed, ranked highly in the MultiWOZ benchmark, and had publicly available implementations. The following are the models adopted for each module of the pipeline system.

[0057] (As an NLU module) • BERT NLU (represented as "BERT" in Table 1) A classification model based on BERT (Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171-4186). (Chen, Q.; Zhuo, Z.; and Wang, W. 2019. BERT for Joint Intent Classification and Slot Filling. arXiv preprint arXiv:1902.10909.)

[0058] (As a DST module) • Rule-based DST (referred to as "Rule" in Table 1) ·D3ST :T5 (Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, PJ 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 1-67.) Latest word level DST (Zhao, J.; Gupta, R.; Cao, Y.; Yu, D.; Wang, M.; Lee, H.; Rastogi, A.; Shafran, I.; and Wu, Y. 2022. Description-driven task-oriented dialog modeling. arXiv preprint arXiv:2201.08904.)

[0059] (As a Policy module) • Rule-based policy (represented as "Rule" in Table 1) • PPO Policy (represented as "PPO" in Table 1) A policy fine-tuned using PPO (Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.) · LAVA : Word-level policy (Lubis, N.; Geishauser, C.; Heck, M.; Lin, H.-c.; Moresi, M.; van Niekerk, C.; and Gasic, M. 2020. LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimization. In Proceedings of the 28th International Conference on Computational Linguistics, 465-479.)

[0060] (As an NLG module) • Template-based NLG (referred to as "Template" in Table 1) SC-GPT Module based on GPT-2 (Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9.) (Peng, B.; Zhu, C.; Li, C.; Li, X.; Li, J.; Zeng, M.; and Gao, J. 2020. Few-shot Natural Language Generation for Task- Oriented Dialog. In Findings of the 2021 Conference on Empirical Methods in Natural Language Processing, 172- 182.)

[0061] The end-to-end system employs the following two representation models: PPTOD is a T5-based dialogue model fine-tuned on MultiWOZ. The LLM-based model performs word-level DST and word-level Policy based on Few Shot Learning using a small number of examples obtained from MultiWOZ. As the LLM, GPT-4o mini, provided by the OpenAI API, was used. ·PPTOD(Su, Y.; Shu, L.; Mansimov, E.; Gupta, A.; Cai, D.; Lai, Y.-A.; and Zhang, Y. 2022. Multi-Task Pre-Training for Plug-and- Play Task-Oriented Dialogue System. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 4661-4676.) • LLM-based model (represented as "LLM" in Table 1) (Hudecek, V.; and Dusek, O. 2023. Are Large Language Models All You Need for Task-Oriented Dialogue? In Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue, 216-228.)

[0062] (Evaluation Criteria) In the performance evaluation, each dialogue system engaged in 1024 interactions with a user simulator. Each interaction was set with 1024 different user goals under test. The average score of the 1024 interactions was used as the final score. Inform Recall / Precision / F1 were used as evaluation metrics. These metrics assess whether the system responded appropriately to the information requested by the user during the interaction. Furthermore, Goal Match Rate was used to evaluate whether the conditions of the entity (a specific facility such as a hotel) presented by the system matched the user's goal. Similarly, the conditions of the entity booked by the system were evaluated as Book Rate. The maximum number of turns per interaction was set to 20, and the task was considered successful only if Inform Recall, Goal Match Rate, and Book Rate all reached 1.0 within this maximum number of turns.

[0063] (UniPPN implementation) As the backbone model for UniPPN, we used a medium-sized GPT-2 model with 355M parameters (Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9.).

[0064] (Imitation learning) Post-processing demonstration data D for each dialogue system 1:M To construct this, we sampled 10,000 turns of interaction between the dialogue system and the user simulator. 1:M To sample turns with similar contexts in the construction, we adopted GTE-base(Li, Z.; Zhang, X.; Zhang, Y.; Long, D.; Xie, P.; and Zhang, M. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.) as the embedding model, and used the last three utterances as the context. The copy rate α was set to 0.1 throughout the evaluation.

[0065] (Reinforcement learning) As an axlimeter for the value function, we used a feedforward network that outputs scalar values, adding it to the 124M parameters of GPT-2. For the reward function R(t,m), we set a small negative value of R(t,m) = -0.1 until the dialogue ended, and then set R(T,M) = 2 × M in the final step when the task was achieved. We performed a total of 200 training iterations, sampling 1024 turns as training data in each iteration. The checkpoint of the final iteration was used as test data.

[0066] (Comparative example) The following two methods were used as comparative examples (baselines) for performance evaluation. • Original dialogue system without post-processing • A method of post-processing the output of all modules by combining conventional BinPPN and GenPPN (hereinafter also referred to as "+BinPPN&GenPPN").

[0067] Since BinPPN and GenPPN cannot be trained together, these two types of PPNs were trained separately using RL. Specifically, reinforcement learning was performed using BinPPN to optimize the post-processing of three modules (NLU, DST, Policy). Then, with these three BinPPNs installed in the system and their parameters frozen, GenPPN was applied to NLG and reinforcement learning was performed again. The implementation and training hyperparameters for BinPPN and GenPPN were the same as those reported in the aforementioned Non-Patent Documents 1 and 2. However, the backbone model for GenPPN was Llama 3.1-8B (Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783.).

[0068] (Evaluation results) Table 1 shows the evaluation results when post-processing is applied to all modules of each system. [Table 1]

[0069] In Table 1, Systems 1-5 are pipeline systems, and Systems 6-7 are end-to-end systems. Two cases were evaluated in which post-processing was applied to pipeline systems consisting of multiple modules: the case with BinPPN&GenPPN and the case with UniPPN. BinPPN&GenPPN is SYS LAVA Furthermore, because it could not be applied to end-to-end systems, only UniPPN was applied to these systems for evaluation.

[0070] In pipeline systems, the performance of cases where UniPPN was applied significantly outperformed that of cases where BinPPN & GenPPN were applied. For example, in SYS where the performance improvement from applying BinPPN & GenPPN was limited. RULE yaSYS PPO Even in systems like this, applying UniPPN significantly improved the success rate of task completion. In particular, SYS RULE The system consists of high-performance modules carefully crafted using handwritten rules, and considering its original success rate of around 84 points, it is noteworthy that applying UniPPN improved the success rate to around 91 points. Furthermore, even for systems where BinPPN and GenPPN could not be applied, applying UniPPN significantly improved the success rate.

[0071] The results shown in Table 1 demonstrate that optimizing the output of all modules holistically in post-processing using UniPPN is more effective than optimizing multiple different PPNs individually. Furthermore, it was shown that UniPPN can be applied to any dialogue system, regardless of the learning capabilities of each module, including rule-based and API-based modules.

[0072] Table 2 shows the evaluation results when BinPPN or UniPPN is applied to the NLU, DST, and Policy modules, which are modules to which BinPPN can be applied, for the systems shown in Table 1. The base systems (systems to which PPN is not applied) for systems 11 to 15 in Table 2 are the same as the base systems for systems 1 to 5 in Table 1. [Table 2]

[0073] In all systems shown in Table 2, the performance improvement achieved by UniPPN outweighed that achieved by BinPPN. This performance difference is likely due to the fact that BinPPN's post-processing capabilities are limited to basic binary operations, leaving little room for improvement, while UniPPN can flexibly generate a variety of information.

[0074] Table 3 shows the evaluation results when GenPPN or UniPPN is applied to the NLG for the systems shown in Table 1. The base systems for systems 21 and 22 in Table 3 are the same as the base systems for systems 1 and 4 in Table 1, respectively. [Table 3]

[0075] In all systems shown in Table 3, the performance improvement achieved by UniPPN was equivalent to that achieved by GenPPN. Considering that GenPPN's learning algorithm requires the system's internal DAs (Dialogue Acts) and feedback on whether the user understood those DAs, UniPPN, which only requires the final dialogue result as a reward, is promising.

[0076] (Effects of this embodiment) As described above, the module output modification model MO of this embodiment is a trained model having parameters learned by reinforcement learning. The module output modification model MO is applicable to a task-oriented dialogue system DS. The dialogue system DS is composed of multiple modules, each of which performs processing on input data and outputs output data, and through processing by these multiple modules, it supports the achievement of a specific task in response to a user request. The dialogue system DS has a language understanding module, a state tracking module, a policy module, and a language generation module. The module output modification model MO causes the computer to function to perform post-processing on all modules of the dialogue system DS, thereby retrospectively modifying the output data from the modules. According to the module output modification model MO of this embodiment, a single model can perform post-processing to retrospectively modify the output data from all modules constituting the dialogue system DS, thereby significantly improving the overall performance of the dialogue system DS.

[0077] In the dialogue system DS of this embodiment, the multiple modules are connected in series. According to the module output modification model MO of this embodiment, the performance of the dialogue system DS having such a pipeline structure can be significantly improved.

[0078] In the dialogue system DS of this embodiment, the input and output data of each module are text sequences, and the post-processing performed by the module output modification model MO is a sequence conversion task. Therefore, post-processing can be performed uniformly on the output of all modules constituting the dialogue system DS, and the performance of the dialogue system DS can be significantly improved.

[0079] The reinforcement learning for generating the module output modification model MO in this embodiment employs a module-level Markov decision process that uses the unit of post-processing by each module as the unit of time step in the Markov decision process. Therefore, the learning of the module output modification model MO can be performed stably, and the performance of the dialogue system DS can be significantly improved.

[0080] In the generation of the module output modification model MO of this embodiment, imitation learning is performed to learn the format of the input and output data before reinforcement learning. Therefore, the learning of the module output modification model MO can be performed efficiently. This imitation learning is performed by supervised fine tuning. Therefore, the format of the input and output data can be learned efficiently. This imitation learning is performed using a combination of the module output in a specific turn and the module output randomly sampled from another turn as demonstration data. Therefore, demonstration data used for imitation learning can be generated efficiently. The other turn is selected based on its similarity to the specific turn. Therefore, high-quality demonstration data used for imitation learning can be generated efficiently.

[0081] The post-processing performed by the module output modification model MO of this embodiment includes a process of copying the output data from the module when there is no need to modify it. Therefore, it is possible to avoid modifying the output data from the module even when there is no need to change it, and the overall system performance can be effectively improved.

[0082] (modified version) The technologies disclosed herein are not limited to the embodiments described above and can be modified in various forms without departing from their essence, for example, the following modifications are possible.

[0083] The configuration of the information processing device 100 in the above embodiment is merely an example and can be modified in various ways. Similarly, the processing content in the above embodiment is merely an example and can be modified in various ways. For example, in the above embodiment, the information processing device 100 obtains the module output modification model MO by generating the module output modification model MO itself, but the information processing device 100 may also obtain the module output modification model MO generated by another device via the interface unit 150.

[0084] In the above embodiment, a module-level Markov decision process is employed in reinforcement learning, but a general Markov decision process (where the unit of time step is the system's response in one turn) may also be employed.

[0085] In the above embodiment, imitation learning is performed by supervised fine-tuning, but imitation learning may be performed by other methods. In the above embodiment, imitation learning may not be performed at all.

[0086] In the above embodiment, a module output modification model MO applied to a task-oriented dialogue system DS was described, but the application of the technology disclosed herein is not limited to this. The technology disclosed herein is also applicable to dialogue systems other than task-oriented systems (e.g., casual conversation systems). Furthermore, the technology disclosed herein is also applicable to systems other than dialogue systems (e.g., robotics systems). The application of the technology disclosed herein is not limited to systems in which multiple modules are connected in series, but is applicable to systems in general that consist of multiple modules.

[0087] In the above embodiment, some of the configuration implemented by hardware may be replaced with software, and conversely, some of the configuration implemented by software may be replaced with hardware. [Explanation of Symbols]

[0088] 100: Information processing unit 110: Control unit 111: Dialogue processing unit 112: Original model acquisition unit 113: Model acquisition unit 114: Post-processing execution unit 120: Storage unit 130: Display unit 140: Operation input unit 150: Interface unit 190: Bus CP: Dialogue processing program MO: Module output modification model

Claims

1. A trained model having parameters learned by reinforcement learning, A trained model for a system that consists of multiple modules, each of which performs processing on input data and outputs output data, and which achieves a specific purpose through the processing performed by the multiple modules, wherein the computer is configured to perform post-processing to modify the output data from all of the modules after the fact.

2. A trained model according to claim 1, In the aforementioned system, the multiple modules are connected in series, forming a trained model.

3. A trained model according to claim 1 or claim 2, The input data and the output data are text sequences. The aforementioned post-processing is a sequence transformation task performed on the trained model.

4. A trained model according to claim 1 or claim 2, The reinforcement learning described above is a trained model that employs a module-level Markov decision process, where the unit of post-processing by the module is used as the unit of time step in the Markov decision process.

5. A trained model according to claim 1 or claim 2, A trained model in which, prior to the reinforcement learning described above, imitation learning is performed to learn the format of the input data and the output data.

6. A trained model according to claim 5, The aforementioned imitation learning is performed using a trained model with supervised fine-tuning.

7. A trained model according to claim 6, The aforementioned imitation learning is performed using a trained model that employs a combination of the output of the module in a specific turn and the output of the module randomly sampled from another turn as demonstration data.

8. A trained model according to claim 7, The aforementioned other turn is selected based on its similarity to the aforementioned specific turn, using a trained model.

9. A trained model according to claim 1 or claim 2, The post-processing includes a trained model that copies the output data when it is not necessary to modify the output data from the module.

10. A trained model according to claim 1 or claim 2, The system is a pre-trained model that is a task-oriented dialogue system that supports the achievement of specific tasks in response to user requests, for the purpose described above.

11. A trained model according to claim 10, The system is a trained model having, as modules, a language understanding module, a state tracking module, a policy module, and a language generation module.

12. An information processing device, A model acquisition unit that acquires a trained model having parameters learned by reinforcement learning, A system comprising multiple modules, each performing processing on input data and outputting output data, and which achieves a specific objective through the processing performed by the multiple modules, includes a post-processing execution unit that uses the trained model to perform post-processing on all the modules to retrospectively modify the output data from the modules, An information processing device equipped with the following features.

13. Information processing method, The process involves a computer acquiring a trained model with parameters learned through reinforcement learning, A computer system is composed of multiple modules, each of which performs processing on input data and outputs output data, and which achieves a specific purpose through the processing performed by the multiple modules. The system includes a step of performing post-processing to modify the output data from all of the modules using the trained model. An information processing method comprising:

14. It is a computer program, On the computer, The process of obtaining a trained model with parameters learned through reinforcement learning, A system comprising multiple modules, each performing processing on input data and outputting output data, and which achieves a specific objective through the processing performed by the multiple modules, wherein a post-processing step is performed using the trained model to modify the output data from all the modules after the fact, A computer program that executes something.

Images