Information processing device and information processing method

Abstract

This information processing device calculates Attention between a first output sequence that is a sequence of calculation results obtained in the process of obtaining the output of a first task and a second output sequence that is a sequence of calculation results obtained in the process of obtaining the output of a second task by using a model that has been trained for the first task and the second task. The information processing device combines the first task or the second task and an untrained third task on the basis of Attention and thereby improves the accuracy of generating the third task.

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing device and an information processing method.

[0002] Methods for processing multiple tasks with a single model, such as multi-task, universal, and large-language models, are being actively researched. These models can not only output answers for multiple different tasks for a single input, but also generate results for unlearned tasks (zero shot).

[0003] Vaswani et al., “Attention is all you need,” Proc. NeurIPS, vol. 30, 2017T. Kano et al., “Summarize while Translating: Universal Model with Parallel Decoding for Summarization and Translation” in Proc. ASRU, 2023

[0004] However, conventional techniques have a problem in that the accuracy of the results of generating unlearned tasks is low.

[0005] It is known that the accuracy of the zero-shot generation results is lower than that of the trained task results. Furthermore, due to the constraint of zero-shot, it is difficult to improve the accuracy of the model through training.

[0006] In order to solve the above-mentioned problems and achieve the object, an information processing device is characterized by having a calculation unit that uses a model that has been trained on a first task and a second task to calculate attention between a first output series, which is a series of calculation results obtained in the process of obtaining the output of the first task, and a second output series, which is a series of calculation results obtained in the process of obtaining the output of the second task, and calculates the output of the third task by combining the first task or the second task with an untrained third task based on the attention.

[0007] According to the present invention, it is possible to improve the accuracy of the results of generating unlearned tasks.

[0008] Fig. 1 is a diagram illustrating an example of the configuration of a learning device according to a first embodiment. Fig. 2 is a diagram illustrating an example of the configuration of a model. Fig. 3 is a flowchart illustrating the flow of executing a zero-shot task. Fig. 4 is a diagram illustrating experimental results. Fig. 5 is a diagram illustrating an example of a computer that executes a program.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An information processing apparatus and an information processing method according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0010] The learning device of this embodiment is an example of an information processing device. The learning device learns a model that is based on Transformer and can process multiple tasks. The difference between the existing Transformer and the model of this embodiment will be described later.

[0011] [Configuration of First Embodiment] The configuration of a learning device according to the first embodiment will be described using Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a learning device according to the first embodiment. For example, the learning device 10 is a personal computer, a server device, a smartphone, a tablet terminal, or the like.

[0012] The learning device 10 includes a communication unit 11, a storage unit 12, and a control unit 13. The learning device 10 can perform both inference processing using a model and model learning processing.

[0013] The communication unit 11 performs data communication with other devices via a network, and is, for example, a network interface card (NIC).

[0014] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk. Note that the storage unit 12 may also be a rewritable semiconductor memory such as a random access memory (RAM), flash memory, or non-volatile static random access memory (NVSRAM). The storage unit 12 stores an operating system (OS) and various programs executed by the learning device 10.

[0015] The storage unit 12 stores model information 121. The model information 121 is information related to a model. For example, the model information 121 includes parameters such as weights and biases of a neural network that constitutes the model.

[0016] The control unit 13 controls the entire learning device 10. The control unit 13 is, for example, an electronic circuit such as a central processing unit (CPU), a micro processing unit (MPU), or a graphics processing unit (GPU), or an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 13 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory.

[0017] The control unit 13 functions as various processing units by running various programs. For example, the control unit 13 includes a calculation unit 131 and an update unit 132.

[0018] The calculation unit 131 inputs data necessary for the model constructed from the model information 121 and performs calculations to obtain an output. The update unit 132 updates the parameters of the model based on the output of the model. For example, the update unit 132 updates the parameters using the backpropagation algorithm.

[0019] [Multi-task Learning and Zero-shot Tasks] Multi-task learning and zero-shot tasks will now be described. A general sequence model using deep learning such as Transformer performs learning specialized for one task. In contrast, multi-task learning involves learning multiple tasks with one model. The model of this embodiment particularly uses prompts to output different tasks separately.

[0020] Now, input X∈R I×D , Prompt P M ∈R K×D are sequences of distributed representations. The model f(·) is a set of inputs X and P M Output Y of different tasks M ∈R J×V where I and J are the input and output sequence lengths, M is the number of multitasks, D is the number of dimensions, and V is the dictionary size. Also, the indices of I, J, K, and M are represented as i, j, k, and m. The model f(·) is generated by input X and P m The optimal Y for m is output as shown in equations (1) and (2).

[0021]

[0022]

[0023] Here, Enc(·) and Dec(·) represent the Transformer-type Encoder and Decoder, respectively. The Decoder generates outputs using an autoregressive method, predicting the next output based on its previous output.

[0024] At this time, unknown Prompt P z By giving the model, the model can obtain the output result y z In this embodiment, such a model is called a universal model. Since most parameters are shared between tasks, a universal model can generate an output based on the correlation with previously learned tasks when given a distributed representation of an unknown prompt.

[0025] On the other hand, since the zero-shot task is an untrained task, it is known that the generation accuracy is significantly worse than that of the trained task. This problem can also be understood as follows: Given input X, P M Y generated for M =[y 1 ,… ,y M ] are generated sentences from the same input, but the output accuracy of each task differs depending on whether or not learning has been performed and the difficulty of the problem. Each generated sentence contains output errors, but the locations where errors occur and the results do not match between tasks because the generation processes are independent. As a result, there may be inconsistencies in the content of the generated sentences between tasks.

[0026] Therefore, in this embodiment, the consistency of the output results of each task is improved, thereby improving the output accuracy of the zero-shot task. For example, in this embodiment, the output accuracy of the zero-shot task is improved by incorporating target-cross attention, which will be described later, into the Transformer.

[0027] [Model Configuration] The model configuration will be described using Fig. 2. Fig. 2 is a diagram showing an example of the model configuration. As shown in Fig. 2, model 300 has an Encoder 312 and a Decoder 322. Model 300 also has an Embedding layer 311, an Embedding layer 321, and an Embedding layer 331. Embedding converts input data into a format suitable for Encoder 312 or Decoder 322.

[0028] Here, the model 300 can perform three types of tasks: autoencoder, translation, and summarization, using text written in English or Portuguese as input.

[0029] The model 300 is trained on six tasks: English autoencoding, Portuguese autoencoding, English to Portuguese translation, Portuguese to English translation, English summarization, and Portuguese summarization.

[0030] On the other hand, model 300 does not perform learning on the two tasks of translating from English to Portuguese and summarizing, and translating from Portuguese to English and summarizing, and processes these as zero-shot tasks.

[0031] The language combination handled by the model 300 is not limited to English and Portuguese, but may be any language combination.

[0032] The model 300 includes an EN Linear 341, a Softmax 342, a PT Linear 351, and a Softmax 352. The EN Linear 341 is a linear transformation that operates when the language of the output text is English. The PT Linear 351 is a linear transformation that operates when the language of the output text is Portuguese. The Softmax 342 and the Softmax 352 are Softmax functions.

[0033] The Decoder 322 has K Transformer layers (different from the K that represents the size of the Prompt described above). One layer of the Decoder 322 has a Masked Multi-head Attention 3221 (self-attention mechanism), an Add & Norm 3222, a Source-cross Multi-head Attention 3223, an Add & Norm 3224, a Target-cross Multi-head Attention 3225, an FNN (Feedforward Neural Network) 3226, and an Add & Norm 3227.

[0034] Let m be the index of a first task (e.g., a translation task from English to Portuguese) and n be the index of a second task (e.g., a task to summarize a Portuguese text). Each block in region 322_n is used when task n is performed.

[0035] However, the blocks in area 322_n corresponding to Masked Multi-head Attention 3321, Add & Norm 3322, Source-cross Multi-head Attention 3323, Add & Norm 3324, Target-cross Multi-head Attention 3325, FNN (Feedforward Neural Network) 3326, and Add & Norm 3327 are the same (the parameters are common).

[0036] In this embodiment, the above-mentioned Prompt is extended to 2×K×D is created (a bar just above P). ￣P (hereafter referred to as the extended prompt) is a tensor that combines the prompts of any two tasks. The extended prompts of task m and task n (where m ≠ n) are expressed as [p m ,p n ] is written as follows.

[0037] The Decoder 322 generates y in equation (3) as shown in equation (4) for the input Transcription X and extended prompt P. Dec(·) is the Decoder 322.

[0038]

[0039]

[0040] When executing the Zero Shot task, the decoder 322 simultaneously solves two tasks m and n for the input X. m and A in equation (6) n are distributed representation sequences that combine the prompt and output sequence in the time direction. Equation (7) is a tensor consisting of elements of task, sequence length, and number of dimensions.

[0041]

[0042]

[0043]

[0044] The decoder 322 processes the tensor using Masked Multi-head Attention 3221 (SelfAtt(·)) (Equation (8)). The decoder 322 further processes the tensor using Source-cross Multi-head Attention 3223 (src(·)) (Equation (9)).

[0045]

[0046]

[0047] In this case, the first dimension of the tensor is the task direction (e.g., P m , P n ) the dependency relationship cannot be clearly considered. m and A n In order to consider the relationship between the targets, Target-cross Multi-head Attention 3225 (tgt(·)) is also introduced between targets (Equations (10) and (11)).

[0048]

[0049]

[0050] The Target-cross Multi-head Attention 3225 calculates Attention using itself as the Query and different task (subtask) sequences as Key and Value. This allows the Target-cross Multi-head Attention 3225 to generate output that takes the subtask sequence into consideration, similar to the Source-cross Multi-head Attention 3323.

[0051] In this way, the calculation unit 131 uses a model that has learned the first task and the second task to calculate attention between a first output sequence, which is a sequence of calculation results obtained in the process of obtaining the output of the first task, and a second output sequence, which is a sequence of calculation results obtained in the process of obtaining the output of the second task. Based on the attention, the calculation unit 131 combines the first task or the second task with an unlearned third task to calculate the output of the third task.

[0052] Furthermore, the calculation unit 131 calculates the output of the third task using a model that is a Transformer having a function for calculating attention between tasks in the Transformer layer.

[0053] For example, the calculation unit 131 calculates the output of a third task using a model that has been trained on a first task, which is translation from a first language to a second language, and a second task, which is summarizing text in the second language.

[0054] The output of the Target-cross Multi-head Attention 3225 is processed by the FFN 3226 (Equations (12) and (13)), and finally processed by the EN Linear 341 and Softmax 342 (Equation (14)).

[0055]

[0056]

[0057]

[0058] Here, task m is a translation task from English to Portuguese, and task n is a task of summarizing Portuguese text. Model 300 has been trained on tasks m and n. The zero-shot task is to output Portuguese text that summarizes the input English text.

[0059] In this case, Source-cross Multi-head Attention extracts information about task m and highly related English word information based on an inner product calculation, and adds the extracted information to the tensor of the generated sequence of task m. Furthermore, Target-cross Multi-head Attention 3325 reflects information for identifying words in the summary of the Portuguese translation that are highly related to a certain English word in the tensor based on information about task n.

[0060] [Processing Flow of the First Embodiment] Figure 3 is a flowchart showing the flow when executing a zero-shot task. When learning each task, the model is updated based on the output text. The initial value of k is set to 0.

[0061] As shown in FIG. 3, first, the learning device 10 encodes the transcription using the encoder 312 (step S101).

[0062] Next, the learning device 10 defines a distributed representation sequence that combines the prompt and output sequence in the time direction for each of the two tasks (step S102).The learning device 10 also defines a tensor that combines the distributed representation sequences of the two tasks and is composed of elements of the task, sequence length, and number of dimensions (step S103).

[0063] Next, the learning device 10 processes the tensor using Self Attention (Masked Multi-head Attention 3221) and Source-cross Attention (Source-cross Multi-head Attention 3323) by the Decoder 322 (step S104).

[0064] Here, the learning device 10 further processes the tensor using target-cross attention using the decoder 322 (step S105).

[0065] The learning device 10 increments k by 1 (step S106), and if k is not equal to or greater than K (the total number of transfer layers) (step S107, No), the learning device 10 returns to step S104 and repeats the process. If k is equal to or greater than K (step S107, Yes), the learning device 10 outputs text for one of the tasks based on the tensor (step S108). For example, when outputting Portuguese text, the learning device 10 performs processing using PT Linear 351 and Softmax 352.

[0066] [Effects of the First Embodiment] As described above, the calculation unit 131 uses a model that has learned the first task and the second task to calculate attention between a first output sequence, which is a sequence of calculation results obtained in the process of obtaining the output of the first task, and a second output sequence, which is a sequence of calculation results obtained in the process of obtaining the output of the second task. Based on the attention, the calculation unit 131 combines the first task or the second task with an unlearned third task to calculate the output of the third task.

[0067] In this way, the learning device 10 generates a highly consistent multi-task by enhancing the interrelationship between the first task and the second task based on attention.The learning device 10 then generates an unknown zero-shot task (third task) by combining it with the known first task or second task, thereby improving the accuracy of zero-shot task generation.

[0068] [Experimental Results] Using a model trained on a combination of the three tasks of translation, summarization, and autoencoding, we conducted a comparative experiment in which the combined task of translation and summarization was set as a zero-shot task. In the zero-shot task, the model simultaneously translates and summarizes the input document.

[0069] In conventional techniques (e.g., existing Transformers), each of the three tasks is trained independently, and the correlation between output sequences is not explicitly considered in the Decoder. In this embodiment, by introducing tgt-cross attention, sequences from different tasks can be mutually referenced, improving the consistency of output results.

[0070] Figure 4 shows the results of the experiment. In the experiment, a comparison experiment was conducted on a zero-shot task of translating and summarizing between Portuguese and English using How2 data, using a universal model constructed as in the embodiment. The evaluation method used was METEOR and ROUGE, scales known in language processing for measuring the degree of agreement between sentences.

[0071] "1 Pivot" in Figure 4 is a method of building a translation model and a summarization model separately, translating, and then summarizing. "3 Universal" is a conventional technology. "4 + target-cross" and "5 ++ parallel-beam" are embodiments. "5 ++ parallel-beam" differs from "4 + target-cross" in that it performs beam-search when generating output. As shown in Figure 4, this embodiment can solve zero-shot tasks with higher accuracy than conventional technologies.

[0072] [Program] In one embodiment, the study device 10 can be implemented by installing a program that executes the above-described processes as package software or online software on a desired computer. For example, by executing the above-described program on an information processing device, the information processing device can function as the study device 10. The information processing device referred to here includes desktop and notebook personal computers. Other information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

[0073] The learning device 10 can also be implemented as a server device that provides services related to the above processing to a client terminal used by a user. The server device may be implemented as a web server or as a cloud that provides services related to the above processing through outsourcing.

[0074] 5 is a diagram showing an example of a computer that executes a program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0075] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0076] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the learning device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, on the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the learning device 10 is stored on the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0077] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0078] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0079] REFERENCE SIGNS LIST 10 Learning device 11 Communication unit 12 Storage unit 13 Control unit 121 Model information 131 Calculation unit 132 Update unit

Claims

1. An information processing device having a calculation unit that uses a model that has been trained on a first task and a second task to calculate attention between a first output sequence, which is a sequence of calculation results obtained in the process of obtaining the output of the first task, and a second output sequence, which is a sequence of calculation results obtained in the process of obtaining the output of the second task, and calculates the output of the third task by combining the first task or the second task with an untrained third task based on the attention.

2. The information processing device according to claim 1, characterized in that the calculation unit calculates the output of the third task using the model which is a Transformer having a function for calculating attention between tasks in a Transformer layer.

3. The information processing device according to claim 1, characterized in that the model has been trained on the first task, which is translation from a first language to a second language, and the second task, which is summarizing text in the second language, and the generation process of the first task or the generation process of the second task is combined with the generation process of the third task, which has not yet been trained.

4. An information processing method executed by an information processing device, comprising: a step of calculating attention between a first output series, which is a series of calculation results obtained in the process of obtaining the output of a first task, and a second output series, which is a series of calculation results obtained in the process of obtaining the output of the second task, using a model that has been trained on a first task and a second task; and a step of calculating the output of the third task by combining the first task or the second task with an untrained third task based on the attention.

Images