Learning device, learning method, learning program, inference device, inference method, and inference program

Abstract

The filtering unit (22) calculates the score of each of a plurality of elements included in the teacher source data (311) in the first teacher data (31) including the teacher source data (311) and the teacher destination data (312) using the attention mechanism in the first model (41). The filtering unit (22) filters the plurality of elements included in the teacher source data (311) based on the score of each of the plurality of elements to generate teacher source filtered data (321). The filtering data learning unit (23) receives as input second teacher data (32) including the teacher source filtered data (321) and the teacher destination data (312), and learns parameters of a second model (42) that converts the teacher source filtered data (321) into the teacher destination data (312).

Description

[Technical Field]

[0001] The present disclosure relates to a technique using a model with an attention mechanism. [Background technology]

[0002] In machine translation and other sequence conversion tasks using machine learning techniques, neural network models consisting of an encoder and a decoder are used. Hereinafter, a neural network model consisting of an encoder and a decoder is referred to as an encoder-decoder model. It is known that introducing an attention mechanism into an encoder-decoder model can significantly improve accuracy. In the case of machine translation, an attention mechanism is a data-driven mechanism that determines which word information in a target language sentence to use in the decoder.

[0003] Currently, the Transformer is a mainstream encoder-decoder model that combines attention mechanisms and fully connected layers to parallelize the encoder. The Transformer is a model that constructs an encoder and decoder by stacking a combination of a multi-head attention mechanism or a masked multi-head attention mechanism and a fully connected layer. Hereinafter, the combination of the multi-head attention mechanism or the masked multi-head attention mechanism and the fully connected layer will be considered as one layer, and will be referred to as the Transformer layer.

[0004] The attention mechanism of the Transformer calculates the combinations between all elements in the input sequence. If the number of elements in the input sequence is n, then the combinations between all elements are n 2 Therefore, the attention mechanism of the Transformer requires GPU memory usage of O(n 2 ) GPU stands for Graphics Processing Unit. Therefore, as the input sequence gets longer, the computational cost required for processing increases quadratically.

[0005] Patent Document 1 describes a method of dividing a long text into topic units to generate blocked text, summarizing the text for each block, and generating a summary result for each topic. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2023-034235 Summary of the Invention [Problem to be solved by the invention]

[0007] The technology described in Patent Document 1 still requires increased computational costs when the length of the blocked text corresponding to a specific topic increases. When the blocked text is long, it is possible to shorten the input sequence by further dividing it into smaller blocks, thereby reducing computational costs. However, because each blocked text is processed independently, further dividing it into smaller blocks can result in a loss of context, which can degrade the quality of the generated text. In other words, it is not possible to appropriately adjust the length of the input sequence while maintaining the quality of the generated result. The present disclosure aims to make it possible to appropriately adjust the length of an input sequence while maintaining the quality of the generated result. [Means for solving the problem]

[0008] The learning device according to the present disclosure includes: a filtering unit that calculates a score for each of a plurality of elements included in the teacher source data in first teacher data including teacher source data, which is the sequence data of the conversion source, and teacher destination data, which is the sequence data of the conversion destination, using an attention mechanism in a first model that infers sequence data of the conversion source from the sequence data of the conversion source, and filters the plurality of elements included in the teacher source data based on the scores of each of the plurality of elements to generate teacher source filtered data; a filtering data learning unit that receives as input second teacher data including the teacher-source filtered data and the teacher-destination data generated by the filtering unit and learns parameters of a second model that converts the teacher-source filtered data into the teacher-destination data; Equipped with. [Effects of the Invention]

[0009] In this disclosure, an attention mechanism is used to calculate the scores of each of the multiple elements included in the source sequence data, and then the multiple elements are filtered. Because filtering is performed using the scores calculated using the attention mechanism, it is possible to appropriately adjust the length of the source sequence data (the input sequence) while maintaining the quality of the generated results. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a hardware configuration diagram of a learning and inference device 10 according to a first embodiment. [Figure 2] FIG. 1 is a functional configuration diagram of a learning and inference device 10 according to a first embodiment. [Figure 3] FIG. 2 is an explanatory diagram of an encoder-decoder model that is a target of the first embodiment. [Figure 4] 10 is a flowchart of a learning phase process according to the first embodiment. [Figure 5] FIG. 3 is an explanatory diagram of a filtering process according to the first embodiment. [Figure 6] 10 is a flowchart of a filtering process according to the first embodiment. [Figure 7] FIG. 1 is an explanatory diagram of normal decoding according to the first embodiment. [Figure 8] FIG. 2 is an explanatory diagram of Teacher Forcing Decoding according to the first embodiment. [Figure 9] FIG. 4 is an explanatory diagram of a filtering execution process according to the first embodiment. [Figure 10] 4 is a flowchart of processing in an inference phase according to the first embodiment. [Figure 11]FIG. 10 is a functional configuration diagram of a learning device 101 according to a first modified example. [Figure 12] FIG. 10 is a functional configuration diagram of an inference device 102 according to a first modified example. [Figure 13] FIG. 10 is a configuration diagram of a learning and inference device 10 according to a second embodiment. [Figure 14] 10 is a flowchart of a filtering process according to the second embodiment. [Figure 15] FIG. 10 is a configuration diagram of a learning and inference device 10 according to a third embodiment. [Figure 16] 11 is a flowchart of a filtering process according to the third embodiment. [Figure 17] FIG. 11 is an explanatory diagram of a score adjustment process according to the third embodiment. [Figure 18] FIG. 10 is an explanatory diagram of a filtering data learning process according to the fourth embodiment. [Figure 19] FIG. 13 is a configuration diagram of a learning and inference device 10 according to a sixth embodiment. [Figure 20] 13 is a flowchart of a filtering process according to the sixth embodiment. [Figure 21] FIG. 20 is an explanatory diagram of a filtering process according to the sixth embodiment. [Figure 22] 13 is a flowchart of a filtering process according to the seventh embodiment. [Figure 23] FIG. 20 is an explanatory diagram of a filtering process according to the seventh embodiment. [Figure 24] FIG. 13 is a configuration diagram of a learning and inference device 10 according to an eighth embodiment. [Figure 25] 13 is a flowchart of a filtering process according to the eighth embodiment. [Figure 26] FIG. 20 is an explanatory diagram of a filtering execution process according to the eighth embodiment. [Figure 27] FIG. 20 is an explanatory diagram of a filtering correction process according to the eighth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiment 1 ***Configuration Description*** The hardware configuration of a learning and inference device 10 according to the first embodiment will be described with reference to FIG. The learning and reasoning device 10 is a computer. The learning and inference device 10 includes the following hardware components: a processor 11, a memory 12, an input device 13, a display device 14, an external interface 15, and a communication interface 16. The processor 11 is connected to other hardware components via signal lines and controls these other hardware components.

[0012] The processor 11 is an IC that performs processing. IC stands for Integrated Circuit. Specific examples of the processor 11 include a CPU, a DSP, and a GPU. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. GPU stands for Graphics Processing Unit.

[0013] The memory 12 is a storage device that stores data. Specific examples of the memory 12 include an HDD, an SSD, a RAM, and a ROM. HDD stands for Hard Disk Drive. SSD stands for Solid State Drive. RAM stands for Random Access Memory. ROM stands for Read Only Memory.

[0014] The input device 13 is a device for inputting information. Specific examples of the input device 13 include a keyboard, a mouse, and a touch panel. The display device 14 is a device for displaying information. Specific example of the display device 14 is an LCD. LCD is an abbreviation for Liquid Crystal Display. The learning and inference device 10 does not necessarily have to include at least one of the input device 13 and the display device 14.

[0015] The external interface 15 is an interface with an external device such as a recording medium 17. The recording medium 17 is a device for storing data. The processor 11 can read data stored in the recording medium 17 and write data to the recording medium 17 via the external interface 15. Specific examples of the recording medium 17 include a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, and a DVD. SD stands for Secure Digital. DVD stands for Digital Versatile Disk.

[0016] The communication interface 16 is an interface for connecting to a communication network. A specific example of the communication interface 14 is an Ethernet (registered trademark) port.

[0017] The functional configuration of the learning and inference device 10 according to the first embodiment will be described with reference to FIG. The learning and inference device 10 includes, as functional components, a normal data learning unit 21, a filtering unit 22, a filtered data learning unit 23, and a filtered data inference unit 24. The filtering unit 22 includes a score calculation unit 221 and a filtering execution unit 222. The functions of the functional components of the learning and inference device 10 are realized by software.

[0018] The memory 12 stores a program that realizes the function of each functional component of the learning inference device 10. This program is read by the processor 11 and executed by the processor 11. In this way, the function of each functional component of the learning inference device 10 is realized. The program that realizes the function of each functional component of the learning inference device 10 may be stored in the recording medium 17. In this case, the processor 11 reads and executes the program from the recording medium 17 via the external interface 15. The program that realizes the function of each functional component of the learning inference device 10 may also be stored in an external server or the like. In this case, the processor 11 reads and executes the program from the server or the like via the communication interface 16.

[0019] The memory 12 stores data such as first teacher data 31, second teacher data 32, a first model 41, and a second model 42. These data may be stored not in the memory 12 but in the recording medium 17 or an external server, etc.

[0020] 1 shows only one processor 11. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that realize the respective functions.

[0021] ***Explanation of Operation*** The operation of the learning and inference device 10 according to the first embodiment will be described with reference to FIGS. The operation of the learning and inference device 10 includes a learning phase and an inference phase. The operation procedure of the learning phase of the learning inference device 10 according to embodiment 1 corresponds to the learning method according to embodiment 1. Furthermore, the program that realizes the operation of the learning phase of the learning inference device 10 according to embodiment 1 corresponds to the learning program according to embodiment 1. The operation procedure of the inference phase of the learning inference device 10 according to embodiment 1 corresponds to the inference method according to embodiment 1. Furthermore, the program that realizes the operation of the inference phase of the learning inference device 10 according to embodiment 1 corresponds to the inference program according to embodiment 1.

[0022] The encoder / decoder model to be used in the first embodiment will be described with reference to FIG. The encoder-decoder model targeted in the first embodiment is a Transformer, which is an encoder-decoder model that combines an attention mechanism and a fully connected layer to parallelize the encoder. The Transformer configures the encoder and decoder by stacking a combination of a multi-head attention mechanism or a masked multi-head attention mechanism and a fully connected layer.

[0023] The process of the learning phase according to the first embodiment will be described with reference to FIG. (Step S11: Normal data learning process) The normal data learning unit 21 learns parameters of the first model 41 using the first teacher data 31 as input. The first model 41 is an encoder-decoder model that infers destination sequence data from source sequence data. Here, the first model 41 is a Transformer. As a Transformer, the first model 41 has an attention mechanism. The parameters of the first model 41 to be learned are the parameters to be learned of the encoder-decoder model. The first teacher data 31 is any sequence data that can be input and output by the encoder-decoder model. The first teacher data 31 has a data structure including teacher source data 311, which is sequence data to be converted, and teacher destination data 312, which is sequence data to be converted. The first training data 31 is, for example, data for machine translation. In this case, the training source data 311, which is the sequence data from which the conversion is performed, is a sequence of words in the source language. A word sequence is a sentence or a phrase. The training target data 312, which is the sequence data after conversion, is the result of converting the source language into the target language. In other words, the training target data 312 is a sequence of words in the target language that expresses the same meaning as the sequence of words in the source language.

[0024] (Step S12: Filtering process) The filtering unit 22 filters the teacher source data 311 included in the first teacher data 31 using the parameters of the first model 41 obtained by learning in step S11. Filtering the teacher source data 311 means removing some of the multiple elements included in the teacher source data 311. If the first teacher data 31 is data for machine translation, the multiple elements included in the teacher source data 311 are the multiple words that make up the teacher source data 311. In this way, the filtering unit 22 generates teacher source filtered data 321 in which some of the elements have been removed from the teacher source data 311. It is assumed that each square shown in Fig. 5 is an element included in the teacher source data 311. The filtering unit 22 removes some elements marked with an x ​​from among the elements included in the teacher source data 311. In this way, the filtering unit 22 generates teacher source filtered data 321 from the teacher source data 311. In Fig. 5, 50% of the elements included in the teacher source data 311 are removed, and the teacher source filtered data 321 is generated. Then, the filtering unit 22 generates second teacher data 32 including the teacher source filtering data 321 and the teacher destination data 312.

[0025] (Step S13: Filtering data learning process) The filtering data learning unit 23 learns the parameters of the second model 42 using the second teacher data 32 generated in step S12 as input. The second model 42 is an encoder-decoder model that infers destination sequence data from source sequence data. Here, the second model 42 is a Transformer. The parameters of the second model 42 to be learned are the parameters to be learned of the encoder-decoder model. Any optimization method may be used to learn the parameters. For example, Adam may be used as the optimization method. Adam is an abbreviation for ADaptive Moment. The filtering data learning unit 23 writes the parameters of the second model 42 obtained by learning into the memory 12. Alternatively, the filtering data learning unit 23 may output the parameters to the recording medium 17, an external server, or the display device 14.

[0026] The filtering process (step S12 in FIG. 4) according to the first embodiment will be described with reference to FIG. (Step S21: Score calculation process) The score calculation unit 221 uses the attention mechanism in the first model 41 to calculate the score of each of the multiple elements included in the teacher source data 311. Specifically, the score calculation unit 221 calculates the score using the attention mechanism of the first model 41. d,l,h,i The score of each of the multiple elements is calculated using the above. Attention is cross-attention calculated by the multi-head attention mechanism of the decoder. That is, the score calculation unit 221 calculates the attention for each of the multiple elements in the input sequence of the encoder as a score, which is calculated when generating the output sequence of the decoder. In other words, the score calculation unit 221 calculates the internal state of the first model 41 for each of the multiple elements in the input sequence of the encoder as a score. Here, the input sequence of the encoder is the teacher source data 311.

[0027] Attention d,l,h,i In the above, d, l, h, and i respectively represent the following: d represents the number of elements in the decoder output sequence; l represents the number of layers in the Transformer layer; h represents the number of heads in the multi-head attention mechanism; and i represents the number of elements in the encoder input sequence. The score calculation unit 221 calculates the attention scores for d, l, and h for each i. d,l,h,i The score S for element i is calculated by calculating the maximum value of i Calculate S i =MaxAttention(i)=max d,l,h (Attention d,l,h,i) Here, the score calculation unit 221 calculates the score using the Attention d,l,h,i Although the maximum value of is calculated in the above example, a statistical value other than the maximum value may be calculated. Furthermore, the score calculation unit 221 may calculate the score using all d, l, and h, or may calculate the score using only specific d, l, and h. Furthermore, the score calculation unit 221 may obtain multiple scores by generating output sequences of multiple decoders using general decoding such as beam search, extract the maximum value from the multiple scores, and calculate the final score. Note that instead of the maximum value, a score extracted from the multiple scores using another method may also be used as the final score. Furthermore, the score calculation unit 221 may separately prepare a sequence labeling model using scores as teacher signals, and estimate the scores of each of a plurality of elements in the input sequence of the encoder using the sequence labeling model.

[0028] In the above description, attention is assumed to be cross-attention calculated by a multi-head attention mechanism in the decoder. However, attention may be self-attention calculated by a multi-head attention mechanism in the encoder. Furthermore, attention may be a combination of cross-attention and self-attention. Cross-attention and self-attention can be combined by calculating the sum or average of cross-attention and self-attention.

[0029] When calculating Attention, the score calculation unit 221 may use Teacher Forcing Decoding if the target output sequence is available. When calculating Attention, the score calculation unit 221 may use normal decoding if the target output sequence is not available. As shown in Figure 7, normal decoding uses the model's prediction as input for the next prediction, which is the element that the model output the previous time. In Figure 7, first BOS is given to the decoder to obtain output 1. BOS stands for Begin Of Sentence. Next, BOS and output 1 are given to the decoder to obtain output 2. Then, BOS, output 1, and output 2 are given to the decoder to obtain EOS. EOS stands for End Of Sentence. In contrast, Teacher Forcing Decoding provides the target output sequence directly to the model, as shown in Figure 8. In other words, Teacher Forcing Decoding, unlike normal decoding, does not use the results predicted by the model. In Figure 8, the target output sequence is given as BOS + correct answer 1 + correct answer 2 + EOS. First, BOS is given to the decoder to obtain output 1. Next, BOS and correct answer 1 are given to the decoder to obtain output 2. Then, BOS, correct answer 1, and correct answer 2 are given to the decoder to obtain EOS.

[0030] (Step S22: Filtering execution process) The filtering execution unit 222 generates teacher source filtered data 321 by filtering the elements included in the teacher source data 311 based on the score calculated in step S21. Specifically, as shown in Fig. 9, the filtering execution unit 222 generates teacher source filtering data 321 by excluding elements with the lowest scores and a standard ratio from among the elements included in the teacher source data 311. In Fig. 9, the teacher source data 311 includes four elements, and the standard ratio is 50%. Therefore, the filtering execution unit 222 generates teacher source filtering data 321 by excluding two elements with the lowest scores from among the four elements. As a result, input 2 and input 4, which have low scores, are excluded from input 1 to input 4 included in the teacher source data 311, and teacher source filtering data 321 including input 1 and input 3 is generated.

[0031] The filtering execution unit 222 may combine data obtained by filtering using the above-described method with data obtained by filtering the teacher source data 311 using another method to generate the teacher source filtered data 321. A specific example of filtering using another method is to use a maximum value, an average value, a median value, or the like to remove elements with a standard ratio of high, low, or middle scores from the teacher source data 311. Another specific example of filtering using another method is to remove elements with a standard ratio at the beginning, end, or center of the teacher source data 311 from the teacher source data 311. Furthermore, the filtering execution unit 222 may combine data obtained by filtering the teacher source data 311 at different rates to obtain the teacher source filtered data 321. For example, the filtering execution unit 222 may combine data obtained by removing 10% of the teacher source data 311, data obtained by removing 30% of the teacher source data 311, and data obtained by removing 50% of the teacher source data 311 to obtain the teacher source filtered data 321.

[0032] The processing of the inference phase according to the first embodiment will be described with reference to FIG. (Step S31: Filtering process) The filtering unit 22 filters the input data 33 using the parameters of the first model 41 obtained by the learning in step S11. The input data 33 is any sequence data that can be input and output by the encoder-decoder model. In this way, the filtering unit 22 generates input filtered data. The process of filtering the input data 33 is the same as the process of filtering the teacher source data 311 in step S12 of Fig. 4. In other words, the input data 33 is filtered to generate the input filtered data by a process similar to the process of filtering the teacher source data 311 to generate the teacher source filtered data 321.

[0033] The filtering unit 22 may change the filtering ratio when filtering the teacher source data 311 in step S12 of Fig. 4 and when filtering the input data 33 here. For example, the filtering unit 22 performs 50% filtering when filtering the teacher source data 311 in step S12 of Fig. 4, and performs 30% filtering when filtering the input data 33 here.

[0034] (Step S32: Filtering data inference process) The filtering data inference unit 24 infers the output data 34 from the input filtering data generated in step S31 using the second model 42. Using the second model 42 means using the parameters of the second model 42 generated in the learning phase. The filtering data inference unit 24 writes the inferred output data 34 to the memory 12. Alternatively, the filtering data inference unit 24 may output the output data 34 to the recording medium 17 or an external server or display device 14.

[0035] ***Effects of the First Embodiment*** As described above, the learning and inference device 10 according to the first embodiment calculates the score of each of the elements included in the source sequence data using an attention mechanism, and filters the elements. The score calculated using the attention mechanism is higher for elements with a higher contribution. Therefore, by filtering using the score calculated using the attention mechanism, it becomes possible to appropriately adjust the length of the source sequence data while maintaining the quality of the generated results. The source sequence data refers to the teacher source data 311 and the input data 33.

[0036] Since the length of the source sequence data can be shortened, the computational cost can be reduced. In other words, the computational cost required to learn the parameters of the second model 42 can be lower than the computational cost required to learn the parameters of the first model 41. Furthermore, by inferring the output data 34 from the input filtered data, the computational cost can be lower than that required to infer the output data 34 from the input data 33.

[0037] Here, the parameters of the first model 41 are learned using unfiltered first teacher data 31 as input. Therefore, the calculation cost is high and a high-performance computer environment is required. In contrast, the parameters of the second model 42 are learned using filtered second teacher data 32 as input. Therefore, the calculation cost is low and a high-performance computer environment is not required. There are cases where processing using an encoder-decoder model is deployed horizontally across multiple businesses or multiple systems. In this case, a high-performance computing environment for the first model 41 may be centralized in one location, and a low-performance computing environment for the second model 42 may be prepared at the deployment destination. Once the first model 41 is created, the normal data learning process in step S11 in Fig. 4 is not necessary when learning the parameters of the second model 42. In other words, once the first model 41 is created, when learning the parameters of the second model 42, step S11 in Fig. 4 can be skipped and the process can start from step S12. Therefore, the first model 41 can be prepared in a high-performance computer environment, and the parameters of the second model 42 for the deployment destination can be learned in a low-performance computer environment as the deployment destination.

[0038] ***Other Configurations*** <Variation 1> In the first embodiment, an example has been described in which one learning and inference device 10 has both a function to perform the learning phase and a function to perform the inference phase. However, the learning and inference device 10 may be configured to have only the function to perform the learning phase. A learning and inference device 10 configured to have only the function to perform the learning phase is called a learning device 101. Furthermore, the learning and inference device 10 may be configured to have only the function to perform the inference phase. A learning and inference device 10 configured to have only the function to perform the inference phase is called an inference device 102. Specifically, as shown in Fig. 11, the learning device 101 does not include the filtering data inference unit 24 among the functional components of the learning and inference device 10, but includes a normal data learning unit 21, a filtering unit 22, and a filtering data learning unit 23. Also, as shown in Fig. 12, the inference device 102 does not include the normal data learning unit 21 and the filtering data learning unit 23 among the functional components of the learning and inference device 10, but includes the filtering unit 22 and the filtering data inference unit 24.

[0039] If the first model 41 can be acquired from an external source, the learning device 101 does not need to include the normal data learning unit 21.

[0040] <Variation 2> In the first embodiment, the first teacher data 31 is specifically data for machine translation. However, the first teacher data 31 is not limited to this and may be other data as long as it is sequence data. Note that the second teacher data 32 is data obtained by filtering the teacher source data 311 of the first teacher data 31, and is therefore the same type of data as the first teacher data 31. Furthermore, the set of input data 33 and output data 34 is also the same type of data as the first teacher data 31.

[0041] (Example 1) The first training data 31 is data for natural language processing. The training data 311 is a sequence of words in a specific language, such as a sentence or phrase. The training data 312 is a summary result in a specific language, that is, a sequence consisting of fewer words than the training data 311 but retaining the essential meaning of the training data 311. (Example 2) The first training data 31 is data for natural language processing. The training source data 311 is a sequence of words that mean a question. The training destination data 312 is a sequence of words that mean an answer to the question. (Example 3) The first training data 31 is data for speech recognition. The training source data 311 is a sequence of speech data representing human oral utterances. The training target data 312 is a sequence of phonemes, features, or words representing the content of the utterance. (Example 4) The first training data 31 is data for image processing. The training source data 311 is an image, that is, a sequence of colors, brightness, etc. contained in the image. The training destination data 312 is a sequence of text that explains the image. (Example 5) The first teacher data 31 is data for detecting an anomaly. The teacher source data 311 is a sequence of data acquired by a specific sensor. The teacher destination data 312 is a sequence of text that indicates normality or abnormality. (Example 6) The first teacher data 31 is data for predicting an anomaly. The teacher source data 311 is a sequence of data acquired by a specific sensor. The teacher destination data 312 is a sequence of text that indicates the possibility of a future anomaly occurring. (Example 7) The first teacher data 31 is data for demand forecasting. The teacher source data 311 is a sequence of data related to product sales volume in an arbitrary period. The teacher destination data 312 is a sequence of text representing future product demand.

[0042] <Variation 3> The encoder-decoder model targeted in the first embodiment is a Transformer. Examples of the Transformer include BERT, GPT, and T5. BERT is described in Devlin, J., Chang, MW, Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. GPT is described in: Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. T5 is described in the paper: Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, PJ (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1), 5485-5551.

[0043] <Variation 4> In the first embodiment, each functional component is realized by software. However, as a fourth modification, each functional component may be realized by hardware. The following describes the differences between the first embodiment and the fourth modification.

[0044] When each functional component is realized by hardware, the learning and inference device 10 includes an electronic circuit instead of the processor 11 and the memory 12. The electronic circuit is a dedicated circuit for realizing the functions of each functional component, the memory 12, and the storage 13.

[0045] Possible electronic circuits include single circuits, composite circuits, programmed processors, parallel programmed processors, logic ICs, GAs, ASICs, and FPGAs. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be realized by one electronic circuit, or each functional component may be realized by distributing it among a plurality of electronic circuits.

[0046] <Variation 5> As a fifth modification, some of the functional components may be realized by hardware, and other functional components may be realized by software.

[0047] The processor 11, memory 12, and electronic circuitry are collectively referred to as a processing circuit. In other words, the functions of the functional components are realized by the processing circuit.

[0048] Embodiment 2 The second embodiment differs from the first embodiment in that, when the scores are sorted according to the order of elements included in the input sequence, the filtering method is adjusted so that elements close to elements with large changes in score are less likely to be removed. In the second embodiment, this difference will be explained, and explanations of the same points will be omitted.

[0049] ***Configuration Description*** The configuration of a learning and inference device 10 according to the second embodiment will be described with reference to FIG. 2 in that the learning inference device 10 includes a filtering adjustment unit 223 as a functional component. The function of the filtering adjustment unit 223 is realized by software or hardware, similar to the other functional components.

[0050] ***Explanation of Operation*** The filtering process according to the second embodiment will be described with reference to FIG. Here, the filtering process of step S12 in Fig. 4 will be described. However, the same applies to the filtering process of step S31 in Fig. 10. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate input filtered data, which is different.

[0051] The process of step S41 is the same as the process of step S21 in FIG.

[0052] (Step S42: Filtering adjustment process) The filtering adjustment unit 223 adjusts the filtering method so that elements close to elements with large score changes are less likely to be removed when the scores calculated in step S41 are arranged in accordance with the order of the multiple elements included in the teacher source data 311. In other words, the filtering adjustment unit 223 adjusts the filtering method so that elements before and after an element with a steep score change are less likely to be removed. Specifically, the filtering adjustment unit 223 adjusts the range of elements remaining without being removed, the closer it is to an element with a large change in score, the wider the range of elements remaining before and after the element with a large change in score. For example, the filtering adjustment unit 223 calculates a moving average of the score for each element. In this case, the filtering adjustment unit 223 widens the range for calculating the moving average, the closer it is to an element with a large change in score. Then, the filtering adjustment unit 223 causes the filtering execution unit 222 to filter the elements using the value of the moving average. In this way, the filtering adjustment unit 223 smooths the value used for filtering according to the change in score. As a result, elements close to elements with a large change in score become less likely to be removed.

[0053] The filtering adjustment unit 223 may specify the magnitude of the amount of change in the score by calculating a value used to detect a change point, such as a differential value indicating the amount of change in the value.

[0054] The filtering adjustment unit 223 only needs to make adjustments so that elements close to elements whose scores suddenly increase are less likely to be removed. In other words, the filtering adjustment unit 223 does not need to make adjustments so that elements close to elements whose scores suddenly decrease are less likely to be removed.

[0055] (Step S43: Filtering execution process) The filtering execution unit 222 performs filtering in accordance with the filtering method adjusted in step S42. For example, as described in step S42, when the moving average of the scores for each element is calculated, the filtering execution unit 222 generates the teacher source filtering data 321 by excluding elements with the lowest moving average of each element and the standard ratio.

[0056] ***Effects of the Second Embodiment*** As described above, the learning and inference device 10 according to the second embodiment adjusts the filtering method so that elements close to elements with large amounts of change in score are less likely to be removed. If elements close to elements with large score changes are removed, elements before and after the high-score element may be deleted, potentially resulting in a loss of more context than necessary. The learning inference device 10 according to the second embodiment can prevent elements before and after the high-score element from being deleted, thereby preventing a loss of more context than necessary. This prevents a deterioration in the quality of the generated results.

[0057] Embodiment 3 The third embodiment differs from the first and second embodiments in that the score is adjusted by weighting the score using information other than the score. In the third embodiment, this difference will be explained, and explanation of the same points will be omitted. In the third embodiment, a case will be described in which a function is added to the first embodiment. However, it is also possible to add a function to the second embodiment.

[0058] ***Configuration Description*** The configuration of a learning and inference device 10 according to the third embodiment will be described with reference to FIG. 2 in that the learning and inference device 10 includes a score adjustment unit 224 as a functional component. The function of the score adjustment unit 224, like the other functional components, is realized by software or hardware.

[0059] ***Explanation of Operation*** The filtering process according to the third embodiment will be described with reference to FIG. Here, the filtering process of step S12 in Fig. 4 will be described. However, the same applies to the filtering process of step S31 in Fig. 10. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate input filtered data, which is different.

[0060] The process of step S51 is the same as the process of step S21 in FIG.

[0061] (Step S52: Score adjustment process) The score adjustment unit 224 adjusts the score calculated in step S41 by weighting it with information separate from the score. The other information may be, for example, weight information for each element defined by the user. The other information may be statistical information such as TF-IDF, which stands for Term Frequency-Inverse Document Frequency. As shown in Example 1 of Figure 17, the score adjustment unit 224 adjusts the score by weighting using a basic operation such as addition or multiplication using the score and other information. In Example 1 of Figure 17, the score is adjusted by adding the score to a weight that is other information. Also, as shown in Example 2 of Figure 17, the score adjustment unit 224 may adjust the score so that an element that has been assigned a weight exceeding a threshold is not filtered regardless of the score. In Example 2 of Figure 17, a value of ∞ is set for the score so that an element that has a weight that is other information exceeding a threshold of 2.0 is not filtered.

[0062] (Step S53: Filtering execution process) The filtering execution unit 222 performs filtering based on the score adjusted in step S52.

[0063] ***Effects of the Third Embodiment*** As described above, the learning and inference device 10 according to the third embodiment adjusts the score by weighting it with information other than the score. This prevents unintended loss of context that occurs only due to the score of each element in the input sequence. This prevents deterioration in the quality of the generated result.

[0064] Embodiment 4 The fourth embodiment differs from the first to third embodiments in that the model is stably trained by performing curriculum learning in which the ratio of elements to be filtered is used as the degree of difficulty. In the fourth embodiment, this difference will be explained, and explanations of the same points will be omitted. In the fourth embodiment, a case will be described in which a function is added to the first embodiment. However, it is also possible to add a function to the second and third embodiments.

[0065] ***Explanation of Operation*** The process of the learning phase according to the fourth embodiment will be described with reference to FIG. The process in step S11 is the same as in the first embodiment.

[0066] (Step S12: Filtering process) The filtering unit 22 filters the teacher source data 311 included in the first teacher data 31 using the parameters of the first model 41 obtained by the learning in step S11. At this time, the filtering unit 22 sets each of the multiple ratios to a target ratio. Then, the filtering unit 22 filters the teacher source data 311 to remove elements of the target ratio from the elements included in the teacher source data 311, and generates teacher source filtered data 321 corresponding to the target ratio. Then, the filtering unit 22 generates second teacher data 32 including the teacher source filtered data 321 corresponding to the target ratio and the teacher destination data 312. For example, the filtering unit 22 sets the target proportions to 50%, 70%, and 90%. Then, the filtering unit 22 filters the elements included in the teacher source data 311 to remove 50% of the elements, and generates teacher source filtered data 321 corresponding to 50%. Furthermore, the filtering unit 22 filters the elements included in the teacher source data 311 to remove 70% of the elements, and generates teacher source filtered data 321 corresponding to 70%. Furthermore, the filtering unit 22 filters the elements included in the teacher source data 311 to remove 90% of the elements, and generates teacher source filtered data 321 corresponding to 90%.

[0067] (Step S13: Filtering data learning process) The filtering data learning unit 23 sets the teacher source filtering data 321 corresponding to each of the multiple ratios in ascending order of ratio to the target teacher source filtering data 321. Then, the filtering data learning unit 23 receives as input the second teacher data 32 including the target teacher source filtering data 321 and the teacher destination data 312, and learns parameters of a second model 42 that converts the target teacher source filtering data 321 into the teacher destination data 312. For example, suppose that second teacher data 32 has been generated for 50%, 70%, and 90%. In this case, as shown in FIG. 18, first, the filtering data learning unit 23 learns the parameters of the second model 42 using the second teacher data 32 corresponding to 50%. Next, the filtering data learning unit 23 learns the parameters of the second model 42 using the second teacher data 32 corresponding to 70%. Then, the filtering data learning unit 23 learns the parameters of the second model 42 using the second teacher data 32 corresponding to 90%.

[0068] During learning, the filtering data learning unit 23 uses the parameters of the second model 42 obtained by learning one stage earlier as the initial values ​​of the parameters of the second model 42. However, the filtering data learning unit 23 may use the parameters of the second model 42 obtained by learning two or more stages earlier as the initial values ​​of the parameters of the second model 42 during learning.

[0069] ***Effects of the Fourth Embodiment*** As described above, the learning and inference device 10 according to the fourth embodiment performs curriculum learning in which the proportion of elements to be filtered determines the degree of difficulty. This allows for stepwise learning of differences between the pre-filtering teacher source data 311 and the filtered data. This allows for stable learning of the parameters of the second model 42.

[0070] Embodiment 5 The fifth embodiment differs from the first to fourth embodiments in that the parameters of the second model are learned so that the score obtained from the calculation result by the attention mechanism in the first model 41 approaches the score obtained from the calculation result by the attention mechanism in the second model 42. In the fifth embodiment, this difference will be explained, and explanation of the same points will be omitted. In the fifth embodiment, a case will be described in which a function is added to the first embodiment. However, it is also possible to add a function to the second to fourth embodiments.

[0071] ***Explanation of Operation*** The process of the learning phase according to the fifth embodiment will be described with reference to FIG. The processes in steps S11 and S12 are the same as those in the first embodiment.

[0072] (Step S13: Filtering data learning process) The filtering data learning unit 23 learns the parameters of the second model 42 so that the score obtained from the calculation result by the attention mechanism in the first model 41 and the score obtained from the calculation result by the attention mechanism in the second model 42 become closer to each other. Specifically, the filtering data learning unit 23 introduces a difference or similarity between the score obtained from the calculation result by the attention mechanism in the first model 41 and the score obtained from the calculation result by the attention mechanism in the second model 42 into a loss function used when learning the second model 42. Here, the score obtained from the calculation result by the attention mechanism in the first model 41 is the score calculated in step S21. In other words, the score obtained from the calculation result by the attention mechanism in the first model 41 is the score obtained by Attention, which is the attention mechanism of the first model 41. The score obtained from the calculation result by the attention mechanism in the second model 42 is the score calculated by the method described in step S21. In other words, the score obtained from the calculation result by the attention mechanism in the second model 42 is the score obtained by Attention, which is the attention mechanism of the second model 42.

[0073] For example, the filtering data learning unit 23 uses, as the loss function, the mean square error MSE between the score of the first model 41 and the score of the second model 42. The mean square error MSE between the score of the first model 41 and the score of the second model 42 is expressed as in Equation 1. (Formula 1) MSE=(1 / N)Σ i=1 N (A i -B i ) 2 Here, N indicates the number of Attention elements, A indicates the Attention of the first model 41, and B indicates the Attention of the second model 42. The attention here may be cross-attention calculated by a multi-head attention mechanism in the decoder, or self-attention calculated by a multi-head attention mechanism in the encoder. Also, the attention here may be a combination of cross-attention and self-attention.

[0074] The above-described attention is intended to approximate the entire first model 41 and the second model 42. However, it is also possible to approximate a specific layer or a specific head.

[0075] Here, the mean squared error (MSE) is used as the loss function, but other functions such as cross-entropy error or cosine similarity may also be used as the loss function.

[0076] ***Effects of the Fifth Embodiment*** As described above, the learning and inference device 10 according to the fifth embodiment learns the parameters of the second model so that the score of the first model 41 and the score of the second model 42 become closer to each other. This allows the second model 42 to be learned using not only the filtering results but also the internal state of the first model 41 used in the filtering as training data. This allows the learning to proceed stably.

[0077] Embodiment 6 The sixth embodiment differs from the first to fifth embodiments in that the teacher source data 311 is filtered in any length unit and the results are combined to generate the teacher destination data 312. In the sixth embodiment, this difference will be explained, and explanation of the same points will be omitted. In the sixth embodiment, a case will be described in which a function is added to the first embodiment. However, it is also possible to add a function to the second to fifth embodiments.

[0078] ***Configuration Description*** The configuration of a learning and inference device 10 according to the sixth embodiment will be described with reference to FIG. 2 in that the learning and inference device 10 includes a filtering data combination unit 225 as a functional component. The function of the filtering data combination unit 225, like the other functional components, is realized by software or hardware.

[0079] ***Explanation of Operation*** The filtering process according to the sixth embodiment will be described with reference to FIG. Here, the filtering process of step S12 in Fig. 4 will be described. However, the same applies to the filtering process of step S31 in Fig. 10. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate input filtered data, which is different.

[0080] The process of step S61 is the same as the process of step S21 in FIG.

[0081] (Step S62: Filtering execution process) 21 , the filtering execution unit 222 sets each of the multiple split data 35 obtained by splitting the teacher source data 311 as the target split data 35. The filtering execution unit 222 filters the elements included in the target split data 35 to generate split filtered data 36 corresponding to the target split data 35. 21, the filtering execution unit 222 divides the teacher source data 311 into two to generate two divided data 35. Then, the filtering execution unit 222 filters each of the two divided data 35 to generate two divided filtered data 36. The filtering execution unit 222 may divide the teacher source data 311 into pieces of any length. For example, the filtering execution unit 222 may divide the teacher source data 311 by topic. Alternatively, the filtering execution unit 222 may divide the teacher source data 311 into pieces of a length based on the teacher source data 311.

[0082] (Step S63: Filtering data combination process) As shown in FIG. 21, the filtering data combining unit 225 combines the divided filtering data 36 corresponding to each of the plurality of divided data 35 generated in step S62 to generate teacher source filtering data 321. In FIG. 21, the filtering data combining unit 225 combines the divided filtering data 36 corresponding to each of the two divided data 35 to generate the teacher source filtering data 321.

[0083] ***Effects of the Sixth Embodiment*** As described above, the learning and inference device 10 according to the sixth embodiment combines the results obtained by filtering the teacher source data 311 in units of any length to generate the teacher source filtered data 321. This makes it possible to reduce the computational cost of filtering while preventing loss of context, even when the computational cost of filtering becomes high.

[0084] Embodiment 7 The seventh embodiment differs from the sixth embodiment in that filtering and combining of the filtered data are repeated until the length is equal to or less than the specified length. In the seventh embodiment, this difference will be explained, and explanations of the same points will be omitted.

[0085] ***Explanation of Operation*** The filtering process according to the seventh embodiment will be described with reference to FIG. Here, the filtering process of step S12 in Fig. 4 will be described. However, the same applies to the filtering process of step S31 in Fig. 10. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate input filtered data, which is different.

[0086] The processing from step S71 to step S73 is the same as the processing from step S61 to step S63 in FIG.

[0087] (Step S74: Length determination process) The filtering data combining unit 225 determines whether the length of the teacher source filtering data 321 generated in step S73 is equal to or less than a specified length. If the length of the teacher source filtering data 321 is equal to or less than the specified length, the filtering data combining unit 225 ends the process. On the other hand, if the length of the teacher source filtering data 321 is not equal to or less than the specified length, the filtering data combining unit 225 returns the process to step S72.

[0088] 23 , when the process returns to step S72, the filtering execution unit 222 sets each of the one or more pieces of divided data 35 obtained by dividing the teacher source filtering data 321 generated in step S73 as the target divided data 35. The filtering execution unit 222 filters the elements included in the target divided data 35 to generate new divided filtered data 36 corresponding to the target divided data 35. Then, in step S73, the filtering data combination unit 225 combines the newly generated divided filtered data 36 to generate new teacher source filtering data 321. 23, in the first filtering execution process, split filtering data 36 is generated for each of the two split data 35, and teacher source filtering data 321 is generated. Then, in the second filtering execution process, new split filtering data 36 is generated for one of the split data 35, and the split filtering data 36 is set as is as the teacher source filtering data 321.

[0089] ***Effects of the Seventh Embodiment*** As described above, the learning and inference device 10 according to the seventh embodiment repeats filtering and combining the filtered data until the length is equal to or less than the specified length. As a result, even if the input sequence is long, it is possible to generate teacher source filtered data 321 of the intended length while preventing loss of context and suppressing the computational cost associated with filtering.

[0090] Embodiment 8 The eighth embodiment differs from the first to seventh embodiments in that a special token is assigned to the elements removed by filtering. In the fifth embodiment, this difference will be explained, and explanation of the same points will be omitted. In the eighth embodiment, a case will be described in which a function is added to the first embodiment. However, it is also possible to add a function to the second to seventh embodiments.

[0091] ***Configuration Description*** The configuration of a learning and inference device 10 according to the eighth embodiment will be described with reference to FIG. 2 in that the learning and inference device 10 includes a filtering correction unit 226 as a functional component. The function of the filtering correction unit 226, like the other functional components, is realized by software or hardware.

[0092] ***Explanation of Operation*** The filtering process according to the eighth embodiment will be described with reference to FIG. Here, the filtering process of step S12 in Fig. 4 will be described. However, the same applies to the filtering process of step S31 in Fig. 10. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate input filtered data, which is different.

[0093] The process of step S81 is the same as the process of step S21 in FIG.

[0094] (Step S82: Filtering execution process) The filtering execution unit 222 generates teacher source filtered data 321 by filtering the elements included in the teacher source data 311 based on the score calculated in step S21. At this time, as shown in Fig. 26, the filtering execution unit 222 assigns a special token to the elements removed by filtering. In Fig. 26, similar to Fig. 9, input 2 and input 4 are removed, and teacher source filtered data 321 including input 1 and input 3 is generated. Therefore, the filtering execution unit 222 assigns a special token [M] to the removed input 2 and input 4.

[0095] (Step S83: Filtering correction process) The filtering correction unit 226 outputs the results of the filtering performed in step S82 to the display device 14. At this time, the filtering correction unit 226 indicates the elements removed by filtering with a special token [M], and then displays each element of the teacher source data 311. This allows the user to know which elements have been removed. Then, the filtering correction unit 226 accepts designation of elements that the user considers inappropriate to remove. As shown in Fig. 27, the filtering correction unit 226 stops removing elements that have been designated as inappropriate to remove, and adds them to the teacher source filtering data 321. In Fig. 27, input 2 has been designated as inappropriate to remove, and therefore has been added to the teacher source filtering data 321.

[0096] ***Effects of the eighth embodiment*** As described above, the learning and inference device 10 according to the eighth embodiment assigns a special token to the elements removed by filtering, thereby enabling the user to easily identify the elements removed by filtering.

[0097] Furthermore, in the learning and inference device 10 according to the eighth embodiment, the filtering correction unit 226 adds an element designated by the user to the teacher source filtering data 321. This allows the user to correct the filtered results and obtain desirable teacher source filtering data 321.

[0098] In addition, the word "unit" in the above description may be read as a "circuit," "step," "procedure," "process," or "processing circuit."

[0099] The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be combined and implemented. Also, one or more of them may be implemented partially. Note that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as needed. [Explanation of symbols]

[0100] 10 Learning and inference device, 11 Processor, 12 Memory, 13 Input device, 14 Display device, 15 External interface, 16 Communication interface, 17 Recording medium, 21 Normal data learning unit, 22 Filtering unit, 221 Score calculation unit, 222 Filtering execution unit, 223 Filtering adjustment unit, 224 Score adjustment unit, 225 Filtered data combination unit, 226 Filtering correction unit, 23 Filtered data learning unit, 24 Filtered data inference unit, 31 First teacher data, 311 Teacher source data, 312 Teacher destination data, 32 Second teacher data, 321 Teacher source filtered data, 33 Input data, 34 Output data, 35 Split data, 36 Split filtered data, 41 First model, 42 Second model, 101 Learning device, 102 Inference device.

Claims

1. a filtering unit that calculates a score for each of a plurality of elements included in first teacher data including teacher source data, which is the sequence data of the conversion source, and teacher destination data, which is the sequence data of the conversion destination, using an attention mechanism in a first model that infers sequence data of the conversion source from the sequence data of the conversion source, and filters the plurality of elements included in the teacher source data based on the scores of each of the plurality of elements to generate teacher source filtered data; a filtering data learning unit that receives second teacher data including the teacher-source filtered data and the teacher-destination data generated by the filtering unit as input and learns parameters of a second model that converts the teacher-source filtered data into the teacher-destination data; Equipped with the filtering unit filters elements included in each of a plurality of divided data obtained by dividing the teacher source data as target divided data, and generates divided filtered data corresponding to the target divided data; a filtering data combining unit that combines the divided filtering data corresponding to each of the plurality of divided data to generate the teacher source filtering data; A learning device comprising:

2. a filtering unit that calculates a score for each of a plurality of elements included in first teacher data including teacher source data, which is the sequence data of the conversion source, and teacher destination data, which is the sequence data of the conversion destination, using an attention mechanism in a first model that infers sequence data of the conversion source from the sequence data of the conversion source, and filters the plurality of elements included in the teacher source data based on the scores of each of the plurality of elements to generate teacher source filtered data; a filtering data learning unit that receives second teacher data including the teacher-source filtered data and the teacher-destination data generated by the filtering unit as input and learns parameters of a second model that converts the teacher-source filtered data into the teacher-destination data; Equipped with The filtering unit assigns a special token to the filtered out element, Furthermore, a filtering correction unit that returns the specified special token to the original element. A learning device comprising:

3. The filtering unit a score calculation unit that calculates the score for each of the plurality of elements; a filtering adjustment unit that adjusts a filtering method so that, when the scores calculated by the score calculation unit are arranged in accordance with the arrangement order of the plurality of elements included in the teacher source data, an element close to an element with a large amount of change in the score is less likely to be removed; a filtering execution unit that performs filtering according to the filtering method adjusted by the filtering adjustment unit; The learning device according to claim 1 or 2, comprising:

4. The filtering unit a score calculation unit that calculates the score for each of the plurality of elements; a score adjustment unit that adjusts the score calculated by the score calculation unit by weighting the score with information different from the score; a filtering execution unit that performs filtering based on the score adjusted by the score adjustment unit; The learning device according to claim 1 or 2, comprising:

5. the filtering unit filters each of a plurality of ratios as a target ratio to remove elements of the target ratio from elements included in the teacher source data, thereby generating the teacher source filtering data corresponding to the target ratio; The filtering data learning unit sets the teacher-source filtering data corresponding to each of the plurality of ratios as target teacher-source filtering data in order from the lowest ratio, and learns parameters of a second model that converts the target teacher-source filtering data into the teacher-destination data using second teacher data including the target teacher-source filtering data and the teacher-destination data as input. The learning device according to claim 1 or 2.

6. The filtering data learning unit learns parameters of the second model so that a score obtained from a calculation result by an attention mechanism in the first model and a score obtained from a calculation result by an attention mechanism in the second model become closer to each other. The learning device according to claim 1 or 2.

7. when the teacher source filtering data is longer than a specified length, the filtering unit takes one or more divided data obtained by dividing the teacher source filtering data as target divided data, filters elements included in the target divided data, and generates new divided filtered data corresponding to the target divided data; The filtering data combining unit combines the newly generated divided filtering data to newly generate the teacher source filtering data. The learning device according to claim 1 .

8. a filtering unit that calculates a score for each of a plurality of elements included in input data, which is the source sequence data, using an attention mechanism in a first model that infers destination sequence data from source sequence data, and filters the plurality of elements included in the input data based on the scores of each of the plurality of elements to generate input filtered data; a filtering data inference unit that infers output data from the input filtering data generated by the filtering unit using a second model that infers destination sequence data from source sequence data; Equipped with the filtering unit filters elements included in each of a plurality of pieces of split data obtained by splitting the input data as target split data, and generates split filtered data corresponding to the target split data; a filtering data combining unit that combines the divided filtering data corresponding to each of the plurality of divided data to generate the input filtering data; An inference device comprising:

9. a filtering unit that calculates a score for each of a plurality of elements included in input data, which is the source sequence data, using an attention mechanism in a first model that infers destination sequence data from source sequence data, and filters the plurality of elements included in the input data based on the scores of each of the plurality of elements to generate input filtered data; a filtering data inference unit that infers output data from the input filtering data generated by the filtering unit using a second model that infers destination sequence data from source sequence data; Equipped with The filtering unit assigns a special token to the filtered out element, Furthermore, a filtering correction unit that returns the specified special token to the original element. An inference device comprising:

10. The filtering unit a score calculation unit that calculates the score for each of the plurality of elements; a filtering adjustment unit that adjusts a filtering method so that, when the scores calculated by the score calculation unit are arranged in accordance with the arrangement order of the plurality of elements included in the input data, elements near an element having a large amount of change in the score are less likely to be removed; a filtering execution unit that performs filtering according to the filtering method adjusted by the filtering adjustment unit; 10. The inference device according to claim 8 or 9, comprising:

11. The filtering unit a score calculation unit that calculates the score for each of the plurality of elements; a score adjustment unit that adjusts the score calculated by the score calculation unit by weighting the score with information different from the score; a filtering execution unit that performs filtering based on the score adjusted by the score adjustment unit; 10. The inference device according to claim 8 or 9, comprising:

12. The second model is obtained by learning parameters using as inputs teacher source filtered data generated by filtering the plurality of elements included in the teacher source data in first teacher data including teacher source data which is sequence data to be converted and teacher destination data which is sequence data to be converted, and second teacher data including the teacher destination data.

10. The inference device according to claim 8 or 9.

13. The second model is obtained by filtering the elements included in the teacher source data so as to remove elements of the target ratio from elements included in the teacher source data, with each of the plurality of ratios being a target ratio, to generate the teacher source filtered data corresponding to the target ratio, and by using the teacher source filtered data corresponding to each of the plurality of ratios as target teacher source filtered data in order from the lowest ratio, and learning parameters using teacher data including the target teacher source filtered data and the teacher destination data as input. The inference device of claim 12.

14. The second model is obtained by learning parameters so that the score obtained from the calculation result by the attention mechanism in the first model and the score obtained from the calculation result by the attention mechanism in the second model become closer to each other. The inference device of claim 12.

15. when the input filtered data is longer than a specified length, the filtering unit treats each of a plurality of divided data obtained by dividing the input filtered data as target divided data, filters elements included in the target divided data, and generates new divided filtered data corresponding to the target divided data; The filtering data combining unit combines the newly generated divided filtering data to newly generate the input filtering data. The inference device according to claim 8.

16. a computer uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate scores for each of a plurality of elements included in first teacher data including source teacher data, which is source sequence data, and destination teacher data, which is destination sequence data, and filters the plurality of elements included in the source teacher data based on the scores for each of the plurality of elements to generate source teacher filtered data; a computer receives second teacher data including the teacher-source filtered data and the teacher-destination data as input and learns parameters of a second model that converts the teacher-source filtered data into the teacher-destination data; a computer filters elements included in each of a plurality of divided data obtained by dividing the teacher source data as target divided data, thereby generating divided filtered data corresponding to the target divided data; A learning method in which a computer generates the teacher source filtered data by combining the divided filtered data corresponding to each of the plurality of divided data.

17. a computer uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate scores for each of a plurality of elements included in first teacher data including source teacher data, which is source sequence data, and destination teacher data, which is destination sequence data, and filters the plurality of elements included in the source teacher data based on the scores for each of the plurality of elements to generate source teacher filtered data; a computer receives second teacher data including the teacher-source filtered data and the teacher-destination data as input and learns parameters of a second model that converts the teacher-source filtered data into the teacher-destination data; The computer assigns special tokens to the filtered out elements, A learning method in which a computer returns the specified special token to its original element.

18. a filtering process of calculating a score for each of a plurality of elements included in first teacher data including teacher source data, which is the sequence data of the conversion source, and teacher destination data, which is the sequence data of the conversion destination, using an attention mechanism in a first model that infers sequence data of the conversion source from sequence data of the conversion source, and filtering the plurality of elements included in the teacher source data based on the scores of each of the plurality of elements to generate teacher source filtered data; a filtering data learning process in which second teacher data including the teacher-source filtered data and the teacher-destination data generated by the filtering process is used as input to learn parameters of a second model that converts the teacher-source filtered data into the teacher-destination data; and In the filtering process, each of a plurality of divided data obtained by dividing the teacher source data is treated as target divided data, and elements included in the target divided data are filtered to generate divided filtered data corresponding to the target divided data; Furthermore, a filtering data combining process is performed to combine the divided filtering data corresponding to each of the plurality of divided data to generate the teacher source filtering data. A learning program that enables a computer to function as a learning device.

19. a filtering process of calculating a score for each of a plurality of elements included in first teacher data including teacher source data, which is the sequence data of the conversion source, and teacher destination data, which is the sequence data of the conversion destination, using an attention mechanism in a first model that infers sequence data of the conversion source from sequence data of the conversion source, and filtering the plurality of elements included in the teacher source data based on the scores of each of the plurality of elements to generate teacher source filtered data; a filtering data learning process in which second teacher data including the teacher-source filtered data and the teacher-destination data generated by the filtering process is used as input to learn parameters of a second model that converts the teacher-source filtered data into the teacher-destination data; and In the filtering process, a special token is assigned to the filtered out element, Furthermore, a filtering correction process is performed to return the specified special token to the original element. A learning program that enables a computer to function as a learning device.

20. a computer uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate a score for each of a plurality of elements included in input data that is the source sequence data, and filters the plurality of elements included in the input data based on the scores for each of the plurality of elements to generate input filtered data; a computer inferring output data from the input filtered data using a second model for inferring destination sequence data from source sequence data; the computer filters elements included in each of the plurality of divided data obtained by dividing the input data as target divided data, thereby generating divided filtered data corresponding to the target divided data; An inference method in which a computer generates the input filtered data by combining the divided filtered data corresponding to each of the plurality of divided data.

21. a computer uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate a score for each of a plurality of elements included in input data that is the source sequence data, and filters the plurality of elements included in the input data based on the scores for each of the plurality of elements to generate input filtered data; a computer inferring output data from the input filtered data using a second model for inferring destination sequence data from source sequence data; The computer assigns special tokens to the filtered out elements, An inference method in which a computer converts the specified special token back into its original element.

22. a filtering process of calculating a score for each of a plurality of elements included in input data, which is the source sequence data, using an attention mechanism in a first model that infers destination sequence data from source sequence data, and filtering the plurality of elements included in the input data based on the scores of each of the plurality of elements to generate input filtered data; a filtering data inference process for inferring output data from the input filtering data generated by the filtering process using a second model for inferring destination sequence data from source sequence data; and In the filtering process, each of a plurality of pieces of split data obtained by splitting the input data is treated as a target piece of split data, and elements included in the target piece of split data are filtered to generate split filtered data corresponding to the target piece of split data; and a filtering data combining process for combining the divided filtering data corresponding to each of the plurality of divided data to generate the input filtering data. An inference program that enables a computer to function as an inference device.

23. a filtering process of calculating a score for each of a plurality of elements included in input data, which is the source sequence data, using an attention mechanism in a first model that infers destination sequence data from source sequence data, and filtering the plurality of elements included in the input data based on the scores of each of the plurality of elements to generate input filtered data; a filtering data inference process for inferring output data from the input filtering data generated by the filtering process using a second model for inferring destination sequence data from source sequence data; and The filtering process assigns a special token to the filtered out elements, Furthermore, a filtering correction process is performed to return the specified special token to the original element. An inference program that enables a computer to function as an inference device.

Images