Training device, training method, training program, inference device, inference method, and inference program

Abstract

A filtering unit (22) uses an attention mechanism of a first model (41) to calculate respective scores of a plurality of elements included in teacher source data (311) of first teacher data (31), which includes the teacher source data (311) and teacher target data (312). The filtering unit (22) filters, on the basis of the respective scores of the plurality of elements, the plurality of elements included in the teacher source data (311), to generate filtered teacher source data (321). A filtered data training unit (23) receives, as input, second teacher data (32) which includes the filtered teacher source data (321) and the teacher target data (312), to train the parameters of a second model (42), which is for performing conversion from the filtered teacher source data (321) to the teacher target data (312).

Description

Learning device, learning method, learning program, inference device, inference method, and inference program

[0001] The present disclosure relates to a technique using a model with an attention mechanism.

[0002] In machine translation and other sequence conversion tasks using machine learning techniques, neural network models consisting of an encoder and a decoder are used. Hereinafter, a neural network model consisting of an encoder and a decoder is referred to as an encoder-decoder model. It is known that introducing an attention mechanism into an encoder-decoder model can significantly improve accuracy. In the case of machine translation, an attention mechanism is a data-driven mechanism that determines which word information in a target language sentence to use in the decoder.

[0003] Currently, the Transformer is a mainstream encoder-decoder model that combines an attention mechanism and a fully connected layer to parallelize the encoder. The Transformer is a model that configures an encoder and decoder by stacking a combination of a multi-head attention mechanism or a masked multi-head attention mechanism and a fully connected layer. Hereinafter, the combination of a multi-head attention mechanism or a masked multi-head attention mechanism and a fully connected layer will be considered as one layer and will be referred to as a Transformer layer.

[0004] The attention mechanism of the Transformer calculates the combinations between all elements in the input sequence. If the number of elements in the input sequence is n, then the combinations between all elements are n ２ Therefore, the attention mechanism of the Transformer requires GPU memory of O(n ２ ) GPU stands for Graphics Processing Unit. Therefore, as the input sequence becomes longer, the computational cost required for processing increases quadratically.

[0005] Patent Document 1 describes a method of dividing a long text into topic units to generate blocked text, summarizing the text for each block, and generating a summary result for each topic.

[0006] Japanese Patent Application Laid-Open No. 2023-034235

[0007] The technology described in Patent Document 1 still requires increased computational costs when the blocked text corresponding to a specific topic becomes long. When the blocked text is long, it is possible to shorten the input sequence by further dividing it into smaller blocks, thereby reducing computational costs. However, because each blocked text is processed independently, further dividing it into smaller blocks may result in a loss of context and a deterioration in the quality of the generated text. In other words, it is not possible to appropriately adjust the length of the input sequence while maintaining the quality of the generated result. The present disclosure aims to make it possible to appropriately adjust the length of the input sequence while maintaining the quality of the generated result.

[0008] A learning device according to the present disclosure includes: a filtering unit that calculates a score for each of a plurality of elements included in first teacher data, which includes teacher source data that is source sequence data and teacher destination data that is destination sequence data, using an attention mechanism in a first model that infers destination sequence data from source sequence data, and filters the plurality of elements included in the teacher source data based on the scores for each of the plurality of elements to generate teacher source filtered data; and a filtered data learning unit that uses as input second teacher data that includes the teacher source filtered data and the teacher destination data generated by the filtering unit, and learns parameters of a second model that converts the teacher source filtered data into the teacher destination data.

[0009] In this disclosure, an attention mechanism is used to calculate the scores of each of the multiple elements included in the source sequence data, and then the multiple elements are filtered. Because filtering is performed using the scores calculated using the attention mechanism, it is possible to appropriately adjust the length of the source sequence data (the input sequence) while maintaining the quality of the generated results.

[0010] 1 is a hardware configuration diagram of a learning and inference device 10 according to embodiment 1. FIG. 2 is a functional configuration diagram of the learning and inference device 10 according to embodiment 1. FIG. 3 is an explanatory diagram of an encoder-decoder model targeted in embodiment 1. FIG. 4 is a flowchart of processing in the learning phase according to embodiment 1. FIG. 5 is an explanatory diagram of filtering processing according to embodiment 1. FIG. 6 is a flowchart of filtering processing according to embodiment 1. FIG. 7 is an explanatory diagram of normal decoding according to embodiment 1. FIG. 8 is an explanatory diagram of Teacher Forcing Decoding according to embodiment 1. FIG. 9 is an explanatory diagram of filtering execution processing according to embodiment 1. FIG. 10 is a flowchart of processing in the inference phase according to embodiment 1. FIG. 11 is a functional configuration diagram of a learning device 101 according to modification 1. FIG. 12 is a functional configuration diagram of an inference device 102 according to modification 1. FIG. 12 is a configuration diagram of a learning and inference device 10 according to embodiment 2. FIG. 13 is a flowchart of filtering processing according to embodiment 3. FIG. 14 is an explanatory diagram of score adjustment processing according to embodiment 3. FIG. 15 is an explanatory diagram of filtering data learning processing according to embodiment 4. FIG. 13 is a configuration diagram of a learning and inference device 10 according to embodiment 6. FIG. 15 is a flowchart of filtering processing according to embodiment 6. FIG. 16 is an explanatory diagram of filtering processing according to embodiment 6. 10 is a flowchart of a filtering process according to embodiment 7. FIG. 11 is an explanatory diagram of a filtering process according to embodiment 7. FIG. 12 is a configuration diagram of a learning and inference device 10 according to embodiment 8. FIG. 13 is a flowchart of a filtering process according to embodiment 8. FIG. 14 is an explanatory diagram of a filtering execution process according to embodiment 8. FIG. 15 is an explanatory diagram of a filtering correction process according to embodiment 8.

[0011] First Embodiment *** Description of Configuration *** The hardware configuration of a learning inference device 10 according to the first embodiment will be described with reference to Fig. 1. The learning inference device 10 is a computer. The learning inference device 10 includes the following hardware: a processor 11, a memory 12, an input device 13, a display device 14, an external interface 15, and a communication interface 16. The processor 11 is connected to other hardware via signal lines and controls this other hardware.

[0012] The processor 11 is an IC that performs processing. IC stands for Integrated Circuit. Specific examples of the processor 11 include a CPU, a DSP, and a GPU. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. GPU stands for Graphics Processing Unit.

[0013] The memory 12 is a storage device that stores data. Specific examples of the memory 12 include an HDD, an SSD, a RAM, and a ROM. HDD stands for Hard Disk Drive. SSD stands for Solid State Drive. RAM stands for Random Access Memory. ROM stands for Read Only Memory.

[0014] The input device 13 is a device for inputting information. Specific examples of the input device 13 include a keyboard, a mouse, and a touch panel. The display device 14 is a device for displaying information. Specific example of the display device 14 is an LCD. LCD stands for Liquid Crystal Display. The learning and inference device 10 does not necessarily have to include at least one of the input device 13 and the display device 14.

[0015] The external interface 15 is an interface with an external device such as a recording medium 17. The recording medium 17 is a device for storing data. The processor 11 is capable of reading data stored in the recording medium 17 and writing data to the recording medium 17 via the external interface 15. Specific examples of the recording medium 17 include portable recording media such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, and a DVD. SD stands for Secure Digital. DVD stands for Digital Versatile Disk.

[0016] The communication interface 16 is an interface for connecting to a communication network. A specific example of the communication interface 14 is an Ethernet (registered trademark) port.

[0017] The functional configuration of the learning and inference device 10 according to the first embodiment will be described with reference to Fig. 2. The learning and inference device 10 includes, as functional components, a normal data learning unit 21, a filtering unit 22, a filtered data learning unit 23, and a filtered data inference unit 24. The filtering unit 22 includes a score calculation unit 221 and a filtering execution unit 222. The functions of the functional components of the learning and inference device 10 are realized by software.

[0018] The memory 12 stores a program that realizes the function of each functional component of the learning inference device 10. This program is read by the processor 11 and executed by the processor 11. This realizes the function of each functional component of the learning inference device 10. The program that realizes the function of each functional component of the learning inference device 10 may be stored in a recording medium 17. In this case, the processor 11 reads and executes the program from the recording medium 17 via the external interface 15. The program that realizes the function of each functional component of the learning inference device 10 may also be stored in an external server or the like. In this case, the processor 11 reads and executes the program from the server or the like via the communication interface 16.

[0019] The memory 12 stores data such as first teacher data 31, second teacher data 32, a first model 41, and a second model 42. These data may be stored not in the memory 12 but in the recording medium 17 or an external server, etc.

[0020] 1 shows only one processor 11. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that realize the respective functions.

[0021] ***Description of Operation*** The operation of the learning inference device 10 according to embodiment 1 will be described with reference to FIGS. 3 to 10. The operation of the learning inference device 10 includes a learning phase and an inference phase. The operation procedure of the learning phase of the learning inference device 10 according to embodiment 1 corresponds to the learning method according to embodiment 1. Furthermore, a program that realizes the operation of the learning phase of the learning inference device 10 according to embodiment 1 corresponds to the learning program according to embodiment 1. The operation procedure of the inference phase of the learning inference device 10 according to embodiment 1 corresponds to the inference method according to embodiment 1. Furthermore, a program that realizes the operation of the inference phase of the learning inference device 10 according to embodiment 1 corresponds to the inference program according to embodiment 1.

[0022] An encoder-decoder model to be used in the first embodiment will be described with reference to Fig. 3. The encoder-decoder model to be used in the first embodiment is a Transformer, which is an encoder-decoder model in which an attention mechanism and a fully connected layer are combined to parallelize the encoder. The Transformer configures an encoder and a decoder by stacking a combination of a multi-head attention mechanism or a masked multi-head attention mechanism and a fully connected layer.

[0023] The learning phase processing according to the first embodiment will be described with reference to FIG. 4 . (Step S11: Normal Data Learning Processing) The normal data learning unit 21 uses the first teacher data 31 as input to learn parameters of the first model 41. The first model 41 is an encoder-decoder model that infers destination sequence data from source sequence data. Here, the first model 41 is a Transformer. As a Transformer, the first model 41 has an attention mechanism. The parameters of the first model 41 to be learned are the parameters to be learned by the encoder-decoder model. The first teacher data 31 is any sequence data that can be input and output by the encoder-decoder model. The first teacher data 31 has a data structure including teacher source data 311, which is the source sequence data, and teacher destination data 312, which is the destination sequence data. The first teacher data 31 is, for example, data for machine translation. In this case, the teacher source data 311, which is the source sequence data, is a word sequence in the source language. The word sequence is a sentence or a phrase. The converted sequence data, or training data 312, is the result of converting the source language into the target language. In other words, the training data 312 is a sequence of words in the target language that express the same meaning as the sequence of words in the source language.

[0024] (Step S12: Filtering Process) The filtering unit 22 filters the teacher source data 311 included in the first teacher data 31 using the parameters of the first model 41 obtained by the learning in step S11. Filtering the teacher source data 311 means removing some of the elements included in the teacher source data 311. If the first teacher data 31 is data for machine translation, the elements included in the teacher source data 311 are the words that make up the teacher source data 311. In this way, the filtering unit 22 generates teacher source filtered data 321 from which some elements have been removed from the teacher source data 311. Each square shown in FIG. 5 represents an element included in the teacher source data 311. The filtering unit 22 removes some of the elements marked with an "x" from the elements included in the teacher source data 311. In this way, the filtering unit 22 generates teacher source filtered data 321 from the teacher source data 311. In FIG. 5, 50% of the elements included in the teacher source data 311 have been removed, and the teacher source filtered data 321 is generated. Then, the filtering unit 22 generates second teacher data 32 including the teacher source filtering data 321 and the teacher destination data 312 .

[0025] (Step S13: Filtering Data Learning Process) The filtering data learning unit 23 uses the second teacher data 32 generated in step S12 as input to learn parameters of the second model 42. The second model 42 is an encoder / decoder model that infers destination sequence data from source sequence data. Here, the second model 42 is a Transformer. The parameters of the second model 42 to be learned are the parameters of the encoder / decoder model to be learned. Any optimization method may be used to learn the parameters. For example, Adam may be used as the optimization method. Adam is an abbreviation for ADAptive Moment. The filtering data learning unit 23 writes the parameters of the second model 42 obtained by learning to the memory 12. Alternatively, the filtering data learning unit 23 may output the parameters to the recording medium 17, an external server, or a display device 14.

[0026] The filtering process (step S12 in FIG. 4) according to the first embodiment will be described with reference to FIG. 6. (Step S21: Score Calculation Process) The score calculation unit 221 calculates the score of each of the multiple elements included in the teacher source data 311 using the attention mechanism in the first model 41. Specifically, the score calculation unit 221 calculates the score of each of the multiple elements included in the teacher source data 311 using the Attention ｄ，ｌ，ｈ，ｉ The score of each of the multiple elements is calculated using the above formula. Attention is Cross-Attention calculated by the multi-head attention mechanism of the decoder. In other words, the score calculation unit 221 calculates, as a score, Attention for each of the multiple elements in the input sequence of the encoder, which is calculated when generating the output sequence of the decoder. In other words, the score calculation unit 221 calculates, as a score, the internal state of the first model 41 for each of the multiple elements in the input sequence of the encoder. Here, the input sequence of the encoder is the teacher source data 311.

[0027] Attention ｄ，ｌ，ｈ，ｉIn the above, d, l, h, and i respectively indicate the following: d indicates the number of elements in the decoder output sequence; l indicates the number of layers in the Transformer layer; h indicates the number of heads in the multi-head attention mechanism; and i indicates the number of elements in the encoder input sequence. The score calculation unit 221 calculates the Attention scores for d, l, and h for each i. ｄ，ｌ，ｈ，ｉ The score S for element i is calculated by calculating the maximum value of ｉ That is, S ｉ =MaxAttention(i)=max ｄ，ｌ，ｈ (Attention ｄ，ｌ，ｈ，ｉ ) Here, the score calculation unit 221 calculates the Attention ｄ，ｌ，ｈ，ｉ Although the maximum value of is calculated in the above example, a statistical value other than the maximum value may be calculated. Furthermore, the score calculation unit 221 may calculate the score using all d, l, and h, or may calculate the score using only specific d, l, and h. Furthermore, the score calculation unit 221 may obtain multiple scores by generating output sequences from multiple decoders using general decoding such as beam search, extract the maximum value from the multiple scores, and calculate the final score. Note that instead of the maximum value, a score extracted from the multiple scores using another method may be used as the final score. Furthermore, the score calculation unit 221 may separately prepare a sequence labeling model using scores as a teacher signal and estimate the scores of each of the multiple elements in the encoder input sequence using the sequence labeling model.

[0028] In the above description, Attention is Cross-Attention calculated by the multi-head attention mechanism of the decoder. However, Attention may be Self-Attention calculated by the multi-head attention mechanism of the encoder. Furthermore, Attention may be a combination of Cross-Attention and Self-Attention. Cross-Attention and Self-Attention can be combined by calculating the sum or average of Cross-Attention and Self-Attention.

[0029] When calculating Attention, the score calculation unit 221 may use Teacher-Forcing Decoding if the target output sequence is available. When calculating Attention, the score calculation unit 221 may use normal decoding if the target output sequence is not available. As shown in FIG. 7 , normal decoding uses the model's prediction as input for the next prediction. The model's prediction is the element previously output by the model. In FIG. 7 , first, the BOS is provided to the decoder, and output 1 is obtained. BOS stands for Begin Of Sentence. Next, the BOS and output 1 are provided to the decoder, and output 2 is obtained. Then, the BOS, output 1, and output 2 are provided to the decoder, and EOS is obtained. EOS stands for End Of Sentence. In contrast, as shown in Figure 8, Teacher-Forcing Decoding provides the target output sequence directly to the model. In other words, unlike normal decoding, Teacher-Forcing Decoding does not use the results predicted by the model. In Figure 8, the target output sequence is BOS + correct answer 1 + correct answer 2 + EOS. First, BOS is provided to the decoder to obtain output 1. Next, BOS and correct answer 1 are provided to the decoder to obtain output 2. Finally, BOS, correct answer 1, and correct answer 2 are provided to the decoder to obtain EOS.

[0030] (Step S22: Filtering Execution Process) The filtering execution unit 222 filters the elements included in the teacher source data 311 based on the scores calculated in step S21 to generate teacher source filtering data 321. Specifically, as shown in FIG. 9 , the filtering execution unit 222 generates the teacher source filtering data 321 by excluding elements with the lowest scores at a standard rate from among the elements included in the teacher source data 311. In FIG. 9 , the teacher source data 311 includes four elements, and the standard rate is 50%. Therefore, the filtering execution unit 222 generates the teacher source filtering data 321 by excluding the two lowest-scoring elements from the four elements. As a result, inputs 2 and 4, which have the lowest scores, are excluded from inputs 1 to 4 included in the teacher source data 311, and teacher source filtering data 321 including inputs 1 and 3 is generated.

[0031] The filtering execution unit 222 may combine data obtained by filtering using the above-described method with data obtained by filtering the teacher source data 311 using another method to generate the teacher source filtered data 321. A specific example of filtering using another method is removing elements with a standard percentage of high, low, or middle scores from the teacher source data 311 using a maximum value, average value, median, or the like. Another specific example of filtering using another method is removing elements with a standard percentage at the beginning, end, or center of the teacher source data 311 from the teacher source data 311. The filtering execution unit 222 may also combine data obtained by filtering the teacher source data 311 at different rates to generate the teacher source filtered data 321. For example, the filtering execution unit 222 may combine data obtained by removing 10% of the teacher source data 311, data obtained by removing 30% of the teacher source data 311, and data obtained by removing 50% of the teacher source data 311 to generate the teacher source filtered data 321.

[0032] The processing of the inference phase according to the first embodiment will be described with reference to FIG. 10 . (Step S31: Filtering Processing) The filtering unit 22 filters the input data 33 using the parameters of the first model 41 obtained by the learning in step S11. The input data 33 is any sequence data that can be input and output by the encoder-decoder model. In this way, the filtering unit 22 generates input filtered data. The processing for filtering the input data 33 is the same as the processing for filtering the teacher source data 311 in step S12 of FIG. 4 . In other words, the input data 33 is filtered to generate the input filtered data by a processing similar to the processing for filtering the teacher source data 311 to generate the teacher source filtered data 321.

[0033] The filtering unit 22 may change the filtering ratio when filtering the teacher source data 311 in step S12 of Fig. 4 and when filtering the input data 33 here. For example, the filtering unit 22 performs 50% filtering when filtering the teacher source data 311 in step S12 of Fig. 4, and performs 30% filtering when filtering the input data 33 here.

[0034] (Step S32: Filtering Data Inference Process) The filtering data inference unit 24 infers the output data 34 from the input filtering data generated in step S31 using the second model 42. "Using the second model 42" means using the parameters of the second model 42 generated in the learning phase. The filtering data inference unit 24 writes the inferred output data 34 to the memory 12. Alternatively, the filtering data inference unit 24 may output the output data 34 to the recording medium 17 or an external server or display device 14.

[0035] ***Effects of First Embodiment*** As described above, the learning and inference device 10 according to the first embodiment uses an attention mechanism to calculate the score of each of multiple elements included in source sequence data and filters the multiple elements. The score calculated using the attention mechanism is higher for elements with a higher degree of contribution. Therefore, by filtering using the score calculated using the attention mechanism, it becomes possible to appropriately adjust the length of the source sequence data while maintaining the quality of the generated results. The source sequence data refers to the teacher source data 311 and the input data 33.

[0036] Since the length of the source sequence data can be shortened, the computational cost can be reduced. That is, the computational cost required to learn the parameters of the second model 42 can be lower than the computational cost required to learn the parameters of the first model 41. Furthermore, by inferring the output data 34 from the input filtering data, the computational cost can be lower than that required to infer the output data 34 from the input data 33.

[0037] Here, the parameters of the first model 41 are learned using unfiltered first teacher data 31 as input. Therefore, computational costs are high and a high-performance computing environment is required. In contrast, the parameters of the second model 42 are learned using filtered second teacher data 32 as input. Therefore, computational costs are low and a high-performance computing environment is not required. There are cases where processing using an encoder-decoder model is deployed horizontally across multiple businesses or multiple systems. In this case, it is conceivable to centralize a high-performance computing environment for the first model 41 in one location and prepare a lower-performance computing environment for the second model 42 at the deployment destination. Note that once the first model 41 is created, the normal data learning process of step S11 in FIG. 4 is not required when learning the parameters of the second model 42. In other words, once the first model 41 is created, when learning the parameters of the second model 42, step S11 in FIG. 4 can be skipped and processing can begin from step S12. Therefore, the first model 41 may be prepared in a high-performance computer environment, and the parameters of the second model 42 for the deployment destination may be learned in the low-performance computer environment of the deployment destination.

[0038] ***Other Configurations*** <Variation 1> In the first embodiment, an example has been described in which a single learning and inference device 10 has both a function for performing the learning phase and a function for performing the inference phase. However, the learning and inference device 10 may have only a function for performing the learning phase. A learning and inference device 10 having only a function for performing the learning phase will be referred to as a learning device 101. Furthermore, the learning and inference device 10 may have only a function for performing the inference phase. A learning and inference device 10 having only a function for performing the inference phase will be referred to as an inference device 102. Specifically, as shown in FIG. 11 , the learning device 101 does not have the filtering data inference unit 24 of the functional components of the learning and inference device 10, but has a normal data learning unit 21, a filtering unit 22, and a filtering data learning unit 23. Furthermore, as shown in FIG. 12 , the inference device 102 does not have the normal data learning unit 21 and the filtering data learning unit 23 of the functional components of the learning and inference device 10, but has a filtering unit 22 and a filtering data inference unit 24.

[0039] If the first model 41 can be acquired from an external source, the learning device 101 does not need to include the normal data learning unit 21 .

[0040] <Variation 2> In the first embodiment, the first teacher data 31 is specifically data for machine translation. However, the first teacher data 31 is not limited to this and may be other data as long as it is sequence data. Note that the second teacher data 32 is data obtained by filtering the teacher source data 311 of the first teacher data 31, and is therefore the same type of data as the first teacher data 31. Furthermore, the set of input data 33 and output data 34 is also the same type of data as the first teacher data 31.

[0041] (Example 1) The first teacher data 31 is data for natural language processing. The teacher source data 311 is a sequence of words in a specific language, such as a sentence or phrase. The teacher destination data 312 is a summary result in a specific language, i.e., a sequence consisting of fewer words than the teacher source data 311 but retaining the essential meaning of the teacher source data 311. (Example 2) The first teacher data 31 is data for natural language processing. The teacher source data 311 is a sequence of words that represent a question. The teacher destination data 312 is a sequence of words that represent an answer to the question. (Example 3) The first teacher data 31 is data for speech recognition. The teacher source data 311 is a sequence of audio data representing human oral utterances. The teacher destination data 312 is a sequence of phonemes, features, or words that represent the content of the utterance. (Example 4) The first teacher data 31 is data for image processing. The teacher data 311 is an image, i.e., a sequence of colors, brightness, etc. contained in the image. The teacher data 312 is a sequence of text that describes the image. (Example 5) The first teacher data 31 is data for anomaly detection. The teacher data 311 is a sequence of data acquired by a specific sensor. The teacher data 312 is a text sequence that represents normality or anomaly. (Example 6) The first teacher data 31 is data for anomaly prediction. The teacher data 311 is a sequence of data acquired by a specific sensor. The teacher data 312 is a text sequence that represents the possibility of a future anomaly occurring. (Example 7) The first teacher data 31 is data for demand forecasting. The teacher data 311 is a sequence of data related to product sales volume over a given period of time. The teacher data 312 is a text sequence that represents future product demand.

[0042] <Modification 3> The encoder-decoder model targeted in the first embodiment is a Transformer. Examples of Transformers include BERT, GPT, and T5. BERT is described in the literature: Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. GPT is described in the literature: Radford, A. , Narasimhan, K. , Salimans, T. , & Sutskever, I. (2018). Improving language understanding by generative pre-training. T5 is described in the literature: Raffel, C. , Shazeer, N. , Roberts, A. , Lee, K. , Narang, S. , Matena, M. , . .. .. & Liu, P. J. (2020). "Exploring the limits of transfer learning with a unified text-to-text transformer," in The Journal of Machine Learning Research, 21(1), 5485-5551.

[0043] <Modification 4> In the first embodiment, each functional component is realized by software. However, in a modification 4, each functional component may be realized by hardware. The following describes the differences between this modification 4 and the first embodiment.

[0044] When each functional component is realized by hardware, the learning and inference device 10 includes an electronic circuit instead of the processor 11 and the memory 12. The electronic circuit is a dedicated circuit that realizes the functions of each functional component, the memory 12, and the storage 13.

[0045] Possible electronic circuits include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, and an FPGA. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be realized by a single electronic circuit, or each functional component may be distributed across multiple electronic circuits.

[0046] <Modification 5> As a modification 5, some of the functional components may be realized by hardware, and other functional components may be realized by software.

[0047] The processor 11, the memory 12, and the electronic circuitry are collectively referred to as a processing circuit. That is, the functions of the respective functional components are realized by the processing circuit.

[0048] Embodiment 2. Embodiment 2 differs from embodiment 1 in that, when scores are sorted according to the order of multiple elements included in an input sequence, the filtering method is adjusted so that elements close to elements with large amounts of change in score are less likely to be removed. In embodiment 2, this difference will be explained, and explanations of the same points will be omitted.

[0049] ***Description of Configuration*** The configuration of the learning inference device 10 according to embodiment 2 will be described with reference to Fig. 13. The learning inference device 10 differs from the learning inference device 10 shown in Fig. 2 in that it includes a filtering adjustment unit 223 as a functional component. The function of the filtering adjustment unit 223, like the other functional components, is realized by software or hardware.

[0050] ***Description of Operation*** The filtering process according to the second embodiment will be described with reference to Fig. 14. Here, the filtering process of step S12 in Fig. 4 will be described. However, the filtering process of step S31 in Fig. 10 is similar. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate input filtered data, which is different.

[0051] The process of step S41 is the same as the process of step S21 in FIG.

[0052] (Step S42: Filtering Adjustment Process) When the scores calculated in step S41 are arranged according to the order of the multiple elements included in the teacher source data 311, the filtering adjustment unit 223 adjusts the filtering method so that elements close to elements with large score changes are less likely to be removed. In other words, the filtering adjustment unit 223 adjusts the filtering method so that elements before and after elements with rapid score changes are less likely to be removed. Specifically, the filtering adjustment unit 223 adjusts the filtering method so that a wider range of elements remain before and after elements with large score changes. For example, the filtering adjustment unit 223 calculates a moving average of the score for each element. In this case, the filtering adjustment unit 223 widens the range for calculating the moving average before and after elements with large score changes. Then, the filtering adjustment unit 223 causes the filtering execution unit 222 to filter the elements using the moving average value. As a result, the filtering adjustment unit 223 smooths the value used for filtering according to the score change. As a result, elements close to elements with large score changes are less likely to be removed.

[0053] The filtering adjustment unit 223 may specify the magnitude of the amount of change in the score by calculating a value used to detect a change point, such as a differential value indicating the amount of change in the value.

[0054] The filtering adjustment unit 223 only needs to make adjustments so that elements close to elements whose scores suddenly increase are less likely to be removed. In other words, the filtering adjustment unit 223 does not need to make adjustments so that elements close to elements whose scores suddenly decrease are less likely to be removed.

[0055] (Step S43: Filtering Execution Process) The filtering execution unit 222 performs filtering according to the filtering method adjusted in step S42. For example, as described in step S42, when the moving average of the scores for each element is calculated, the filtering execution unit 222 generates the teacher source filtering data 321 by excluding elements with the lowest moving averages at the reference ratio.

[0056] ***Effects of Embodiment 2*** As described above, the learning inference device 10 according to embodiment 2 adjusts the filtering method so that elements close to elements with large score changes are less likely to be removed. If elements close to elements with large score changes are removed, elements before and after the high-score element may be deleted, resulting in a loss of more context than necessary. The learning inference device 10 according to embodiment 2 can prevent elements before and after the high-score element from being deleted, thereby preventing a loss of more context than necessary. This makes it possible to prevent a deterioration in the quality of the generation results.

[0057] Embodiment 3. Embodiment 3 differs from embodiments 1 and 2 in that the score is adjusted by weighting the score with information other than the score. In embodiment 3, this difference will be explained, and explanation of the same points will be omitted. In embodiment 3, a case where a function is added to embodiment 1 will be explained. However, it is also possible to add a function to embodiment 2.

[0058] ***Description of Configuration*** The configuration of the learning inference device 10 according to embodiment 3 will be described with reference to Fig. 15. The learning inference device 10 differs from the learning inference device 10 shown in Fig. 2 in that it includes a score adjustment unit 224 as a functional component. The function of the score adjustment unit 224, like the other functional components, is realized by software or hardware.

[0059] ***Description of Operation*** The filtering process according to the third embodiment will be described with reference to Fig. 16. Here, the filtering process of step S12 in Fig. 4 will be described. However, the filtering process of step S31 in Fig. 10 is similar. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate input filtered data, which is different.

[0060] The process of step S51 is the same as the process of step S21 in FIG.

[0061] (Step S52: Score Adjustment Process) The score adjustment unit 224 adjusts the score calculated in step S41 by weighting it with information other than the score. The other information may be, for example, weight information for each element defined by the user. The other information may be statistical information such as TF-IDF. TF-IDF stands for Term Frequency-Inverse Document Frequency. As shown in Example 1 of FIG. 17 , the score adjustment unit 224 adjusts the score by weighting it using basic operations such as addition or multiplication using the score and the other information. In Example 1 of FIG. 17 , the score is adjusted by adding a weight, which is other information, to the score. Furthermore, as shown in Example 2 of FIG. 17 , the score adjustment unit 224 may adjust the score so that elements assigned a weight exceeding a threshold are not filtered regardless of the score. In Example 2 of FIG. 17 , the score is set to a value of ∞ so that elements whose weight, which is other information, exceeds a threshold of 2.0 are not filtered.

[0062] (Step S53: Filtering Execution Process) The filtering execution unit 222 performs filtering based on the score adjusted in step S52.

[0063] ***Effects of Embodiment 3*** As described above, the learning and inference device 10 according to Embodiment 3 adjusts the score by weighting the score using information separate from the score. This makes it possible to prevent unintended loss of context that occurs solely due to the scores of each element in the input sequence. This makes it possible to prevent deterioration in the quality of the generated results.

[0064] Embodiment 4. Embodiment 4 differs from Embodiments 1 to 3 in that a model is stably trained by performing curriculum learning in which the proportion of elements to be filtered is used as the difficulty level. In Embodiment 4, this difference will be explained, and explanations of the same points will be omitted. In Embodiment 4, a case where a function is added to Embodiment 1 will be explained. However, it is also possible to add a function to Embodiments 2 and 3.

[0065] ***Description of Operation*** The process of the learning phase according to the fourth embodiment will be described with reference to Fig. 4. The process of step S11 is the same as that of the first embodiment.

[0066] (Step S12: Filtering Process) The filtering unit 22 filters the teacher source data 311 included in the first teacher data 31 using the parameters of the first model 41 obtained by learning in step S11. At this time, the filtering unit 22 sets each of a plurality of proportions as a target proportion. Then, the filtering unit 22 performs filtering to remove elements of the target proportion from the elements included in the teacher source data 311, thereby generating teacher source filtering data 321 corresponding to the target proportion. Then, the filtering unit 22 generates second teacher data 32 including the teacher source filtering data 321 corresponding to the target proportion and the teacher destination data 312. For example, the filtering unit 22 sets 50%, 70%, and 90% as the target proportions. Then, the filtering unit 22 performs filtering to remove 50% of the elements included in the teacher source data 311, thereby generating teacher source filtering data 321 corresponding to 50%. The filtering unit 22 also filters the teacher source data 311 to remove 70% of the elements, thereby generating teacher source filtered data 321 corresponding to 70%. The filtering unit 22 also filters the teacher source data 311 to remove 90% of the elements, thereby generating teacher source filtered data 321 corresponding to 90%.

[0067] (Step S13: Filtering Data Learning Process) The filtering data learning unit 23 sets the teacher-source filtering data 321 corresponding to each of the multiple ratios as the target teacher-source filtering data 321 in ascending order of ratio. Then, the filtering data learning unit 23 receives as input the second teacher data 32 including the target teacher-source filtering data 321 and the teacher-destination data 312, and learns parameters of a second model 42 that converts the target teacher-source filtering data 321 into the teacher-destination data 312. For example, assume that second teacher data 32 has been generated for 50%, 70%, and 90%. In this case, as shown in FIG. 18 , the filtering data learning unit 23 first learns the parameters of the second model 42 using the second teacher data 32 corresponding to 50%. Next, the filtering data learning unit 23 learns the parameters of the second model 42 using the second teacher data 32 corresponding to 70%. Then, the filtering data learning unit 23 learns the parameters of the second model 42 using the second teacher data 32 corresponding to 90%.

[0068] During learning, the filtering data learning unit 23 uses the parameters of the second model 42 obtained by learning one stage earlier as the initial values ​​of the parameters of the second model 42. However, the filtering data learning unit 23 may use the parameters of the second model 42 obtained by learning two or more stages earlier as the initial values ​​of the parameters of the second model 42 during learning.

[0069] ***Effects of Embodiment 4*** As described above, the learning and inference device 10 according to Embodiment 4 performs curriculum learning in which the proportion of filtered elements determines the degree of difficulty. This allows for stepwise learning of differences between the pre-filtering teacher source data 311 and the filtered data. This allows for stable learning of the parameters of the second model 42.

[0070] Embodiment 5. Embodiment 5 differs from embodiments 1 to 4 in that the parameters of the second model are learned so that the score obtained from the calculation result by the attention mechanism in the first model 41 and the score obtained from the calculation result by the attention mechanism in the second model 42 become closer. In embodiment 5, this difference will be explained, and explanation of the same points will be omitted. In embodiment 5, a case where a function is added to embodiment 1 will be explained. However, it is also possible to add a function to embodiments 2 to 4.

[0071] ***Description of Operation*** The process of the learning phase according to the fifth embodiment will be described with reference to Fig. 4. The processes in steps S11 and S12 are the same as those in the first embodiment.

[0072] (Step S13: Filtering Data Learning Process) The filtering data learning unit 23 learns parameters of the second model 42 so that the score obtained from the calculation result by the attention mechanism in the first model 41 and the score obtained from the calculation result by the attention mechanism in the second model 42 are close to each other. Specifically, the filtering data learning unit 23 introduces a difference or similarity between the score obtained from the calculation result by the attention mechanism in the first model 41 and the score obtained from the calculation result by the attention mechanism in the second model 42 into a loss function used when learning the second model 42. Here, the score obtained from the calculation result by the attention mechanism in the first model 41 is the score calculated in step S21. In other words, the score obtained from the calculation result by the attention mechanism in the first model 41 is the score obtained by Attention, which is the attention mechanism of the first model 41. The score obtained from the calculation result by the attention mechanism in the second model 42 is the score calculated by the method described in step S21. In other words, the score obtained from the calculation result by the attention mechanism in the second model 42 is a score obtained by Attention, which is the attention mechanism of the second model 42.

[0073] For example, the filtering data learning unit 23 uses the mean square error MSE between the score of the first model 41 and the score of the second model 42 as the loss function. The mean square error MSE between the score of the first model 41 and the score of the second model 42 is expressed as in Equation 1. (Equation 1) MSE=(1 / N)Σ ｉ＝１ Ｎ (A ｉ -B ｉ ) ２ Here, N indicates the number of Attention elements. A indicates Attention of the first model 41. B indicates Attention of the second model 42. The Attention here may be Cross-Attention calculated by the multi-head attention mechanism of the decoder, or may be Self-Attention calculated by the multi-head attention mechanism of the encoder. Furthermore, the Attention here may be a combination of Cross-Attention and Self-Attention.

[0074] The above-described attention is to approximate the entire first model 41 and the second model 42. However, it is also possible to approximate a specific layer or a specific head.

[0075] Here, the mean square error (MSE) is used as the loss function, but other functions such as cross-entropy error or cosine similarity may also be used as the loss function.

[0076] ***Effects of Embodiment 5*** As described above, the learning and inference device 10 according to Embodiment 5 learns the parameters of the second model so that the score of the first model 41 and the score of the second model 42 become closer. This allows the second model 42 to be learned using not only the filtering results but also the internal state of the first model 41 used in the filtering as training data. This allows learning to proceed stably.

[0077] Sixth Embodiment The sixth embodiment differs from the first to fifth embodiments in that the teacher source data 311 is filtered in units of any length, and the results are combined to generate the teacher destination data 312. In the sixth embodiment, this difference will be explained, and explanations of the same points will be omitted. In the sixth embodiment, a case where a function is added to the first embodiment will be explained. However, it is also possible to add a function to the second to fifth embodiments.

[0078] ***Description of Configuration*** The configuration of the learning inference device 10 according to embodiment 6 will be described with reference to Fig. 19. The learning inference device 10 differs from the learning inference device 10 shown in Fig. 2 in that it includes a filtering data combination unit 225 as a functional component. The function of the filtering data combination unit 225, like the other functional components, is realized by software or hardware.

[0079] ***Description of Operation*** The filtering process according to the sixth embodiment will be described with reference to Fig. 20. Here, the filtering process of step S12 in Fig. 4 will be described. However, the filtering process of step S31 in Fig. 10 is similar. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate the teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate the input filtered data, which is different.

[0080] The process of step S61 is the same as the process of step S21 in FIG.

[0081] (Step S62: Filtering Execution Process) As shown in FIG. 21 , the filtering execution unit 222 sets each of the multiple split data 35 obtained by dividing the teacher source data 311 as the target split data 35. The filtering execution unit 222 filters elements included in the target split data 35 to generate split filtered data 36 corresponding to the target split data 35. In FIG. 21 , the filtering execution unit 222 divides the teacher source data 311 into two to generate two split data 35. Then, the filtering execution unit 222 filters each of the two split data 35 to generate two split filtered data 36. Note that the filtering execution unit 222 may divide the teacher source data 311 into pieces of any length. For example, the filtering execution unit 222 may divide the teacher source data 311 by topic. Alternatively, the filtering execution unit 222 may divide the teacher source data 311 into pieces of length based on the teacher source data 311.

[0082] 21 , the filtering data combining unit 225 combines the divided filtering data 36 corresponding to each of the plurality of divided data 35 generated in step S62 to generate teacher source filtering data 321. In FIG. 21 , the filtering data combining unit 225 combines the divided filtering data 36 corresponding to each of the two divided data 35 to generate the teacher source filtering data 321.

[0083] ***Effects of Embodiment 6*** As described above, the learning and inference device 10 according to Embodiment 6 combines the results obtained by filtering the teacher source data 311 in units of any length, to generate the teacher source filtered data 321. This makes it possible to reduce the computational cost of filtering while preventing loss of context, even in cases where the computational cost of filtering becomes high.

[0084] Embodiment 7. Embodiment 7 differs from embodiment 6 in that filtering and combining of filtered data are repeated until the length is equal to or less than the specified length. In embodiment 7, this difference will be explained, and explanation of the same points will be omitted.

[0085] ***Description of Operation*** The filtering process according to the seventh embodiment will be described with reference to Fig. 22. Here, the filtering process of step S12 in Fig. 4 will be described. However, the filtering process of step S31 in Fig. 10 is similar. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate the teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate the input filtered data, which is different.

[0086] The processes from step S71 to step S73 are the same as the processes from step S61 to step S63 in FIG.

[0087] (Step S74: Length Determination Process) The filtering data combination unit 225 determines whether the length of the teacher source filtering data 321 generated in step S73 is equal to or less than a specified length. If the length of the teacher source filtering data 321 is equal to or less than the specified length, the filtering data combination unit 225 ends the process. On the other hand, if the length of the teacher source filtering data 321 is not equal to or less than the specified length, the filtering data combination unit 225 returns the process to step S72.

[0088] As shown in FIG. 23 , when the process returns to step S72, the filtering execution unit 222 sets one or more pieces of split data 35 obtained by splitting the teacher-source filtering data 321 generated in step S73 as the target split data 35. The filtering execution unit 222 filters elements included in the target split data 35 to generate new split filtering data 36 corresponding to the target split data 35. Then, in step S73, the filtering data combining unit 225 combines the newly generated split filtering data 36 to generate new teacher-source filtering data 321. In FIG. 23 , split filtering data 36 is generated for each of the two split data 35 in the first filtering execution process, and the teacher-source filtering data 321 is generated. Then, in the second filtering execution process, new split filtering data 36 is generated for one piece of split data 35, and the split filtering data 36 is set as is as the teacher-source filtering data 321.

[0089] ***Effects of Embodiment 7*** As described above, the learning and inference device 10 according to Embodiment 7 repeats filtering and combining the filtered data until the length becomes equal to or less than the specified length. As a result, even when the input sequence is long, it is possible to generate teacher-source filtered data 321 of the intended length while preventing loss of context and suppressing the computational cost associated with filtering.

[0090] Embodiment 8. Embodiment 8 differs from embodiments 1 to 7 in that a special token is assigned to the elements removed by filtering. In embodiment 5, this difference will be explained, and explanation of the same points will be omitted. In embodiment 8, a case where a function is added to embodiment 1 will be explained. However, it is also possible to add a function to embodiments 2 to 7.

[0091] ***Description of Configuration*** The configuration of the learning inference device 10 according to embodiment 8 will be described with reference to Fig. 24. The learning inference device 10 differs from the learning inference device 10 shown in Fig. 2 in that it includes a filtering correction unit 226 as a functional component. The function of the filtering correction unit 226, like the other functional components, is realized by software or hardware.

[0092] ***Description of Operation*** The filtering process according to the eighth embodiment will be described with reference to Fig. 25. Here, the filtering process of step S12 in Fig. 4 will be described. However, the filtering process of step S31 in Fig. 10 is similar. In step S12 in Fig. 4, the teacher source data 311 is filtered to generate the teacher source filtered data 321, but in step S31 in Fig. 10, the input data 33 is filtered to generate the input filtered data, which is different.

[0093] The process of step S81 is the same as the process of step S21 in FIG.

[0094] (Step S82: Filtering Execution Process) The filtering execution unit 222 filters the elements included in the teacher source data 311 based on the score calculated in step S21, to generate teacher source filtered data 321. At this time, as shown in Fig. 26, the filtering execution unit 222 assigns a special token to the elements removed by filtering. In Fig. 26, as in Fig. 9, input 2 and input 4 have been removed, and teacher source filtered data 321 including input 1 and input 3 has been generated. Therefore, the filtering execution unit 222 assigns a special token [M] to the removed input 2 and input 4.

[0095] (Step S83: Filtering Correction Process) The filtering correction unit 226 outputs the results of the filtering performed in step S82 to the display device 14. At this time, the filtering correction unit 226 indicates the elements removed by filtering with a special token [M] and then displays each element of the teacher source data 311. This allows the user to understand which elements have been removed. The filtering correction unit 226 then accepts designation of elements that the user considers inappropriate for removal. As shown in FIG. 27 , the filtering correction unit 226 stops removing elements designated as inappropriate for removal and adds them to the teacher source filtering data 321. In FIG. 27 , input 2 has been designated as inappropriate for removal, and has therefore been added to the teacher source filtering data 321.

[0096] ***Effects of Embodiment 8*** As described above, the learning and inference device 10 according to Embodiment 8 assigns a special token to elements removed by filtering. This allows the user to easily identify elements removed by filtering.

[0097] Furthermore, in the learning and inference device 10 according to the eighth embodiment, the filtering correction unit 226 adds an element designated by the user to the teacher-source filtering data 321. This allows the user to correct the filtered results and obtain desirable teacher-source filtering data 321.

[0098] In addition, the word "unit" in the above description may be read as a "circuit," "step," "procedure," "process," or "processing circuit."

[0099] The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be combined and implemented. Furthermore, one or more of them may be implemented partially. Note that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as needed.

[0100] 10 Learning and inference device, 11 Processor, 12 Memory, 13 Input device, 14 Display device, 15 External interface, 16 Communication interface, 17 Recording medium, 21 Normal data learning unit, 22 Filtering unit, 221 Score calculation unit, 222 Filtering execution unit, 223 Filtering adjustment unit, 224 Score adjustment unit, 225 Filtered data combination unit, 226 Filtering correction unit, 23 Filtered data learning unit, 24 Filtered data inference unit, 31 First teacher data, 311 Teacher source data, 312 Teacher destination data, 32 Second teacher data, 321 Teacher source filtered data, 33 Input data, 34 Output data, 35 Split data, 36 Split filtered data, 41 First model, 42 Second model, 101 Learning device, 102 Inference device.

Claims

1. A learning device comprising: a filtering unit that uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate scores for each of a plurality of elements included in first teacher data, which includes source teacher data that is source sequence data and destination teacher data that is destination sequence data, and filters the plurality of elements included in the source teacher data based on the scores for each of the plurality of elements to generate source teacher filtered data; and a filtering data learning unit that uses as input second teacher data that includes the source teacher filtered data and the destination teacher data generated by the filtering unit and learns parameters of a second model that converts the source teacher filtered data into the destination teacher data.

2. The learning device of claim 1, wherein the filtering unit comprises: a score calculation unit that calculates the score for each of the plurality of elements; a filtering adjustment unit that adjusts the filtering method so that, when the scores calculated by the score calculation unit are arranged in accordance with the order of the plurality of elements included in the teacher source data, elements close to elements with large changes in the score are less likely to be removed; and a filtering execution unit that filters in accordance with the filtering method adjusted by the filtering adjustment unit.

3. The learning device according to claim 1 or 2, wherein the filtering unit comprises: a score calculation unit that calculates the score for each of the plurality of elements; a score adjustment unit that adjusts the score calculated by the score calculation unit by weighting the score with information separate from the score; and a filtering execution unit that performs filtering based on the score adjusted by the score adjustment unit.

4. The learning device described in any one of claims 1 to 3, wherein the filtering unit takes each of a plurality of ratios as a target ratio and filters the elements included in the teacher source data to remove elements of the target ratio, thereby generating the teacher source filtered data corresponding to the target ratio; and the filtering data learning unit sets the teacher source filtered data corresponding to each of the plurality of ratios as target teacher source filtered data in order from the lowest ratio, and uses second teacher data including the target teacher source filtered data and the teacher destination data as input, and learns parameters of a second model that converts the target teacher source filtered data into the teacher destination data.

5. A learning device described in any one of claims 1 to 4, wherein the filtering data learning unit learns the parameters of the second model so that the score obtained from the calculation result by the attention mechanism in the first model and the score obtained from the calculation result by the attention mechanism in the second model become closer to each other.

6. The learning device described in any one of claims 1 to 5, wherein the filtering unit treats each of the multiple split data obtained by splitting the teacher source data as target split data, filters elements contained in the target split data, and generates split filtered data corresponding to the target split data; and the learning device further comprises a filtering data combining unit that combines the split filtered data corresponding to each of the multiple split data to generate the teacher source filtered data.

7. The learning device described in claim 6, wherein, when the teacher source filtering data is longer than a specified length, the filtering unit uses one or more split data obtained by splitting the teacher source filtering data as target split data, filters elements contained in the target split data, and generates new split filtering data corresponding to the target split data; and the filtering data combining unit combines the newly generated split filtering data to generate new teacher source filtering data.

8. A learning device as described in any one of claims 1 to 7, wherein the filtering unit assigns a special token to the element removed by filtering, and the learning device further comprises a filtering correction unit that returns the specified special token to the original element.

9. An inference device comprising: a filtering unit that uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate a score for each of a plurality of elements included in input data that is source sequence data, and generates input filtered data by filtering the plurality of elements included in the input data based on the scores for each of the plurality of elements; and a filtered data inference unit that uses a second model that infers destination sequence data from source sequence data to infer output data from the input filtered data generated by the filtering unit.

10. The inference device described in claim 9, wherein the filtering unit comprises: a score calculation unit that calculates the score for each of the plurality of elements; a filtering adjustment unit that adjusts the filtering method so that, when the scores calculated by the score calculation unit are arranged according to the order of the plurality of elements included in the input data, elements near elements with large changes in the score are less likely to be removed; and a filtering execution unit that filters according to the filtering method adjusted by the filtering adjustment unit.

11. An inference device as described in claim 9 or 10, wherein the filtering unit comprises: a score calculation unit that calculates the score for each of the plurality of elements; a score adjustment unit that adjusts the score calculated by the score calculation unit by weighting the score with information separate from the score; and a filtering execution unit that performs filtering based on the score adjusted by the score adjustment unit.

12. An inference device described in any one of claims 9 to 11, wherein the second model is obtained by learning parameters using as inputs second teacher data including teacher source filtered data generated by filtering the plurality of elements included in first teacher data including teacher source data, which is sequence data to be converted, and teacher destination data, which is sequence data to be converted.

13. The inference device described in claim 12, wherein the second model is obtained by filtering elements included in the teacher source data to remove elements of the target proportions, with each of a plurality of proportions being a target proportion, to generate the teacher source filtered data corresponding to the target proportions, and by using the teacher source filtered data corresponding to each of the plurality of proportions as target teacher source filtered data in order from the lowest proportion, and by learning parameters using teacher data including the target teacher source filtered data and the teacher destination data as input.

14. An inference device as described in claim 12 or 13, wherein the second model is obtained by learning parameters so that the score obtained from the calculation results by the attention mechanism in the first model and the score obtained from the calculation results by the attention mechanism in the second model are close to each other.

15. The inference device described in any one of claims 9 to 14, wherein the filtering unit treats each of the multiple split data obtained by splitting the input data as target split data, filters elements contained in the target split data, and generates split filtering data corresponding to the target split data; and the inference device further comprises a filtering data combination unit that combines the split filtering data corresponding to each of the multiple split data to generate the input filtering data.

16. The inference device described in claim 15, wherein, when the input filtering data is longer than a specified length, the filtering unit uses each of the multiple split data obtained by splitting the input filtering data as target split data, filters elements contained in the target split data, and generates new split filtering data corresponding to the target split data, and the filtering data combining unit combines the newly generated split filtering data to generate new input filtering data.

17. An inference device as described in any one of claims 9 to 16, wherein the filtering unit assigns a special token to the element filtered out, and the inference device further comprises a filtering correction unit that returns the specified special token to the original element.

18. A learning method in which a computer uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate scores for each of a plurality of elements included in first training data, the first training data including source data, which is source sequence data, and destination data, which is destination sequence data, and filters the plurality of elements included in the source data based on the scores for each of the plurality of elements to generate source filtered data; and the computer uses as input second training data including the source filtered data and the destination data to learn parameters of a second model that converts the source filtered data into the destination data.

19. A learning program that causes a computer to function as a learning device that performs the following steps: a filtering process that uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate scores for each of a plurality of elements included in the teacher source data in first teacher data, which includes teacher source data that is source sequence data and teacher destination data that is destination sequence data, and filters the plurality of elements included in the teacher source data based on the scores for each of the plurality of elements to generate teacher source filtered data; and a filtering data learning process that uses as input second teacher data that includes the teacher source filtered data generated by the filtering process and the teacher destination data, and learns parameters of a second model that converts the teacher source filtered data into the teacher destination data.

20. An inference method in which a computer uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate the score of each of multiple elements included in input data, which is source sequence data, and filters the multiple elements included in the input data based on the scores of each of the multiple elements to generate input filtered data; and a computer uses a second model that infers destination sequence data from source sequence data to infer output data from the input filtered data.

21. An inference program that causes a computer to function as an inference device that performs a filtering process that uses an attention mechanism in a first model that infers destination sequence data from source sequence data to calculate the score of each of multiple elements included in input data, which is source sequence data, and filters the multiple elements included in the input data based on the scores of each of the multiple elements to generate input filtered data; and a filtered data inference process that uses a second model that infers destination sequence data from source sequence data to infer output data from the input filtered data generated by the filtering process.

Images