Learning method, inference method, learning device, inference device, and program

Abstract

To allow for calculating a value according to the differences between domains about token importance.SOLUTION: A learning device is configured to: calculate an appearance probability vector where an appearance probability of each token included in a text aggregate of a first domain serves as a component; calculate a vector series about each of a correct-answer text and an incorrect-answer text when a text corresponding to a query is searched from the aggregate; calculate a sparse vector where each token importance in a preliminarily learned language model serves as a component on the basis of the vector series and the appearance probability vector about each of the correct-answer text and the incorrect-answer text; calculate a matching score between the sparse vector and the query about each of the correct-answer text and the incorrect-answer text; and update the parameters of the preliminarily learned language model used in calculation of the vector series and the sparse vector on the basis of the matching score of each of the correct-answer text and the incorrect-answer text.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a learning method, an inference method, a learning device, an inference device, and a program. [Background technology]

[0002] A search task is a task in which a query in natural language is input and a specified number of texts related to the query are output from a corpus, which is a collection of texts to be searched.

[0003] SPARTA in Non-Patent Document 1 can be applied to any search task, and focuses in particular on the case where the domain (field, topic, etc.) of the corpus during training and inference is different. During training, the service provider trains the model using data from a general domain (source domain). The model user does not train the model for reasons such as not having sufficient training data, or not wanting to place the corpus in the cloud for model training because it is highly confidential. The corpus owned by the user (target domain) is searched only during inference.

[0004] SPARTA uses BERT (Non-Patent Document 2) to convert each text s in the corpus into a sparse vector. If the vocabulary size of BERT is V, the sparse vector is a V-dimensional vector. The v-th component of the sparse vector corresponds to the v-th token in the vocabulary. The importance of token v in text s becomes the value of the v-th component of the sparse vector.

[0005] By storing sparse vectors as an inverted index, it is possible to perform a fast search when a query is given. There are various existing researches and implementations of fast searches using inverted indexes.

[0006] Let i be the position index in text s, and H(s) i Let be the output of BERT for the i-th token of text s, v be a token in vocabulary V, and e vLet be the word embedding of token v in BERT, then the v-th component of the sparse vector mentioned above can be defined as follows:

[0007]

number

[0008] When a query q is input, the matching score between the query q and the text s can be defined by the following equation (2).

[0009]

number

[0010] [Non-Patent Document 1] Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee, "SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval", In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 565-575, [online], Internet<URL:https: / / doi.org / 10.18653 / v1 / 2021.naaclmain.47> [Non-Patent Document 2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, 4171-4186, [online], Internet<URL:https: / / doi.org / 10.18653 / v1 / N19-1423> Summary of the Invention [Problem to be solved by the invention]

[0011] In a setting where the domains of the training and inference corpora are different, equation (1) should reflect rarity in the target domain: we want to assign small values ​​to tokens that are common in the target domain and large values ​​to tokens that are rare in the target domain, regardless of their frequency in the source domain.

[0012] However, the technology of Non-Patent Document 1 does not take into account differences in domains.

[0013] The present invention has been made in consideration of the above-mentioned points, and has an object to make it possible to calculate a value of the importance of a token according to differences in domain. [Means for solving the problem]

[0014] In order to solve the above problem, a computer executes the following steps: a token probability calculation step for calculating an occurrence probability vector whose components are the occurrence probability of each token included in a set of texts in a first domain; a language encoding step for calculating a vector series for each of correct answer text and incorrect answer text when searching for text corresponding to a certain query from the set; a vector calculation step for calculating a sparse vector whose components are the importance of each token in a pre-trained language model for each of the correct answer text and the incorrect answer text, based on the vector series and the occurrence probability vector; a matching calculation step for calculating a matching score between the sparse vector and the query for each of the correct answer text and the incorrect answer text; and a parameter update step for updating parameters of the pre-trained language model used by the language encoding step and the vector calculation step, based on the matching scores for each of the correct answer text and the incorrect answer text. Effect of the Invention

[0015] It is possible to calculate the importance of a token according to differences in domain. [Brief description of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of a learning device 10 according to an embodiment of the present invention. [Diagram 2] 1 is a diagram illustrating an example of a functional configuration of a learning device 10 according to an embodiment of the present invention. [Diagram 3] 1 is a flowchart illustrating an example of a processing procedure executed by the learning device 10. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of the inference device 20 when generating an inverted index table. [Diagram 5] 11 is a flowchart illustrating an example of a processing procedure for generating a transposed index table. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of the inference device 20 when a search is executed. [Figure 7] 13 is a flowchart illustrating an example of a procedure for a search process. [Figure 8] FIG. 13 shows data statistics and cross-domain evaluation results. [Figure 9] FIG. 1 shows the sentences and contexts used as input in an experiment and the top 10 tokens based on matching scores. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the hardware configuration of a learning device 10 in an embodiment of the present invention. The learning device 10 in Fig. 1 has a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, an interface device 105, and the like, which are all connected to each other via a bus B.

[0018] The program for implementing the processing in the learning device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 via the drive device 100 into the auxiliary storage device 102. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, and the like.

[0019] When an instruction to start a program is received, the memory device 103 reads out and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the learning device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0020] 2 is a diagram showing an example of a functional configuration of a learning device 10 in an embodiment of the present invention. During training (learning), the learning device 10 has a token probability calculation unit 11, a language encoding unit 12, a vector calculation unit 13, a matching calculation unit 14, a loss calculation unit 15, and a parameter update unit 16. Each of these units is realized by a process in which one or more programs installed in the learning device 10 are executed by a processor 104.

[0021] The token probability calculation unit 11 inputs a set of texts (a set of token sequences) and quantifies the rarity of tokens in the set of texts. Specifically, the token probability calculation unit 11 calculates an occurrence probability vector p d Calculate p dIn this embodiment, the text is a character string of any length. For example, the text may be divided into sentences or other units. d The length (number of components) of is V. V is the number of vocabulary words (number of types of tokens) in BERT, which is a pre-trained language model used by the language encoding unit 12 and the vector calculation unit 13. v is the v-th token in the pre-trained language model. The pre-trained language model employs subwords to separate tokens. Subwords are a way of separating tokens that are (often) shorter than words, and unknown words are always divided into subwords when they appear. For example, "unhappy" is divided into "un" and "happy". Therefore, the vocabulary is determined by the pre-trained language model and is not influenced by the corpus. In other words, the vector p d The number of dimensions of the sparse vector is the same during training and during inference. In this embodiment, an example in which BERT is used as a pre-trained language model will be described, but any pre-trained language model other than BERT may be used.

[0022] Furthermore, the occurrence probability of token v basically means the number of times token v appears in a text set divided by the total number of tokens in the text set. However, smoothing may be performed. Smoothing is a technique that increases the occurrence probability of all tokens by +1 or +0.1 to prevent the occurrence of a token with an occurrence probability of 0. In this case, the denominator is also increased by +V or +V / 10.

[0023] The language encoding unit 12 is a model (e.g., a neural network) that inputs text (a token sequence) and outputs a vector sequence as a text feature. Any model that satisfies such input / output conditions can be used, and the language encoding unit 12 may be realized using a pre-trained language model (BERT), for example. In this embodiment, the output of the language encoding unit 12 to which text s is input is represented by H(s).

[0024] The vector calculation unit 13 calculates the importance of a token by taking into account the rarity of the token. Specifically, the vector calculation unit 13 calculates the importance of a token by taking into account the rarity of the token. d (vector) and output a sparse vector (vector) based on the following equation (3) using a pre-trained language model (BERT).

[0025]

number

[0026] The vector calculation unit 13 outputs the sparse vector as is during training, and outputs only the K components with large values ​​during inference.

[0027] The target corpus (the source domain during training, the target domain during inference) is the rare (p d,v The smaller the score (f STT (v,s;p d )) will be larger, and the more common the token, the smaller the score. However, p d,v p d is the vth component of

[0028] The matching calculation unit 14 inputs a sparse vector (vector) and a query q (token sequence) in a natural language, and calculates and outputs a matching score, which is a scalar value, based on the following formula (2).

[0029]

number

[0030] In other words, the sum of the components of the sparse vector that correspond to the tokens that make up the query q is the matching score. If the same token appears more than once in the query, the component of the sparse vector that corresponds to that token may be added either once or the number of times that token appears.

[0031] The loss calculation unit 15 calculates a loss function value based on the matching score using a method commonly used in neural networks. For example, the loss calculation unit 15 calculates a CrossEntropy loss so that the correct text has a large matching score and the incorrect text has a small matching score.

[0032] The parameter update unit 16 calculates a gradient based on the loss function value, and updates the parameters of the pre-trained language model used by the language encoding unit 12 and the vector calculation unit 13. The parameter update generally uses the backpropagation method.

[0033] FIG. 3 is a flowchart illustrating an example of a processing procedure executed by the learning device 10. As shown in FIG.

[0034] In step S101, the token probability calculation unit 11 calculates one token probability p d (vector). The training data is a set of pairs of queries and ground truth texts. The ground truth texts are in the source domain D sIn step S101, a set of all pairs of correct texts is treated as one corpus, and a token probability p d (vector) is calculated. Note that the domain means a field or topic.

[0035] Next, the language encoding unit 12 samples a pair of a query and a supervised text from the training data (S102). Hereinafter, the sampled query and supervised text are referred to as a "target query" and a "target text", respectively.

[0036] Next, the language encoding unit 12 calculates text features of the target text and text features of text different from the target text based on the pre-trained language model (BERT), and outputs each text feature (S103). Here, the text different from the target text refers to text that does not correspond to the target query (hereinafter, referred to as "incorrect text"). For example, the incorrect text may be randomly selected from a set other than the set to which the target query belongs in the training data.

[0037] Next, the vector calculation unit 13 calculates the parameters (e v ), for each text feature (target text, incorrect answer text) calculated in step S103, the token probability p d and the text feature quantity, and output the sparse vector (S104). Hereinafter, the sparse vector for the target text is referred to as a "correct sparse vector," and the sparse vector for the incorrect text is referred to as an "incorrect sparse vector."

[0038] Next, the matching calculation unit 14 calculates a matching score between the sparse vector and the target query for each sparse vector (correct sparse vector, incorrect sparse vector) calculated in step S104 based on equation (2), and outputs the matching score (S105).

[0039] Next, the loss calculation unit 15 calculates a loss function value using the matching scores of the correct sparse vector and the incorrect sparse vector (S106). The loss function value may be calculated based on a general loss function in a neural network.

[0040] Next, the parameter update unit 16 updates the parameters of the pre-trained language model (BERT) using the loss function value (S107).

[0041] For example, the loss function can be the CrossEntropy loss. When the CrossEntropy loss is used, learning is performed so that the matching score of the target text is relatively larger than the matching score of the incorrect text.

[0042] Steps S102 to S107 are repeated a prescribed number of times (S108). When steps S102 to S107 have been executed a prescribed number of times (Yes in S108), the processing procedure in FIG.

[0043] Next, the inference process will be described. Fig. 4 is a diagram showing an example of the functional configuration of the inference device 20 when generating an inverted index table. In Fig. 4, the same parts as in Fig. 2 are given the same reference numerals and their description will be omitted.

[0044] In FIG. 4, the inference device 20 includes a token probability calculation unit 11, a language encoding unit 12, a vector calculation unit 13, and a transposed index generation unit 17. Each of these units is realized by a process in which one or more programs installed in the inference device 20 are executed by a processor of the inference device 20. The inference device 20 may have the same hardware configuration as that of FIG. 2. The inference device 20 may be realized using the same computer as the learning device 10, or may be realized using a different computer. For example, the learning device 10 may be a computer used by a service provider, and the inference device 20 may be a computer that holds a corpus c1 and is used by a user of the inference device 20.

[0045] The transposed index generation unit 17 associates the sparse vector calculated based on the token probability calculation unit 11, the trained language encoding unit 12, and the trained vector calculation unit 13 with the ID of the text for each text included in the corpus c1 and stores them in the transposed index table t1.

[0046] Corpus c1 is a target domain D t The target domain Dt is a set of texts in the source domain S t It is a different domain.

[0047] FIG. 5 is a flowchart illustrating an example of a processing procedure for generating an inverted index table.

[0048] In step S201, the token probability calculation unit 11 calculates a target domain D t For corpus c1, token probability p d Calculate (vector).

[0049] Next, the language encoding unit 12 extracts (samples) one text (hereinafter, referred to as "target text s") from the corpus c1 (S202).

[0050] Next, the language encoding unit 12 calculates text features H(s) of the target text s based on the pre-trained language model (BERT) that has been trained (with updated parameters) by the training device 10, and outputs the text features H(s) (S203).

[0051] Next, the vector calculation unit 13 calculates the text feature H(s) and the parameters (e v ) and the token probability p calculated in step S201 d and are substituted into equation (3) to calculate a sparse vector (S204).

[0052] Next, the transposed index generating unit 17 stores the sparse vector in the transposed index table t1 in association with the ID of the target text (S205). That is, the transposed index table t1 is a set of pairs of sparse vectors and IDs of texts (hereinafter, referred to as "text IDs"). The text ID is information for identifying each text in the corpus c1.

[0053] When steps S202 to S205 have been executed for all the texts in the corpus c1 (Yes in S206), the processing procedure in FIG. 5 ends.

[0054] Fig. 6 is a diagram showing an example of the functional configuration of the inference device 20 when a search is performed. In Fig. 6, the same parts as those in Fig. 2 or Fig. 4 are given the same reference numerals, and their explanation will be omitted. In Fig. 6, the inference device 20 has a search unit 18. The search unit 18 is realized by a process in which one or more programs installed in the inference device 20 are executed by a processor of the inference device 20.

[0055] The search unit 18 inputs the inverted index table t1 (a set of sparse vectors) and a query (a token sequence) in a natural language, and outputs text corresponding to the query in the corpus c1. Specifically, the search unit 18 uses the matching calculation unit 14 to output text corresponding to a sparse vector that maximizes the matching score with the query based on the inverted index.

[0056] FIG. 7 is a flowchart illustrating an example of a processing procedure of the search process.

[0057] In step S301, the search unit 18 accepts an input of a query (hereinafter, referred to as a "target query").

[0058] Next, the search unit 18 acquires a set of pairs of a sparse vector corresponding to the target query (hereinafter, referred to as a "corresponding sparse vector") and a text ID from the inverted index table t1 (S302). The sparse vector corresponding to the target query is a sparse vector whose components corresponding to any one or more tokens constituting the target query are greater than 0.

[0059] Next, the search unit 18 inputs the target query and the corresponding sparse vector for each corresponding sparse vector to the matching calculation unit 14, and the matching calculation unit 14 calculates a matching score between the target query and the corresponding sparse vector and outputs the matching score (S303). As a result, the search unit 18 obtains a matching score for each corresponding sparse vector.

[0060] Next, the search unit 18 searches the target domain D for the text corresponding to the text id that is a pair with the corresponding sparse vector with the maximum matching score. t is obtained from corpus c1 (S304).

[0061] Next, the search unit 18 outputs the acquired text as a search result corresponding to the query (S305). [experiment] Next, an experiment conducted by the present inventor regarding this embodiment will be described.

[0062] <Datasets and metrics> We used MultiReQA (Reference 1, Non-Patent Document 2) for the experiments. MultiReQA is created from MRQA, a cross-domain QA dataset, by splitting passages into sentences. Following Reference 1, we removed four domains from the 12 domains in the original MRQA dataset because most of the questions become unclear when separated from their original context. The bibliographic information for each reference is provided below.

[0063] However, we found that the previously published MultiReQA underestimated the model's performance. For example, the performance in the biomedical domain was reported to be less than 20 points with MRR@10. We observed that identical questions associated with different sentences (e.g., "Q: What type of enzyme is peroxiredoxin 2 (PRDX2)?" and "A: Antioxidant") caused underestimation. Because these are factual questions, the answers of all the sentences connected should be allowed. We solved this problem by merging the identical questions and their answers into one instance. Furthermore, we removed two domains.

[0064] As a result, the version of MultiReQA used in this experiment contains six domains. Figure 8 shows the data statistics and cross-domain evaluation results. The neural model is trained on the SQuAD dataset, and the datasets in parentheses are not used for training. The scores indicate MRR / Recall@1. The training and evaluation splits of SQuAD were used as the training dataset and the development dataset for early stopping, respectively. The evaluation splits of the other datasets were used as the test dataset. In addition, the training splits except for SQuAD were used as the development dataset for the hyperparameter search common to all domains.

[0065] <Implementation> Following Non-Patent Document 1, we concatenate the original sentence and the surrounding context within the sentence as the model input s. The input is truncated to the first 256 tokens. We use BERT's segment ID to distinguish between sentences and contexts.

[0066] As candidates for negative sentences (incorrect texts) for the question, we used the sentences that were linked, excluding the correct sentences. d We used the top 100 sentences obtained from BM25 (Reference 2) from the original answer of the question. We excluded sentences that contained the original answer of the question from the negative sentence candidates. At each step, we sampled one negative sentence from each candidate.

[0067] As encoders, we used the pre-trained DistilBERT-base-uncased model (66 million parameters) (Reference 3) and the BERT-base-uncased model (110 million parameters) (Reference 4). STT (s,v)} v∈V completed in 0.9 and 1.5 seconds per sentence using DistilBERT and BERT on a MacBook Air (M1, 16 GB memory). PyTorch (ver. 1.12.1) (Ref. 5) and Transformer (ver. 4.21.1) (Ref. 6) were used. The model was trained for 50 epochs with a patience of 3 for early stopping. The Adam optimizer (Ref. 7) was used with a learning rate of 1e-5, linear learning rate decay, and 500 warm-up steps. Four NVIDIA Quadro RTX8000 (48 GB) GPUs were used. The batch size was 96 for DistilBERT-based and 48 for BERT-based. Gradient caching (Ref. 8) was used to backpropagate in-batch negative gradients across multiple GPUs. λ in Eq. (3) was set to 0.1. In accordance with Non-Patent Document 1, we set K, the number of components to be stored in the transposed index as a sparse vector during inference, to 2000.

[0068] <Result> As shown in Figure 8, this embodiment (STATIC) outperformed the conventional technology SPARTA. This embodiment can model the importance of tokens in the inference corpus, so performance improved when evaluating in the source domain in addition to the target domain. The performance improvement in Natural Questions was large, but this is thought to be because Natural Questions is written in HTML, which makes it possible to suppress the negative effects of HTML tags.

[0069] <Analysis> Fig. 9 shows the sentence and context used as input in the experiment and the top 10 tokens with the highest matching scores. In Fig. 9, when [Sentence and Context] is set as text s in the medical domain, the matching scores (f(v,s) for the conventional technology (SPARTA) and f(v,s) for this embodiment (STATIC) are shown. STT (v,s,p d )) The top 10 tokens v and their matching scores are shown.

[0070] In the conventional technology, high scores are assigned to tokens that are common in the medical domain, such as ##otype and syndrome, whereas in the present embodiment, it can be seen that high scores are assigned to important words in this sentence, such as marker and turner.

[0071] As described above, according to the present embodiment, it is possible to calculate a value of the importance of a token according to the difference in domain. As a result, it is possible to calculate a matching score that takes the difference in domain into consideration, and it is possible to perform a high-performance text search even if the domains to which the text sets used during training and learning belong are different.

[0072] [References] [References 1] Mandy Guo, Yinfei Yang, Daniel Cer, Qinlan Shen, and Noah Constant. 2021. MultiReQA [Reference 2] Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework [Reference 3] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, [Reference 4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, 4171-4186. https: / / doi.org / 10.18653 / v1 / N19-1423 [Reference 5] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory [Reference 6] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, 38-45. https: / / www.aclweb.org / anthology / 2020.emnlp-demos.6 [Reference 7] Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization [Reference 8] Jiawei Han, Luyu Gao, Yunyi Zhang and Jamie Callan. 2021. Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup. In Proceedings of the 6th Workshop on Representation Learning for NLP. Although the embodiment of the present invention has been described in detail above, the present invention is not limited to such specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0073] 10 Learning Device 11 Token Probability Calculation Unit 12 Language encoding section 13 Vector Calculation Section 14 Matching Calculation Section 15 Loss calculation section 16 Parameter Update Section 17 Transposed index generator 18 Search Department 20 Reasoning device 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory device 104 processors 105 Interface device B Bus

Claims

1. a token probability calculation step for calculating an occurrence probability vector having components each representing the occurrence probability of each token included in a set of texts of a first domain; a language encoding procedure for computing a vector sequence for each correct and incorrect text when searching for text corresponding to a certain query from said set; a vector calculation step of calculating a sparse vector having components each representing the importance of each token in a pre-trained language model for each of the correct text and the incorrect text based on the vector sequence and the occurrence probability vector; a matching calculation step of calculating a matching score between the sparse vector and the query for each of the correct text and the incorrect text; a parameter update step of updating parameters of a pre-trained language model used in the language encoding step and the vector calculation step based on the matching scores of the correct text and the incorrect text; A learning method characterized in that the above is executed by a computer.

2. The vector calculation step is to calculate a vector corresponding to a token v in the pre-trained language model from the vector sequence of the text s by H(s) ｉ Let v be the word embedding of the token v, which is the parameter of the pre-trained language model, e ｖ Let the occurrence probability vector be p ｄ In this case, the component of the sparse vector corresponding to the token v is expressed as [0010] Calculate based on 2. The learning method according to claim 1 .

3. a token probability calculation step of calculating an occurrence probability vector having components each representing the occurrence probability of each token included in a set of texts of a second domain; a linguistic encoding procedure for computing a sequence of vectors for each of said texts; a vector calculation step of calculating, for each of the texts, a sparse vector having components each representing the importance of each token in a pre-trained language model based on the vector sequence and the occurrence probability vector; an inverted index generation step of storing the sparse vectors for each text in an inverted index table; The computer executes The language encoding step and the vector calculation step utilize a pre-trained language model whose parameters have been updated by the learning method according to claim 1.

13. An inference method comprising:

4. a matching calculation step for calculating a matching score between an input query and the sparse vectors stored in the inverted index table; a searching step of outputting the text corresponding to the sparse vector based on the matching score; 4. The inference method according to claim 3, wherein the inference method is executed by a computer.

5. a token probability calculation unit configured to calculate an occurrence probability vector having components each representing an occurrence probability of each token included in a set of texts of a first domain; a language encoding unit configured to calculate a vector sequence for each of a correct text and an incorrect text when searching the set for a text corresponding to a certain query; a vector calculation unit configured to calculate, for each of the correct text and the incorrect text, a sparse vector having the importance of each token in a pre-trained language model as a component based on the vector sequence and the occurrence probability vector; a matching calculation unit configured to calculate a matching score between the sparse vector and the query for each of the correct text and the incorrect text; a parameter update unit configured to update parameters of a pre-trained language model used by the language encoding unit and the vector calculation unit based on the matching scores of the correct text and the incorrect text; A learning device comprising:

6. a token probability calculation unit configured to calculate an occurrence probability vector having components each representing an occurrence probability of each token included in a set of texts of a second domain; a linguistic encoder configured to compute a sequence of vectors for each of said texts; a vector calculation unit configured to calculate, for each of the texts, a sparse vector having components each representing the importance of each token in a pre-trained language model based on the vector sequence and the occurrence probability vector; an inverted index generator configured to store the sparse vector for each text in an inverted index table; having The language encoding unit and the vector calculation unit are configured to use a pre-trained language model whose parameters have been updated by the learning device according to claim 5. An inference device comprising:

7. a token probability calculation step for calculating an occurrence probability vector having components each representing the occurrence probability of each token included in a set of texts of a first domain; a language encoding procedure for computing a vector sequence for each correct and incorrect text when searching for text corresponding to a certain query from said set; a vector calculation step of calculating a sparse vector having components each representing the importance of each token in a pre-trained language model for each of the correct text and the incorrect text based on the vector sequence and the occurrence probability vector; a matching calculation step of calculating a matching score between the sparse vector and the query for each of the correct text and the incorrect text; a parameter update step of updating parameters of a pre-trained language model used in the language encoding step and the vector calculation step based on the matching scores of the correct text and the incorrect text; A program characterized by causing a computer to execute the above.

8. a token probability calculation step of calculating an occurrence probability vector having components each representing the occurrence probability of each token included in a set of texts of a second domain; a linguistic encoding procedure for computing a sequence of vectors for each of said texts; a vector calculation step of calculating, for each of the texts, a sparse vector having components each representing the importance of each token in a pre-trained language model based on the vector sequence and the occurrence probability vector; an inverted index generation step of storing the sparse vectors for each text in an inverted index table; Run the following on your computer: The language encoding step and the vector calculation step utilize a pre-trained language model whose parameters have been updated by the learning method according to claim 1. A program characterized by:

Images