Information processing program, information processing method, and information processing device

Abstract

To support optimization of output of a language model.SOLUTION: According to an embodiment, an information processing program causes a computer to execute acquiring processing, inputting processing, calculating processing and outputting processing. The acquiring processing acquires a plurality of word strings related to an object sentence. The inputting processing inputs each of a plurality of combined sentences obtained by combining each of the plurality of acquired word strings to the object sentence and the object sentence to the language model. The calculating processing calculates a certainty factor in output in the case of inputting the object sentence to the language model on the basis of a difference from each distribution in the case of inputting each of the plurality of combination sentences to the language model. The outputting processing outputs an output result in the case of inputting the object sentence to the language model on the basis of the calculated certainty factor.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Conventionally, natural language processing using language models (LMs) generated by machine learning has been progressing. Natural language processing using such language models has demonstrated high performance in various tasks such as summarizing news articles and answering questions in dialogue systems.

[0003] Language models generated by machine learning are not good at dealing with irregular situations such as unlearned cases. For this reason, natural language processing using language models can produce incorrect output, such as outputting information not written in the main text in a news article summary or giving answers that are not based on facts in a dialogue system.

[0004] In natural language processing using such language models, a conventional technique for preventing erroneous output is known that calculates the confidence of the output of the language model and refrains from answering if the confidence is below a threshold. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Selective Question Answering under Domain Shift, Amita Kamath et al., Computer Science Department, Stanford University, 2020 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the above conventional technology, even if the language model outputs an erroneous output, the confidence level may be calculated to be high. Therefore, if a confidence level close to the correct answer is calculated, the erroneous output is not prevented and is output, which is insufficient for optimizing the output.

[0007] According to one aspect, an object of the present invention is to provide an information processing program, an information processing method, and an information processing device that can assist in optimizing the output of a language model. [Means for solving the problem]

[0008] In one proposal, the information processing program causes a computer to execute an acquisition process, an input process, a calculation process, and an output process. The acquisition process acquires a plurality of word strings related to a target sentence. The input process inputs the target sentence and a plurality of combined sentences obtained by combining each of the acquired plurality of word strings with the target sentence into a language model. The calculation process calculates a confidence level in the output when the target sentence is input into the language model based on the difference from each distribution of the output result when each of the plurality of combined sentences is input into the language model. The output process outputs the output result when the target sentence is input into the language model based on the calculated confidence level. Effect of the Invention

[0009] It can help optimize the output of language models. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram for explaining an outline of an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating an example of the operation of the information processing device according to the embodiment. [Figure 4] FIG. 4 is an explanatory diagram for explaining calculation of confidence levels and output of answers according to confidence levels. [Diagram 5] FIG. 5 is an explanatory diagram for explaining specific examples of responses for each case. [Figure 6] FIG. 6 is an explanatory diagram illustrating an example of a computer configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an information processing program, an information processing method, and an information processing device according to the embodiments will be described with reference to the drawings. Configurations having the same functions in the embodiments are given the same reference numerals, and duplicated descriptions will be omitted. Note that the information processing program, the information processing method, and the information processing device described in the following embodiments are merely examples, and do not limit the embodiments. In addition, the following embodiments may be appropriately combined within a range that does not cause contradiction.

[0012] 1 is an explanatory diagram for explaining an overview of an embodiment. As shown in Fig. 1, in an information processing device according to the embodiment, natural language processing is performed on an input sentence x, which is a processing target sentence, using a language model M1 generated by machine learning.

[0013] Natural language processing using the language model M1 may be any of summarizing news articles, answers in a dialogue system, translation in a translation system, and the like. For example, in summarizing news articles, an original sentence is input to the language model M1 as an input sentence x, and information on the summary sentence (probability distribution P(y|x) of a word string) is obtained as an output (y) of the language model M1. In answering in a dialogue system, a question sentence is input to the language model M1 as an input sentence x, and a probability distribution of a word string related to the answer sentence is obtained as an output of the language model M1. In translation in a translation system, an original sentence is input to the language model M1 as an input sentence x, and a probability distribution of a word string related to a translated sentence is obtained as an output of the language model M1. In the embodiment, a case where an answer in a dialogue system is obtained using the language model M1 is illustrated.

[0014] In the information processing device of the embodiment, when an input sentence x is input to a language model M1, the output result (answer sentence based on the probability distribution P(y|x)) is determined as follows, thereby suppressing erroneous output and helping to optimize the output of the language model M1.

[0015] First, the information processing device acquires dummy contexts (c1, c2...) related to the input sentence x using a corpus, which is a database that accumulates various documents, as a string of words related to the input sentence x. Next, the information processing device combines each of the acquired dummy contexts (c1, c2...) with the input sentence x to obtain combined sentences (c1+x, c2+x...). The dummy contexts (c1, c2...c j ) can also be written as (1) below.

[0016]

number

[0017] Next, in the information processing device, each combined sentence is input to the language model M1, and a probability distribution of word strings in each output result is obtained. The probability distribution of word strings obtained by inputting each combined sentence to the language model M1 can also be expressed as follows (2).

[0018]

number

[0019] Next, the information processing device compares the probability distributions of the combined sentences to determine the difference (degree of change). The difference in probability distribution includes the difference in the dummy contexts (c1, c2, ... c j ) context dependency is apparent.

[0020] For example, the greater the difference in probability distribution, the greater the probability of j) is highly context-dependent, and the output result of the language model M1 is influenced by the dummy context. Therefore, the greater the difference in the probability distribution, the lower the confidence in the output result when the input sentence x is input to the language model M1, and the more likely the output result is to be an error.

[0021] In addition, the smaller the difference in the probability distribution, the more likely it is that the dummy contexts (c1, c2…c j ) has low context dependency, and the output result of the language model M1 is not affected by the dummy context. Therefore, the smaller the difference in probability distribution, the higher the confidence in the output result when the input sentence x is input to the language model M1, and the less likely the output result is to be an error.

[0022] In the information processing device, dummy contexts (c1, c2...c j ) is used to calculate the confidence of the output when input sentence x is input to language model M1 based on the difference in the probability distribution of each combined sentence.

[0023] Next, the information processing device outputs the output result (answer sentence based on the probability distribution P(y|x)) when the input sentence x is input to the language model M1 based on the calculated confidence level. For example, if the information processing device determines that the output result (answer sentence) by the language model M1 is unlikely to contain an error and outputs the obtained answer sentence when the confidence level exceeds a preset threshold. Also, if the information processing device determines that the output result (answer sentence) by the language model M1 is likely to contain an error and suppresses the output of the obtained answer sentence when the confidence level does not exceed a preset threshold. In this way, the information processing device can support the optimization of the output of the language model M1.

[0024] 2 is a block diagram showing an example of a functional configuration of an information processing device according to an embodiment. As shown in FIG. 2, the information processing device 1 includes an input / output unit 10, a storage unit 20, and a control unit 30.

[0025] The input / output unit 10 controls an input / output interface such as a GUI (Graphical User Interface) when the control unit 30 inputs and outputs various information. For example, the input / output unit 10 controls an input / output interface with an input device such as a keyboard or a microphone, or a display device such as a liquid crystal display device, connected to the information processing device 1. The input / output unit 10 also controls a communication interface for performing data communication with an external device connected via a communication network such as a LAN (Local Area Network).

[0026] For example, the information processing device 1 accepts input of an input sentence x via the input / output unit 10. In addition, the information processing device 1 outputs a processing result for the input sentence x (for example, a response sentence) via the input / output unit 10.

[0027] The storage unit 20 corresponds to, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk drive (HDD), etc. The storage unit 20 stores a dummy context corpus 21, document retrieval parameters 22, language model parameters 23, confidence calculation parameters 24, document generation model parameters 25, etc.

[0028] The dummy context corpus 21 is a set of dummy contexts (c1, c2...c j ) is a corpus for obtaining the corpus. This corpus does not need to be stored in the information processing device 1, and for example, a corpus stored in an external information processing device via the input / output unit 10 may be used.

[0029] The document search parameters 22 are dummy contexts (c1, c2...c3) related to the input sentence x from the dummy context corpus 21. j For example, the document search parameters 22 include a threshold value for determining whether or not a document is related based on the similarity of the document during document search.

[0030] The language model parameters 23 are parameter information related to the language model M1. For example, the language model parameters 23 are parameters for constructing a machine learning model related to the language model M1, such as a gradient boosting tree or a neural network.

[0031] The confidence factor calculation parameters 24 are parameter information used in a formula for calculating the confidence factor. For example, the confidence factor calculation parameters 24 include coefficient values ​​(weighting values) used in the formula for calculating the confidence factor.

[0032] The document generation model parameters 25 are parameter information related to a machine learning model (document generation model) that generates (outputs) dummy document data related to input document data. For example, the document generation model parameters 25 are parameters for constructing a machine learning model related to a document generation model such as a gradient boosting tree or a neural network.

[0033] The control unit 30 has a dummy context acquisition unit 31, an answer acquisition unit 32, a confidence factor calculation unit 33, and an output unit 34. The control unit 30 can be realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The control unit 30 can also be realized by hardwired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0034] The dummy context acquisition unit 31 is a processing unit that acquires a string of words related to a target sentence (input sentence x), that is, dummy contexts (c1, c2, c3 . . . ).

[0035] Specifically, the dummy context acquisition unit 31 acquires, based on the input sentence x, a plurality of dummy contexts from the dummy context corpus 21 in order of similarity according to the parameters included in the document search parameters 22 as dummy contexts related to the input sentence x. As an example, the dummy context acquisition unit 31 acquires the input sentence x and the context c of the document included in the dummy context corpus 21 in the order of similarity according to the parameters included in the document search parameters 22 as dummy contexts related to the input sentence x. j We prepare two encoders that vectorize each of the k contexts c j is used as a dummy context.

[0036] In addition, the dummy context acquisition unit 31 may acquire multiple dummy contexts based on the output result (probability distribution of word strings) obtained by inputting the input sentence x into a machine learning model (document generation model) constructed based on the document generation model parameters 25.

[0037] The answer acquisition unit 32 is a processing unit that acquires an answer sentence to the input sentence x based on the output result when the input sentence x is input to the language model M1. Specifically, the answer acquisition unit 32 inputs information about the input sentence x to the language model M1 constructed based on the language model parameters 23, and acquires a probability distribution for a word string (arrangement of words) corresponding to the answer sentence from the language model M1. As an example, the answer acquisition unit 32 inputs the input sentence x to the language model M1, and acquires a predicted label (y0) for each word and a probability mass function such as the following formula (3) that indicates the distribution of the label probability. The answer acquisition unit 32 acquires an answer sentence based on the probability distribution (probability mass function) of the predicted label (y0) output from the language model M1 in this way.

[0038]

number

[0039] The confidence calculation unit 33 is a processing unit that calculates the confidence levels described above. Specifically, the confidence calculation unit 33 combines each of the dummy contexts (c1, c2...) acquired by the dummy context acquisition unit 31 with the input sentence x to obtain combined sentences (c1+x, c2+x...). Next, the confidence calculation unit 33 inputs each combined sentence into a language model M1 constructed based on the language model parameters 23, and obtains a probability distribution corresponding to each combined sentence. As an example, the confidence calculation unit 33 inputs the combined sentences exemplified in (1) into the language model M1 to obtain a predicted label (y j ) and obtain the probability mass function (probability distribution) shown in the following equation (4), which shows the distribution of the label probability.

[0040]

number

[0041] Next, the confidence calculation unit 33 calculates the confidence in the output when the input sentence x is input to the language model M1, based on the difference from each probability distribution when each of the multiple combined sentences is input to the language model M1.

[0042] Specifically, the confidence calculation unit 33 calculates k dummy contexts (c j The variance of the probability distribution after the probability distribution C is assigned is calculated using the following formula (5). The confidence calculation unit 33 sets the variance value based on each of the probability distributions calculated in this way as an index value of the confidence C.

[0043]

number

[0044] Furthermore, the confidence calculation unit 33 calculates the average of KL (Kullback-Leibler) divergence as the distance between the probability distributions before and after the addition of the dummy context, as shown in the following formula (6). The confidence calculation unit 33 may use the distance value based on each of the probability distributions thus calculated as an index value of the confidence C.

[0045]

number

[0046] The output unit 34 is a processing unit that outputs the output result (answer sentence based on the predicted label (y0)) when the input sentence x is input to the language model M1 based on the confidence C calculated by the confidence calculation unit 33 to a display or an external device via the input / output unit 10. Specifically, the output unit 34 compares the confidence C calculated by the confidence calculation unit 33 with a preset threshold (β), and refrains from outputting the answer sentence when C<β. Moreover, the output unit 34 outputs the answer sentence when C≧β.

[0047] Fig. 3 is a flowchart showing an example of the operation of the information processing device 1 according to the embodiment. S1 in Fig. 3 is a flowchart for generating a dummy context using the dummy context corpus 21. S2 in Fig. 3 is a flowchart for generating a dummy context using a machine learning model (document generation model) constructed based on the document generation model parameters 25.

[0048] First, a case (S1) of generating dummy contexts using the dummy context corpus 21 will be described. As shown in S1, when the process starts, the dummy context acquisition unit 31 extracts multiple dummy contexts from the dummy context corpus 21 in order of similarity based on the input sentence x. Next, the dummy context acquisition unit 31 selects, for example, three dummy contexts (c1, c2, c3) in order of similarity according to the parameters included in the document retrieval parameters 22 (S11).

[0049] Next, the answer acquisition unit 32 and the confidence calculation unit 33 perform an input process (S12) to input the input sentence x and a combined sentence obtained by combining the input sentence x with a dummy context to the language model M1 constructed based on the language model parameters 23. As a result, the answer acquisition unit 32 obtains a predicted label (y0) when input to the language model M1 and a probability distribution of the label. In addition, the confidence calculation unit 33 performs an output probability calculation of the probability distribution corresponding to each combined sentence (S13).

[0050] Next, the confidence calculation unit 33 calculates the confidence C of the output when the input sentence x is input to the language model M1 based on the difference with each probability distribution obtained by the output probability calculation (S14). Next, the output unit 34 outputs the output result when the input sentence x is input to the language model M1 based on the confidence C calculated by the confidence calculation unit 33 (S15).

[0051] Next, a description will be given of a case (S2) in which a dummy context is generated using a document generation model constructed based on the document generation model parameters 25. As shown in S2, when the process starts, the dummy context acquisition unit 31 constructs a machine learning model (document generation model) based on the document generation model parameters 25.

[0052] Next, the dummy context acquisition unit 31 generates multiple dummy contexts based on the output result (probability distribution of word strings) obtained by inputting the input sentence x to the constructed machine learning model (document generation model) (S11a). For example, the dummy context acquisition unit 31 generates multiple dummy contexts by changing combinations of words whose probability values ​​in the probability distribution are higher than a specific threshold. The processes from S11a onwards are performed in the same manner as S1.

[0053] 4 is an explanatory diagram for explaining calculation of confidence C and output of an answer according to confidence C. As shown in FIG. 4, in the information processing device 1, based on an input sentence x combining a context (p, q), from among contexts (c1, c2, c3, c4, ...) included in the dummy context corpus 21, a context similar to the context (p, q) of the input sentence x is obtained as a dummy context (c1, c2, c3).

[0054] Next, in the information processing device 1, a combined sentence in which each of the dummy contexts (c1, c2, c3) is combined with the input sentence x is input to the language model M1, and predicted labels (y1, y2, y3) and a probability distribution of the labels are obtained.

[0055] Based on the difference with each of these probability distributions, the information processing device 1 calculates the confidence C of the output when the input sentence x is input to the language model M1. Next, based on the confidence C, the information processing device 1 outputs the output result (y) when the input sentence x is input to the language model M1. Specifically, the information processing device 1 compares the confidence C with a preset threshold (β), and refrains from answering y when C<β. Moreover, the information processing device 1 answers y when C≧β.

[0056] FIG. 5 is an explanatory diagram for explaining specific examples of answers for each case. In FIG. 5, case R1 is a case in which the output result (y) when the input sentence x is input to the language model M1 is an incorrect answer. Case R2 is a case in which the output result (y) when the input sentence x is input to the language model M1 is an incorrect answer, and an answer is refrained from based on the confidence C calculated by the information processing device 1 according to the embodiment. Case R2 is a case in which the output result (y) when the input sentence x is input to the language model M1 is a correct answer, and an answer is made based on the confidence C calculated by the information processing device 1 according to the embodiment.

[0057] As shown in case R1, the probability distribution of the output result (y) when the input sentence x is input to the language model M1 may result in a high value of the confidence C (0.9 in the illustrated example). This may result in the wrong answer being output as is.

[0058] In the information processing device 1 according to the embodiment, the probability distributions of combined sentences in which each of the dummy contexts (c1, c2, c3) is combined with the input sentence x are compared, and a confidence level C is obtained based on the difference (degree of change).

[0059] Therefore, in case R2, where the difference in probability distribution is large and the output result when the input sentence x is input to the language model M1 has a high context dependency on the dummy context (c1, c2, c3), the value of the confidence C for an incorrect answer is low (0.3 in the illustrated example). For this reason, in case R2, the answer using the language model M1 is refrained from being given as it is highly likely to be incorrect.

[0060] Furthermore, in case R3, where the difference in probability distribution is small and the context dependency of the dummy context (c1, c2, c3) is low with respect to the output result when the input sentence x is input to the language model M1, the value of the confidence factor C for the correct answer is high (0.9 in the illustrated example). Therefore, in case R3, the answer by the language model M1 is output as being highly likely to be the correct answer. In this way, the information processing device 1 according to the embodiment can support the optimization of the output of the language model M1.

[0061] As described above, the information processing device 1 acquires a plurality of word strings (c1, c2, c3...) related to the target sentence (input sentence x). The information processing device 1 inputs the target sentence and each of a plurality of combined sentences obtained by combining each of the acquired plurality of word strings with the target sentence into the language model M1. The information processing device 1 calculates the confidence C of the output when the target sentence is input into the language model M1 based on the difference from each distribution of the output result when each of the plurality of combined sentences is input into the language model M1. The information processing device 1 outputs the output result when the target sentence is input into the language model M1 based on the calculated confidence C.

[0062] The difference between the distributions of the output results in the multiple combined sentences indicates the context dependency of the output result of the language model M1 for the target sentence. Therefore, the information processing device 1 can obtain a confidence level according to the context dependency of the output result of the language model M1 for the target sentence, and outputs the language model M1 based on this confidence level, thereby supporting optimization of the output of the language model M1.

[0063] Furthermore, the information processing device 1 calculates a variance based on each distribution of the output results when each of the multiple combined sentences is input to the language model M1, and sets the calculated variance as an index value of the confidence C. In this way, the information processing device 1 can obtain the confidence C taking into account the context dependency, using the variance based on each distribution of the output results in the multiple combined sentences as an index value of the confidence C.

[0064] Furthermore, the information processing device 1 calculates a distance based on each distribution of the output results when each of the multiple combined sentences is input to the language model M1, and sets the calculated distance as an index value of the confidence C. In this way, the information processing device 1 can obtain the confidence C taking into account the context dependency by setting the distance based on each distribution of the output results in the multiple combined sentences as an index value of the confidence C.

[0065] Furthermore, the information processing device 1 acquires a plurality of word strings (c1, c2, c3...) related to the target sentence in the dummy context corpus 21 based on the similarity with the target sentence. This allows the information processing device 1 to acquire a plurality of word strings related to the target sentence from the dummy context corpus 21.

[0066] Note that each component of each device shown in the figure does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0067] Furthermore, the various processing functions of the dummy context acquisition unit 31, answer acquisition unit 32, confidence factor calculation unit 33 and output unit 34 performed by the control unit 30 of the information processing device 1 may be executed in whole or in part on a CPU (or a microcomputer such as an MPU or MCU (Micro Controller Unit)). Needless to say, the various processing functions may be executed in whole or in part on a program analyzed and executed by the CPU (or a microcomputer such as an MPU or MCU), or on hardware using wired logic. The various processing functions performed by the information processing device 1 may be executed by multiple computers working together using cloud computing.

[0068] The various processes described in the above embodiment can be realized by executing a program prepared in advance on a computer. Therefore, an example of a computer configuration (hardware) that executes a program having the same functions as those in the above embodiment will be described below. Figure 6 is an explanatory diagram for explaining an example of the computer configuration.

[0069] 6, the computer 200 includes a CPU 201 that executes various arithmetic processes, an input device 202 that accepts data input, a monitor 203, and a speaker 204. The computer 200 also includes a medium reading device 205 that reads programs and the like from a storage medium, an interface device 206 for connecting to various devices, and a communication device 207 for connecting to external devices by wire or wirelessly. The information processing device 1 also includes a RAM 208 that temporarily stores various information, and a hard disk device 209. The various units (201 to 209) in the computer 200 are connected to a bus 210.

[0070] The hard disk drive 209 stores a program 211 for executing various processes in the functional configuration (e.g., the dummy context acquisition unit 31, the answer acquisition unit 32, the confidence factor calculation unit 33, and the output unit 34) described in the above embodiment. The hard disk drive 209 also stores various data 212 referenced by the program 211. The input device 202, for example, accepts input of operation information from an operator. The monitor 203, for example, displays various screens operated by the operator. The interface device 206 is connected to, for example, a printing device. The communication device 207 is connected to a communication network such as a LAN (Local Area Network), and exchanges various information with external devices via the communication network.

[0071] The CPU 201 reads out the program 211 stored in the hard disk drive 209, expands it in the RAM 208, and executes it to perform various processes related to the above-mentioned functional configuration (for example, the dummy context acquisition unit 31, the answer acquisition unit 32, the confidence factor calculation unit 33, and the output unit 34). The program 211 does not have to be stored in the hard disk drive 209. For example, the program 211 stored in a storage medium readable by the computer 200 may be read out and executed. The storage medium readable by the computer 200 corresponds to, for example, a portable storage medium such as a CD-ROM or a DVD disk, a USB (Universal Serial Bus) memory, a semiconductor memory such as a flash memory, a hard disk drive, etc. Also, the program 211 may be stored in a device connected to a public line, the Internet, a LAN, etc., and the computer 200 may read out and execute the program 211 from the device.

[0072] The following supplementary notes are further disclosed regarding the above embodiment.

[0073] (Appendix 1) Obtain multiple word sequences related to the target sentence, inputting a plurality of combined sentences obtained by combining each of the acquired plurality of word strings with the target sentence and the target sentence into a language model; calculating a confidence level of an output when the target sentence is input to the language model based on a difference between the distribution of an output result when each of the plurality of combined sentences is input to the language model; outputting an output result when the target sentence is input to the language model based on the calculated confidence level; An information processing program that causes a computer to execute a process.

[0074] (Additional Note 2) The calculation process calculates a variance based on each of the distributions, and sets the calculated variance as an index value of the confidence level. 2. The information processing program according to claim 1,

[0075] (Additional Note 3) The calculation process calculates a distance based on each of the distributions, and sets the calculated distance as an index value of the confidence level. 2. The information processing program according to claim 1,

[0076] (Appendix 4) The acquiring process acquires a plurality of word strings related to the target sentence in the corpus based on the similarity with the target sentence. 4. The information processing program according to claim 1,

[0077] (Appendix 5) Obtain multiple word sequences related to the target sentence, inputting a plurality of combined sentences obtained by combining each of the acquired plurality of word strings with the target sentence and the target sentence into a language model; calculating a confidence level of an output when the target sentence is input to the language model based on a difference between the distribution of an output result when each of the plurality of combined sentences is input to the language model; outputting an output result when the target sentence is input to the language model based on the calculated confidence level; An information processing method characterized in that the processing is executed by a computer.

[0078] (Additional Note 6) The calculation process calculates a variance based on each of the distributions, and sets the calculated variance as an index value of the confidence level. 6. The information processing method according to claim 5,

[0079] (Appendix 7) The calculation process calculates a distance based on each of the distributions, and sets the calculated distance as an index value of the confidence level. 6. The information processing method according to claim 5,

[0080] (Appendix 8) The acquiring process acquires a plurality of word strings related to the target sentence in the corpus based on the similarity with the target sentence. 8. The information processing method according to claim 5,

[0081] (Appendix 9) Obtain multiple word sequences related to the target sentence, inputting a plurality of combined sentences obtained by combining each of the acquired plurality of word strings with the target sentence and the target sentence into a language model; calculating a confidence level of an output when the target sentence is input to the language model based on a difference between the distribution of an output result when each of the plurality of combined sentences is input to the language model; outputting an output result when the target sentence is input to the language model based on the calculated confidence level; 1. An information processing device comprising: a control unit that executes processing.

[0082] (Additional Note 10) The calculation process calculates a variance based on each of the distributions, and sets the calculated variance as an index value of the confidence level. 10. The information processing device according to claim 9,

[0083] (Additional Note 11) The calculation process calculates a distance based on each of the distributions, and sets the calculated distance as an index value of the confidence level. 10. The information processing device according to claim 9,

[0084] (Appendix 12) The acquiring process acquires a plurality of word strings related to the target sentence in the corpus based on the similarity with the target sentence. 12. The information processing device according to any one of claims 9 to 11. [Explanation of symbols]

[0085] 1...Information processing device 10...Input / output section 20...Storage section 21…Dummy context corpus 22…Document search parameters 23…Language model parameters 24…Confidence calculation parameters 25…Document generation model parameters 30...Control section 31…Dummy context acquisition section 32…Answer acquisition part 33…Confidence calculation part 34...Output section 200…Computer 201…CPU 202...Input device 203…Monitor 204…Speaker 205... Media reader 206...Interface device 207...Communication equipment 208…RAM 209...Hard disk drive 210…Bus 211…Program 212...Various data c...dummy context C…Confidence level M1…Language model R1~R3…Case x...input statement

Claims

1. A predetermined number of word strings related to the target sentence are obtained from a corpus in order of similarity to the target sentence; inputting a plurality of combined sentences obtained by combining each of the acquired plurality of word strings with the target sentence and the target sentence into a language model; calculating a confidence level of an output when the target sentence is input to the language model based on a difference between the distribution of an output result when each of the plurality of combined sentences is input to the language model; outputting an output result when the target sentence is input to the language model based on the calculated confidence level; An information processing program that causes a computer to execute a process.

2. The calculation process includes calculating a variance based on each of the distributions, and setting the calculated variance as an index value of the confidence.

2. The information processing program according to claim 1,

3. the calculating process calculates a distance based on each of the distributions, and sets the calculated distance as an index value of the confidence level.

2. The information processing program according to claim 1,

4. A predetermined number of word strings related to the target sentence are obtained from the corpus in order of similarity to the target sentence; inputting a plurality of combined sentences obtained by combining each of the acquired plurality of word strings with the target sentence and the target sentence into a language model; calculating a confidence level of an output when the target sentence is input to the language model based on a difference between the distribution of an output result when each of the plurality of combined sentences is input to the language model; outputting an output result when the target sentence is input to the language model based on the calculated confidence level; An information processing method characterized in that the processing is executed by a computer.

5. A predetermined number of word strings related to the target sentence are obtained from the corpus in order of similarity to the target sentence; inputting a plurality of combined sentences obtained by combining each of the acquired plurality of word strings with the target sentence and the target sentence into a language model; calculating a confidence level of an output when the target sentence is input to the language model based on a difference between the distribution of an output result when each of the plurality of combined sentences is input to the language model; outputting an output result when the target sentence is input to the language model based on the calculated confidence level; 1. An information processing device comprising: a control unit that executes processing.

Images