Language processing device, image processing method, and program

Abstract

The purpose of the present invention is to more appropriately calculate reliability of prediction.　A language processing device (3) has: a language understanding unit (33) that extracts a feature value from text data; a feature value conversion unit (42) that outputs a reply start point score, a reply end point score, and a reply possibility score using the feature value as an input; an n-best extraction unit (43) that extracts a predetermined n number of reply fidelity scores on the basis of the reply start point score and the reply end point score; and an adjustment unit (44) that determines n adjusted reply fidelity scores from the n reply fidelity scores and determines an adjusted reply possibility score from the reply possibility score, the language understanding unit (33), the feature value conversion unit (42), and the adjustment unit (44) performing a process on the basis of a model parameter of a neural network, and performing learning of the model parameter on the basis of the n adjusted reply fidelity scores and the adjusted reply possibility score, as well as a correct reply section and a correct reply possibility.

Description

[Technical Field]

[0001] The present disclosure relates to a language processing device, an image processing method, and a program. [Background technology]

[0002] The task of answering a user's question by referring to text (character information) written in natural language and extracting a section of the text is called extractive machine comprehension. Extractive machine comprehension is based on techniques such as BERT (Bidirectional Encoder Representations from Transformers). It is common to solve this problem using a discriminative deep learning model (Non-Patent Document 1). A representative dataset for extractive machine reading comprehension is SQuAD 2.0 (Non-Patent Document 2). A typical example of a task in which a discriminative deep learning model is used is a task in which an image with a single digit between 0 and 9 written on it is input and the correct label (in this case, the digit) is output. A discriminative deep learning model can output a probability distribution with the label set as its base as the probability that each label is true. Here, the "base" is the set of values ​​that the random variable can take. The label set consists of 10 digits between 0 and 9. The "probability that each label is true" can be rephrased as the reliability of the prediction. In a discriminative deep learning model for extraction-based machine reading comprehension, the label set is a set of text positions. In other words, where L is the text length, {1, ..., L} is the label set. The interval to be extracted can be determined by selecting the start and end points of the interval from the label set. It is also possible to prepare labels {unanswerable, answerable} to take answerability into consideration. In this way, the preparation of two or three classifiers for start point, end point, and answerability is a characteristic of a discriminative deep learning model for extraction-based machine reading comprehension. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [Non-patent document 2] Know What You Don't Know: Unanswerable Questions for SQuAD Summary of the Invention [Problem to be solved by the invention]

[0004] However, deep learning models are generally known to be overconfident. That is, the probability p(m) that a label m output by a discriminative deep learning model is true tends to be larger than the probability that m is actually true. This overconfidence phenomenon poses a major problem when presenting the reliability of the output to the user.

[0005] The present invention has been made in view of the above points, and has an object to calculate the reliability of prediction more appropriately than conventional methods. [Means for solving the problem]

[0006] In order to solve the above problem, the invention of claim 1 is a language processing device comprising: a language understanding unit that extracts features from text data; a feature conversion unit that takes the features as input and outputs an answer starting point score, an answer ending point score, and an answer possibility score; an n-best extraction unit that extracts a predetermined n number of answer compatibility scores based on the answer starting point score and the answer ending point score; and an adjustment unit that calculates n adjusted answer compatibility scores from the n answer compatibility scores and calculates an adjusted answer possibility score from the answer possibility scores, wherein the language understanding unit, the feature conversion unit, and the adjustment unit perform processing based on model parameters of a neural network, and learn the model parameters based on the n adjusted answer compatibility scores and the adjusted answer possibility scores, as well as the correct answer section and the correct answer possibility. [Effects of the Invention]

[0007] As described above, the present invention has the effect of being able to calculate the reliability of prediction more appropriately than in the past. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram of a communication system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a diagram illustrating the hardware configuration of a language processing device and a communication terminal. [Figure 3] FIG. 2 is a functional configuration diagram of a learning phase for training data of a language processing device. [Figure 4] FIG. 2 is a functional configuration diagram of the language processing device in a learning phase for development data. [Figure 5] FIG. 1 is a functional configuration diagram of an inference phase for test data of a language processing device. [Figure 6] 10 is a flowchart showing processing in a learning phase using training data. [Figure 7] 10 is a flowchart showing processing in a learning phase using development data. [Figure 8] 10 is a flowchart illustrating processing in the inference phase using test data. [Figure 9] FIG. 10 is a diagram showing evaluation results. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] [System configuration of the embodiment] First, an outline of the configuration of a communication system 1 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of a communication system according to an embodiment of the present invention.

[0011] 1, a communication system 1 of this embodiment is constructed by a language processing device 3 and a communication terminal 5. The communication terminal 5 is managed and used by a user.

[0012] Furthermore, the language processing device 3 and the communication terminal 5 can communicate with each other via a communication network 100 such as the Internet. The communication network 100 may be connected wirelessly or via a wired connection.

[0013] The language processing device 3 is configured by one or more computers. When the language processing device 3 is configured by multiple computers, it may be referred to as a "language processing device" or a "language processing-like system."

[0014] The language processing device 3 is a computer that more appropriately calculates the reliability of predictions (inferences) when a deep learning model is used. The language processing device 3 outputs result data, which is the result of the prediction. Examples of output methods include transmitting the result data to the communication terminal 5, thereby displaying or printing a graph or the like related to the result data on the communication terminal 5 side, displaying the graph or the like on a display connected to the language processing device 3, or printing the graph or the like on a printer or the like connected to the language processing device 3.

[0015] The communication terminal 5 is a computer, and in Fig. 1, a notebook computer is shown as an example, but it is not limited to a notebook computer and may be a desktop computer. The communication terminal may also be a smartphone or a tablet terminal. In Fig. 1, a user operates the communication terminal 5.

[0016] [Hardware configuration of language processing device and communication terminal] Next, the hardware configuration of the language processing device 3 and the communication terminal 5 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the hardware configuration of the language processing device and the communication terminal.

[0017] 2, the language processing device 3 includes a processor 301, a memory 302, an auxiliary storage device 303, a connection device 304, a communication device 305, and a drive device 306. The hardware components constituting the language processing device 3 are connected to each other via a bus 307.

[0018] The processor 301 serves as a control unit that controls the entire language processing device 3, and includes various arithmetic devices such as a CPU (Central Processing Unit). The seed program is read and executed on the memory 302. The processor 301 may include a general-purpose computing on graphics processing unit (GPGPU).

[0019] The memory 302 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 301 and the memory 302 form a so-called computer, and the processor 301 executes various programs read onto the memory 302, thereby enabling the computer to realize various functions.

[0020] The auxiliary storage device 303 stores various programs and various information used when the processor 301 executes the various programs.

[0021] The connection device 304 is a connection device that connects the language processing device 3 to an external device (for example, a display device 310, an operation device 311).

[0022] The communication device 305 is a communication device for transmitting and receiving various types of information to and from other devices.

[0023] The drive device 306 is a device for loading a recording medium 330. The recording medium 330 here includes a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, and the like. The recording medium 330 includes media that record information optically, electrically, or magnetically, such as a disk, a magneto-optical disk, etc. The recording medium 330 may also include semiconductor memory that records information electrically, such as a ROM (Read Only Memory), a flash memory, etc.

[0024] The various programs to be installed in the auxiliary storage device 303 are installed, for example, by setting the distributed recording medium 330 in the drive device 306 and reading the various programs recorded on the recording medium 330 by the drive device 306. Alternatively, the various programs to be installed in the auxiliary storage device 303 may be installed by being downloaded from a network via the communication device 305.

[0025] FIG. 2 also shows the hardware configuration of the communication terminal 5, but the only difference is that the reference numerals have changed from the 300s to the 500s, and the configurations are the same, so a description of these will be omitted.

[0026] [Functional configuration of language processing device] Next, the functional configuration of the language processing device will be described with reference to FIG.

[0027] <Functional configuration of the learning phase for training data> Fig. 3 is a functional configuration diagram of the learning phase of the language processing device for training data. In Fig. 3, the language processing device 3 has a reception unit 31, a selection unit 32, a language understanding unit (BERT) 33, a loss calculation unit 35, a parameter update unit 36, and a linear transformation layer 42. Each of these units is a function realized by an instruction from the processor 301 in Fig. 2 based on a program.

[0028] Furthermore, the memory 302 or the auxiliary storage device 303 in FIG. 2 stores the corpus c, the parameters 33p of the language understanding unit, and the parameters 42p of the linear transformation layer 42.

[0029] The receiving unit 31 receives a plurality of training data (a set of inputs X and answers Y) from the outside, and inputs them as a corpus c.

[0030] The selection unit 32 selects one piece of data (input X and answer Y) to be processed from multiple pieces of training data in the corpus c. Note that answer Y is composed of three labels: the start point of the answer, the end point of the answer, and the possibility of the answer. The first two are collectively referred to as the correct answer interval, and the remaining one as the possibility of the correct answer. The language understanding unit 33 vectorizes the text patterns of the text data (input X) and The language understanding unit 33 extracts features from the text data. For example, BERT (Bidirectional Encoder Representations from Transformers) is used as a neural network model. It is used.

[0031] Specifically, the language understanding unit 33 divides the text data into predetermined words, expresses them as word vectors, and inputs the input X to the neural network. Based on the model parameters, the input X is converted into H, which is a feature quantity shown below.

[0032]

number

[0033] The loss calculation unit 35 calculates a loss function value based on the output data of the feature conversion unit 42.

[0034] The parameter update unit 36 ​​updates the parameters 33p and 42p based on the output data from the loss calculation unit 35.

[0035] The feature transformer 42 uses the linear transformation layer parameters 42p to linearly transform the feature acquired from the language understanding unit 33. The feature transformer 42 will be described in detail later.

[0036] In an extraction-type machine reading comprehension task, the feature transform unit 42 is provided with one or two transformation layers (linear transformation layers 42a and 42b). One linear transformation layer 42a

[0037]

number

[0038]

number

[0039]

number

[0040] On the other hand, another linear transformation layer 42b

[0041]

number

[0042] In the training data, S, E, and NA are learned using the CrossEntropy loss. The parameters 33p and 42p learned in this training phase are all fixed.

[0043] <Function configuration for the learning phase of development data> Fig. 4 is a functional configuration diagram of the learning phase of development data for a language processing device. In Fig. 4, the language processing device 3 has a selection unit 32, a language understanding unit (BERT) 33, a loss calculation unit 35, a parameter update unit 36, a feature conversion unit 42, an n-best extraction unit 43, and an adjustment unit 44. Each of these units is a function realized by an instruction from the processor 301 in Fig. 2 based on a program.

[0044] 2 stores a corpus c, parameters 33p of the language understanding unit, parameters 42p of the feature conversion unit 42, and parameters 44p of the adjustment unit 44. Note that functional components similar to those in the learning phase of the training data are denoted by the same reference numerals, and their description will be omitted.

[0045] In the learning phase for the development data, the linear transformation layer 42a Ans ) output is subjected to Temperature Scaling, and the linear transformation layer 42b (Linear NoAns ) output, Biased Temperature Scaling is used. This is because the number of unanswerable data and answerable data is imbalanced. Here, we will explain using Temperature Scaling and Biased Temperature Scaling. However, any calibration method such as that described in <Reference 1> can also be used. .

[0046] <Reference 1> On Calibration of Modern Neural Networks The n-best extraction unit 43 extracts a predetermined n answer intervals from the features of each text data based on the answer start point score and answer end point score output from the feature conversion unit 42, thereby extracting the start point feature and end point feature as feature of the answer candidate. The answer interval can be determined by using Equation 3, Equation 4, and the answer compatibility score, and determining the start point and end point so that the start point is before the end point. The answer compatibility score is a value based on the start point score and end point score. For example, it is represented by the sum or product of the start point and end point scores, and in this embodiment, the sum is used in Equation 13. An example of extracting n features is to extract n features in descending order of the answer compatibility score. Any other extraction method can be used. For example, an extraction method in which an extracted answer candidate shares words with a higher-ranked answer candidate may be adopted, ignoring that answer candidate. A section corresponding to a named entity extracted by an external named entity extraction tool or the like may be used as a negative example. During training, a method is performed to ensure that the true answer (answer Y) is always included in the n extractions. Do it. The temperature scaling performed by the adjustment unit 44 is performed by adjusting the temperature parameter

[0047]

number

[0048]

number

[0049] The biased temperature scaling performed by the adjustment unit 44 is performed by adjusting the temperature parameter

[0050]

number

[0051]

number

[0052]

number

[0053]

number

[0054] Here, k is the number of labels. NoAns ) where k=2. Bias The existence of parameter B makes it possible to treat labels equally even in imbalanced data. .

[0055] In this embodiment, first, the linear transformation layer 42b

[0056]

number

[0057] The true answer relevance score is

[0058]

number

[0059]

number

[0060]

number

[0061] Negative examples are answer candidates other than the true answer selected by the n-best extraction unit 43.

[0062] Next, the adjustment unit 44 performs temperature scaling on the vector A of the obtained answer fitness scores. A Prepare and A / T A The CrossEntropy loss is calculated using the score as the temperature parameter T A Only. T A Since Temperature Scaling scales the scores for all labels equally, the label that achieves the maximum value does not change before and after training, and the output answer does not change either. The adjusted answer relevance score is a probability distribution

[0063]

number

[0064] In addition, the answer possibility score, which is the output of the linear transformation layer 42b,

[0065]

number

[0066]

number

[0067] In addition, as a secondary effect of the bias parameter, the label output during inference can be the label with the highest score. Without the bias parameter, it would be necessary to manually set a threshold, for example, to output "answerable" only when the probability of answering is 0.7 or higher. Jiru.

[0068] <Functional configuration of the inference phase for test data> Fig. 5 is a functional configuration diagram of the language processing device in the inference phase for test data. In Fig. 5, the language processing device 3 has a language understanding unit (BERT) 33, a feature conversion unit 42, an n-best extraction unit 43, an adjustment unit 44, and an output unit 39. Each of these units is a function realized by an instruction from the processor 301 in Fig. 2 based on a program.

[0069] 2 stores parameters 33p of the language understanding unit, parameters 42p of the feature conversion unit 42, and parameters 44p of the adjustment unit 44. Note that the same reference numerals are used to designate functional configurations that are the same as those in the learning phase for the training data and the learning phase for the development data, and descriptions thereof will be omitted.

[0070] In the inference phase, the output unit 39 calculates a predicted answer and its predicted answer reliability using the adjusted answer relevance score and adjusted answer possibility score output by the adjustment unit 44. Specifically, the predicted answer and its predicted answer reliability are calculated as follows:

[0071]

number

[0072] In addition, the predicted answerability (output of whether answerable or unanswerable) and the predicted answerability confidence are

[0073]

number

[0074] Finally, the output unit 39 outputs the predicted answer, the predicted answer reliability, the predicted answer possibility, and the predicted answer possibility reliability as result data.

[0075] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described in detail with reference to Figures 6 to 9. The description will be divided into a learning phase using training data, a learning phase using development data, and an inference phase using test data.

[0076] <Outline of processing or operation> As mentioned above, deep learning models are generally known to be overconfident. In other words, the probability p(m) that a label m output by a discriminative deep learning model is true tends to be larger than the probability that m is actually true. This overconfidence phenomenon poses a major problem when presenting the reliability of the output to the user.

[0077] So, here are some common ways to deal with the overconfidence phenomenon:

[0078] First, the model is trained with training data and the parameters are fixed. Next, a small number of parameters are added to the model, and the added parameters are trained with development data. The goal of this training is to make the probability distribution output by the model match the probability that the model's output is actually true.

[0079] There are various methods for configuring the parameters to be added, but one method is called Temperature Scaling. This method is known to be simple and highly efficient (Reference 1).

[0080] In addition, (Biased) Temperature Scaling is a method for cases where the number of data points for each label is imbalanced (Reference 2). <Reference 2>Calibration with Bias-Corrected Temperature Scaling Improves Domain Adaptation Under Label Shift in Modern Neural Networks Furthermore, to deal with the large number of answer candidates in extraction-based machine reading comprehension, we narrow down the candidates to the n-best. In other words, instead of the probability distribution on the label set {1, ..., L} × {1, ..., L}, we use {true answer, negative answer}. Perform learning and inference on {example 1,..., negative example n-1}.

[0081] <Learning phase using training data> Next, the learning phase using training data will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the processing of the learning phase using training data.

[0082] First, the receiving unit 31 receives training data (a set of inputs X and answers Y) from the outside (S11).

[0083] The selection unit 32 selects one piece of data (input X and answer Y) from the teacher data as a processing target (S12).

[0084] The input X data is then input to the language understanding unit 33 and the feature conversion unit 42 in sequence. The above-mentioned processing is carried out in each section (S13).

[0085] The loss calculation unit 35 calculates the loss from the output of the feature conversion unit 42 and the correct answer data Y. The parameter update unit 36 ​​calculates the gradient of the loss and updates the parameters 33p of the language understanding unit 33 and the parameters 42p of the linear transformation layer 42 (S14).

[0086] Next, the selection unit 32 determines whether the processing of steps S13 and S14 has been completed for all data through the operations up to this point (S15). If the processing of all data has not been completed (S15; NO), the process returns to step S13. On the other hand, if the processing of all data has been completed, the selection unit 32 determines whether the repetitive operations of steps S12 to S15 have been completed a specified number of times (S16). If the repetition has not been completed the specified number of times (S16; NO), all data is considered to be unprocessed and the process returns to step S12. On the other hand, if the repetition has been completed the specified number of times (S16; YES), all processing of the learning phase using the training data is completed.

[0087] <Learning phase using development data> Next, the learning phase using development data will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the processing of the learning phase using development data.

[0088] First, the receiving unit 31 receives training data (a set of inputs X and answers Y) from outside (S21).

[0089] The selection unit 32 selects one piece of data (input X and answer Y) from the teacher data as a processing target (S22).

[0090] The selection unit 32 then selects the input X data from the language understanding unit 33, the feature conversion unit 42, and the n-best The data is input to the extraction unit 43 and the adjustment unit 44 in sequence, and the above-mentioned processing is carried out in each unit (S23).

[0091] The loss calculation unit 35 calculates the loss from the output of the adjustment unit 44 and the correct answer data Y, The parameter update unit 36 ​​calculates the gradient of the loss and updates the parameter 44p of the adjustment unit (S24).

[0092] Next, the selection unit 33 determines whether the processing of steps S23 and S24 has been completed for all data through the operations up to this point (S25). If the processing of all data has not been completed (S25; NO), the process returns to step S23. On the other hand, if the processing of all data has been completed, the selection unit 32 determines whether the iterative operations of steps S22 to S25 have been completed a specified number of times (S26). If the iteration has not been completed the specified number of times (S26; NO), all data is considered to be unprocessed and the process returns to step S22. On the other hand, if the iteration has been completed the specified number of times (S26; YES), all processing of the learning phase using the development data is completed.

[0093] <Inference phase using test data> Finally, the inference phase using test data will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the processing of the inference phase using test data.

[0094] First, the input unit 30 receives input of evaluation data (input X) as an evaluation sample sm2. (S31).

[0095] Next, the input X data is passed through the language understanding unit 33, the feature conversion unit 42, and the n-best extraction unit 43. , are sequentially input to the adjustment unit 44, and the above-mentioned processing is carried out in each unit (S32).

[0096] Next, the output unit 39 calculates the answer and the reliability from the output of the adjustment unit 44, and outputs the result data (S33).

[0097] [Evaluation example] Next, an example of evaluation using the language processing device of this embodiment will be described. Fig. 9 is a diagram showing the evaluation results.

[0098] In this evaluation, the SQuAD2.0 dataset was evaluated using a random 90% of the official training data as training data, the remaining 10% as development data, and the official development data as test data.

[0099] In this evaluation example, the method for extracting negative examples during training and the method for extracting answer candidates during inference were those with the largest answer relevance score a. n was set to 3. Also, the temperature parameter T was set to T=exp(T'), where T' is the parameter of the model in the implementation. This is because the temperature parameter needs to take a positive value. This is because...

[0100] The evaluation scale for the answer itself is the exact match rate of the labels "unanswerable" and "answerable," and the exact match rate of the answer interval when the answer is possible. Additionally, the reliability is evaluated using the Expected Calibration Error (ECE). Specifically, it is calculated as follows:

[0101] Let p(x)∈[0,1] be the probability that the model outputs for sample x. Next, divide [0,1] into 10 parts. For example, for the division [0,0.1], calculate the average of p(x) for the set of all samples where p(x)∈[0,0.1]. This is the average confidence in the division [0,0.1]. Also, calculate the perfect match rate of the model output for that set of samples. This is the actual accuracy rate in the division [0,0.1]. The ECE value is calculated by taking the absolute value of the difference between the average confidence and the actual accuracy rate for each division and then taking the micro-average for the division. The ECE value corresponds to the expected value of the difference between the confidence output by the model and the actual accuracy rate, so the smaller the value, the better.

[0102] The comparison methods are a method in which the language processing device 3 of Fig. 3 performs learning on training data and outputs confidence levels in a normal manner, a method in which the language processing device 3 of Fig. 3 also performs learning on training data and narrows down to the n-best only in the inference phase and outputs confidence levels, and a series of processes according to this embodiment. Note that in the series of processes according to this embodiment, narrowing down to the n-best is performed in both the learning phase and the inference phase, and biased Temperature Scaling and Temperature Scaling are performed. The threshold for determining the answerability was set to 0.5. In other words, the label with the larger score was output.

[0103] The results are shown in a table in Figure 9. All units are in %. This embodiment improves ECE without sacrificing the exact agreement rate for the answer interval. The n-best confidence is simply applied to the model. In this case, the ECE worsens, but by using the bias parameter during learning in this embodiment, it contributes to improving the ECE. Regarding answerability, the presence of the bias parameter slightly improves perfect agreement. It can be seen that the ECE also improves.

[0104] [Major Effects of the Embodiments] As described above, according to this embodiment, the language processing device 3 performs temperature scaling (biased temperature scaling) on ​​the discriminative deep learning model in extraction-type machine reading comprehension. By narrowing down the candidates to the n-best and then learning, the probability distribution output by the model can be made closer to the probability that the label is actually true. As a result, this embodiment has the advantage of being able to calculate the reliability of prediction more appropriately than conventional methods.

[0105] 〔supplement〕 The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

[0106] The language processing device 3 can be realized by a computer and a program, and this program can be recorded on a (non-transitory) recording medium or provided via the communication network 100.

[0107] [Additional notes] The above-described embodiment can also be expressed as the following invention.

[0108] [Additional Note 1] A language processing device having a processor, The processor: Extract features from text data based on neural network model parameters, Using the feature amount as an input based on the model parameters of the neural network, outputting an answer start point score, an answer end point score, and an answer possibility score; extracting a predetermined number n of answer compatibility scores based on the answer start score and the answer end score; A language processing device that calculates n adjusted answer compatibility scores from the n answer compatibility scores based on model parameters of the neural network, calculates an adjusted answer possibility score from the answer possibility scores based on model parameters of the neural network, and learns the model parameters based on the n adjusted answer compatibility scores and the adjusted answer possibility score, as well as the correct answer interval and the correct answer possibility.

[0109] [Additional note 2] 2. The language processing device according to claim 1, wherein the n-best extraction step includes a process of extracting the n answer relevance scores in descending order of the highest score.

[0110] [Additional note 3] The n adjusted answer relevance scores and the adjusted answerability scores are calculated based on biased temperature scaling and temperature scaling by the neural network. 3. The language processing device according to claim 1 or 2, wherein the language processing device is obtained by: [Additional Note 4] A language processing device having a processor, The processor: Extract features from text data based on pre-trained model parameters of the neural network, Using the feature amount as an input based on pre-trained model parameters of the neural network, outputting an answer start point score, an answer end point score, and an answer possibility score; extracting a predetermined number n of answer compatibility scores based on the answer start score and the answer end score; calculating n adjusted answer relevance scores from the n answer relevance scores based on pre-trained model parameters of the neural network, and calculating an adjusted answerability score from the answerability scores based on model parameters of the neural network; Calculating a predicted answer, a predicted answer confidence, a predicted answer possibility, and a predicted answer possibility confidence from the n adjusted answer relevance scores and the adjusted answer possibility scores; Language processor.

[0111] [Additional Note 5] The language processor is a language understanding step of extracting features from text data; a feature conversion step of inputting the feature and outputting an answer start score, an answer end score, and an answer possibility score; an n-best extraction step of extracting a predetermined n number of answer fitness scores based on the answer start score and the answer end score; an adjusting step of calculating n adjusted answer relevance scores from the n answer relevance scores and calculating adjusted answerability scores from the answerability scores; Run the language understanding step, the feature conversion step, and the adjustment step include processing based on model parameters of a neural network; A language processing method that learns the model parameters based on the n adjusted answer relevance scores and adjusted answer possibility scores, as well as correct answer intervals and correct answer possibilities.

[0112] [Additional Note 6] A non-transitory recording medium on which a program for causing a computer to execute the method described in appended paragraph 5 is recorded. [Explanation of symbols]

[0113] 1. Communication Systems 3 Language Processing Unit 5. Communication terminals 30 Input section 31 Reception 32 Selection section 33 Language Understanding Department 35 Loss calculation section 36 Parameter update section 39 Output section 42 Feature conversion unit 43 n-best extraction part 44 Adjustment section

Claims

1. a language understanding unit that extracts features from text data; a feature conversion unit that receives the feature as an input and outputs an answer start score, an answer end score, and an answer possibility score; an n-best extraction unit that extracts a predetermined number n of answer compatibility scores based on the answer start point score and the answer end point score; an adjustment unit that calculates n adjusted answer relevance scores from the n answer relevance scores and calculates adjusted answerability scores from the answerability scores; and the language understanding unit, the feature conversion unit, and the adjustment unit perform processing based on model parameters of a neural network; A language processing device that learns the model parameters based on the n adjusted answer relevance scores and the adjusted answer possibility scores, as well as the correct answer interval and the correct answer possibility.

2. The language processing device according to claim 1 , wherein the n-best extraction unit extracts the n answer relevance scores in descending order of the highest score.

3. The language processing device according to claim 1 , wherein the adjustment unit performs processing based on biased temperature scaling and temperature scaling.

4. a language understanding unit that extracts features from text data; a feature conversion unit that receives the feature as an input and outputs an answer start score, an answer end score, and an answer possibility score; an n-best extraction unit that extracts a predetermined number n of answer compatibility scores based on the answer start point score and the answer end point score; an adjustment unit that calculates n adjusted answer relevance scores from the n answer relevance scores and calculates adjusted answerability scores from the answerability scores; an output unit that calculates a predicted answer, a predicted answer reliability, a predicted answer possibility, and a predicted answer possibility reliability from the n adjusted answer relevance scores and the adjusted answer possibility scores; and The language understanding unit and the feature conversion unit perform processing based on the model parameters of a pre-trained neural network, based on the n answer compatibility scores and the answer possibility scores, as well as the correct answer interval and the correct answer possibility.

5. The language processing device according to any one of claims 1 to 3; a communication terminal that receives result data from the language processing device via a communication network; A communication system having:

6. The language processor is a language understanding step of extracting features from text data; a feature conversion step of inputting the feature and outputting an answer start score, an answer end score, and an answer possibility score; an n-best extraction step of extracting a predetermined n number of answer fitness scores based on the answer start score and the answer end score; an adjusting step of calculating n adjusted answer relevance scores from the n answer relevance scores and calculating adjusted answerability scores from the answerability scores; Run the language understanding step, the feature conversion step, and the adjustment step include processing based on model parameters of a neural network; A language processing method, wherein the model parameters are learned based on the n adjusted answer relevance scores and the adjusted answer possibility scores, as well as the correct answer interval and the correct answer possibility.

7. A program causing a computer to execute the method according to claim 6.

Images