A program, estimation apparatus, and method for estimating class-based responses using multiple large-scale language models.

Abstract

This invention provides a program that can improve uncertainty when estimating class-based responses using multiple large-scale language models. [Solution] During the training phase, a response collection means sends the question text of the training data to each large-scale language model and obtains the likelihood of each class as an answer, using training data that associates the question text related to class classification with the correct answer class. A confidence weighting means outputs a weighted likelihood for each large-scale language model by multiplying the likelihood of each class in the answer by the confidence weight of the large-scale language model. The weighted likelihoods of each class in all large-scale language models are input in parallel, and an integrated likelihood of each class is output. The system also functions as a metamodel that backpropagates errors to the confidence weighting means so that the integrated likelihood of each class matches the likelihood of each class in the correct answer.

Description

[Technical Field]

[0001] This invention relates to a technology that uses multiple large language models (LLMs). <000000> [Background technology]

[0002] LLM can estimate answers to given questions. To enable LLM to answer questions even about unlearned knowledge, it can collect useful training data and perform fine-tuning. In contrast, there are conventional data harvesting techniques that extract only useful training data (see, for example, Non-Patent Document 1). This technique introduces a model that estimates the value that each data point has an impact on performance improvement. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Xinyu Lin et al., “Data-efficient Fine-tuning for LLM-based Recommendation,” In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024., [online], [Retrieved February 8, 2025], Internet<URL: https: / / dl.acm.org / doi / 10.1145 / 3626772.<3657807> [Overview of the project] [Problems that the invention aims to solve]

[0004] Even when the technology described in Non-Patent Document 1 is applied to LLM, the predictive performance in classification improves, but the trend of "uncertainty" in the responses remains unchanged. Here, "high uncertainty" means that the LLM is unsure which class to classify it into. The confidence level for each classified class is calculated as [=1-uncertainty]. LLMs generally train using question and ground answer training data, based on the loss between the estimated answer and the ground answer. However, this does not improve uncertainty. Currently, while there are techniques aimed at improving predictive performance, no techniques have been proposed specifically to improve uncertainty.

[0005] Therefore, the present invention aims to provide a program, estimation device, and method that can improve uncertainty when estimating class classification-based responses using multiple large-scale language models. [Means for solving the problem]

[0006] According to the present invention, in a program that operates a computer mounted on a device that estimates answers to question sentences using multiple large-scale language models, During the training phase, training data was used that associated questions about class classification with the correct class answers. For each large-scale language model, a response collection method is provided that sends the question text of the training data and obtains the likelihood of each class as the response. A confidence weighting means that outputs a weighted likelihood for each large-scale language model by multiplying the likelihood of each class of response by the confidence weight of the large-scale language model, A metamodel that takes the weighted likelihoods of each class in all large-scale language models as parallel inputs, outputs the combined likelihood of each class, and backpropagates the error to the confidence weighting means so that the combined likelihood of each class matches the likelihood of each class in the correct answer. It is characterized by enabling the computer to function in this way.

[0007] According to other embodiments of the program of the present invention, The metamodel is An input layer that takes the weighted likelihoods of each class in all large-scale language models as parallel inputs and backpropagates the errors to the confidence weighting means, A hidden layer connected to the input layer, An output layer connected to a hidden layer outputs the combined likelihood of each integrated class, and backpropagates the error so that the combined likelihood of each class matches the likelihood of each correct class. It is also preferable to make the computer function in such a way as to have this feature.

[0008] According to other embodiments of the program of the present invention, The response collection means sends the training data question sentences to each large-scale language model, obtains the classification probability for each class as a response, and outputs the likelihood, which is the classification probability normalized by the softmax function, to the confidence weighting means. It is also preferable to make the computer function in this way.

[0009] According to other embodiments of the program of the present invention, An embedded expression conversion means that converts the class classification question text into an embedded question text expression, and inputs the embedded question text expression into a metamodel. and make it work The metamodel takes further parallel inputs of embedded question representations and outputs the combined likelihood of each integrated class. It is also preferable to make the computer function in this way.

[0010] According to other embodiments of the program of the present invention, Regarding the estimation stage, Using only the large-scale language models with the highest confidence weights in the confidence weighting method, The response collection method involves sending the target question text regarding class classification to a large-scale language model and obtaining the likelihood of each class as a response. The confidence weighting means outputs a weighted likelihood obtained by multiplying the likelihood of each class of response by the confidence weight of the large-scale language model. The metamodel takes the weighted likelihoods of each class in the large-scale language model as input in parallel and outputs the combined likelihood of each class. uncertainty calculation means that receives the integrated likelihood of each class of the estimated answer as input, calculates the Uncertainty of the large language model, and outputs an answer whose uncertainty is equal to or greater than a predetermined threshold to an application as an alarm, and it is also preferable to further cause the computer to function as

[0011] According to another embodiment of the program of the present invention, the uncertainty calculation means sets a higher uncertainty as the highest likelihood among the likelihoods of all classes is lower, sets a higher uncertainty as the difference between the highest likelihood and the second highest likelihood among the likelihoods of all classes is smaller, or sets a higher uncertainty as the entropy of the likelihoods of all classes is higher it is also preferable to cause the computer to function as described above.

[0012] According to another embodiment of the program of the present invention, in a training phase, embedded expression conversion means that converts a question sentence for classification into a question sentence embedded expression and inputs the question sentence embedded expression into a meta-model, and causes the computer to function as, wherein the meta-model further receives the question sentence embedded expressions in parallel and outputs the integrated likelihood of each integrated class, in an estimation phase, the embedded expression conversion means converts a target question sentence related to classification into a question sentence embedded expression, and inputs the question sentence embedded expression into the meta-model it is also preferable to cause the computer to function as described above.

[0013] According to the present invention, in an estimation device that estimates an answer to a question sentence using a plurality of large language models, in a training phase, using teacher data in which question sentences related to classification are associated with correct classes, answer collection means that transmits the question sentence of the teacher data to each large language model and acquires the likelihood of each class as an answer, and A confidence weighting means that outputs a weighted likelihood for each large-scale language model by multiplying the likelihood of each class of response by the confidence weight of the large-scale language model, A metamodel that takes the weighted likelihoods of each class in all large-scale language models as parallel inputs, outputs the combined likelihood of each class, and backpropagates the error to the confidence weighting means so that the combined likelihood of each class matches the likelihood of each class in the correct answer. It is characterized by having the following features.

[0014] According to the present invention, in an estimation method for a device that estimates answers to question sentences using multiple large-scale language models, The device is During the training phase, training data was used that associated questions about class classification with the correct class answers. For each large-scale language model, the training data question sentences are sent, and the likelihood of each class is obtained as the answer. The first step involves using a learning model to output a weighted likelihood for each large-scale language model, obtained by multiplying the likelihood of each class of responses by the confidence weight of that large-scale language model. The second step involves using a metamodel to input the weighted likelihoods of each class in all large-scale language models in parallel, outputting the combined likelihood of each class, and backpropagating the error to the confidence weighting means so that the combined likelihood of each class matches the likelihood of each class in the correct answer. It is characterized by performing the following: [Effects of the Invention]

[0015] According to the program, estimation device, and method of the present invention, uncertainty can be improved when estimating responses based on class classification using multiple large-scale language models. [Brief explanation of the drawing]

[0016] [Figure 1] This is a sequence diagram of an estimation device that receives responses from multiple large-scale language models. [Figure 2]This graph shows the accuracy rate relative to the level of confidence, according to the characteristics of LLM. [Figure 3] This is a functional configuration diagram of the estimation device used in the training phase of the present invention. [Figure 4] This graph shows the characteristics of accuracy rate against confidence level for each LLM (Low-Level Memory). [Figure 5] This is an explanatory diagram of the response collection section. [Figure 6] This is an explanatory diagram of the confidence weighting unit and metamodel in the present invention. [Figure 7] This is an explanatory diagram of the metamodel for further inputting embedded representations of the question text. [Figure 8] This graph shows the accuracy rate against confidence level for LLM in this invention. [Figure 9] This is a functional configuration diagram of the estimation device in the estimation stage of the present invention. [Modes for carrying out the invention]

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0018] Figure 1 is a sequence diagram of an estimation device that receives responses from multiple large-scale language models.

[0019] As shown in Figure 1, estimation device 1 sends prompts, such as the following question, to multiple LLM-X, Y, and Z. The question prompts the LLM to perform a classification. For example, let's assume it is a multiple-choice question with four options. -------------------------------------------------------------------------- (Question) This is a multiple-choice question about answering questions. Please answer based on your knowledge and problem-solving abilities. Please output only the options. Question: Where is the Louvre Museum located? Options: A: Paris, B: Lyon, C: Geneva, D: Vichy -------------------------------------------------------------------------- In this case, it is preferable to instruct the prompting question to either output only the answer choice (class ID) or to output the answer choice first when answering.

[0020] LLM returns the "classification probability" (probability distribution) for each class to Estimation Device 1 as its response. Class: ABCD LLM-X:[···,···,···,···] -> Answer A LLM-Y:[···,···,···,···] -> Answer A LLM-Z:[···,···,···,···] -> Answer B

[0021] Here, the estimation device 1 normalizes the LLMs so that the sum of the classification probabilities for each class is 1.0, and calculates the "likelihood" for each class. For example, suppose the following likelihoods for each class are calculated using the softmax function. Class: ABCD LLM-X:[0.45, 0.33, 0.01, 0.21] -> Answer A LLM-Y:[0.99, 0.01, 0.00, 0.00] -> Answer A LLM-Z:[0.30, 0.35, 0.15, 0.20] -> Answer B

[0022] <Regarding uncertainty> The likelihood for each class allows us to calculate the "uncertainty" of the predicted response. The "confidence" is calculated as [=1 - uncertainty].

[0023] Here are three examples of quantitative methods for calculating uncertainty: (Calculation Example 1) The highest likelihood is taken as the confidence level, and [=1-confidence level] is taken as the uncertainty level. That is, the lower the highest likelihood level among all classes, the higher the uncertainty level. Class: ABCD LLM1:[0.45, 0.33, 0.01, 0.21] -> Answer A, confidence 0.45 Uncertainty level 0.55 = 1 - 0.45 LLM2:[0.99, 0.01, 0.00, 0.00] -> Answer A, confidence 0.99 Uncertainty level 0.01 = 1 - 0.99 LLM3:[0.30, 0.35, 0.15, 0.20] -> Answer B, confidence 0.35 Uncertainty level 0.65 = 1 - 0.35

[0024] (Calculation Example 2) The difference between the highest likelihood and the second highest likelihood is taken as the confidence level, and [=1-confidence level] is taken as the uncertainty level. That is, among all the likelihood levels in all classes, the smaller the difference between the highest likelihood and the second highest likelihood, the higher the uncertainty level. Class: ABCD LLM1:[0.45, 0.33, 0.01, 0.21] -> Answer A, confidence 0.12=0.45-0.33 Uncertainty level 0.88 = 1 - 0.12 LLM2:[0.99, 0.01, 0.00, 0.00] -> Answer A, confidence 0.98=0.99-0.01 Uncertainty level 0.02 = 1 - 0.98 LLM3:[0.30, 0.35, 0.15, 0.20] -> Answer B, confidence 0.05=0.35-0.30 Uncertainty level 0.95 = 1 - 0.05

[0025] (Calculation Example 3) The entropy in the likelihood of all classes is defined as the uncertainty. That is, for the likelihood of all classes, a higher entropy corresponds to a higher uncertainty. Class: ABCD LLM1:[0.45, 0.33, 0.01, 0.21] -> Answer A, uncertainty 1.57 LLM2:[0.99, 0.01, 0.00, 0.00] -> Answer A, uncertainty 0.08 LLM3:[0.30, 0.35, 0.15, 0.20] -> Answer B, uncertainty 1.92

[0026] Figure 2 is a graph showing the accuracy rate relative to confidence level, according to the characteristics of LLM.

[0027] The graph in Figure 2 is called a Reliability Diagram, with the horizontal axis representing confidence and the vertical axis representing accuracy. Confidence is expressed as [=1-uncertainty]. In other words, "high confidence" means "low uncertainty," and it means that you are confident in classifying the responses. On the other hand, "low confidence" means "high uncertainty," and it means that you are not confident in classifying the responses.

[0028] According to Figure 2, the following three graphs are represented based on the characteristics of LLM. Figure 2(a) shows that even with increased confidence, the accuracy rate does not increase. In other words, this LLM does not have an accuracy rate proportional to the confidence level, exhibiting the characteristics of an overconfident model. Figure 2(b) shows a relatively high accuracy rate even with low confidence levels. In other words, this LLM has a relatively high accuracy rate relative to its confidence level, exhibiting the characteristics of an underconfident model. Figure 2(c) shows that confidence and accuracy are proportional, indicating that this LLM has the most appropriate model characteristics.

[0029] <<Training Stage>> Figure 3 is a functional configuration diagram of the estimation device during the training phase of the present invention.

[0030] Estimation device 1, in its training phase, estimates responses based on class classification using multiple (e.g., three) LLM-X, Y, and Z models. Estimation device 1 is assumed to have training data that associates question statements related to class classification with the correct class answers. According to Figure 3, the estimation device 1 comprises a response collection unit 11, a confidence weighting unit 12, and a metamodel 13. These functional components are realized by executing a program that enables the computer installed in the estimation device to function. Furthermore, the processing flow of these functional components can also be understood as the estimation method of the device.

[0031] Figure 4 is a graph showing the characteristics of accuracy rate against confidence level for each LLM.

[0032] As shown in Figure 4, the multiple LLMs X, Y, and Z have different characteristics in terms of accuracy relative to confidence. If these LLMs are used as they are, even if the classification performance of the answers is maintained, they cannot be used as LLMs with appropriate confidence levels. In other words, even if the class classification answer is correct, the uncertainty may be high.

[0033] Furthermore, even if we simply ensemble the three LLMs (by mean likelihood), the answer might predict class A while also correctly identifying class B. However, because the answer predicts class A, the classification performance is maintained, and at first glance, this seems to be acceptable.

[0034] Figure 5 is an explanatory diagram of the response collection unit.

[0035] [Response Collection Section 11] The response collection unit 11 sends the training data question text for each LLM and obtains the classification probability for each class as the response. Class: ABCD LLM-X:[···,···,···, ···] -> Answer A LLM-Y:[···,···,···, ···] -> Answer A LLM-Z:[···,···,···, ···] -> Answer B

[0036] The response collection unit 11 normalizes the classification probability of each class for each LLM using the softmax function so that the sum of the classification probabilities of the classes is 1.0, and calculates the "likelihood" for each class. The softmax function, for example, if it has k (=4) outputs, expresses the m-th output ym from the k likelihoods xi as follows: ym = xm / Σ i=1 k xi The class with the highest likelihood can be determined to be the one classified as the answer.

[0037] As shown in Figure 5, the likelihood p for each class is calculated using the softmax function, for example, as follows. Class: ABCD LLM-X:[0.361, 0.235, 0.212, 0.192] -> Answer A LLM-Y:[0.991, 0.001, 0.005, 0.003] -> Answer A LLM-Z:[0.210, 0.622, 0.118, 0.050] -> Answer B

[0038] The response collection unit 11 outputs the likelihood of each class for each LLM to the confidence weighting unit 12.

[0039] Figure 6 is an explanatory diagram of the confidence weighting unit and metamodel in the present invention.

[0040] [Confidence weighting unit 12] The confidence weighting unit 12 outputs a "weighted likelihood" for each LLM by multiplying the likelihood of each class of responses by the "confidence weight" of that LLM. Weighted likelihood = Confidence weight tw × Likelihood p

[0041] The "confidence weights" are trained as learning parameters for each LLM. The confidence weights tw are trained for each LLM by backpropagation from the metamodel 13 so that the loss between the combined likelihood of each class output from the metamodel 13 and the likelihood of each class in the correct answer becomes small (the likelihoods match). In other words, LLMs with high predictive performance that are useful as references will have high confidence weights tw, and LLMs with low predictive performance that are not useful as references will have low confidence weights tw. Then, a confidence weight tw for each LLM is assigned to the likelihood of each class of response.

[0042] Calculating confidence weights using likelihood quantitatively evaluates the likelihood of a response in each LLM. Even if a response correctly predicts the correct class based on likelihood, it's important to consider whether the LLM was unsure of the answer (high uncertainty).

[0043] [Metamodel 13] Metamodel 13 takes the weighted likelihoods tw×p for each class in all LLM-X, Y, and Z as inputs in parallel and outputs the combined likelihood for each class.

[0044] The metamodel 13 may be, for example, a fully connected neural network consisting of three layers: an input layer, a hidden layer, and an output layer. It propagates in one direction from the input layer to the output layer. The input layer has units equal to "number of LLMs × number of classes" and inputs the weighted likelihoods of each class of responses in all LLMs in parallel. The hidden layer is connected to the input layer and consists of multiple layers arranged in a graph. Each layer has multiple units, and the parameters of the function connecting the units from the forward layer to the backward layer are calculated during training. The output layer contains units representing the "number of classes" that will be the answer, and is connected to the hidden layer. It then outputs the combined likelihood of each integrated class.

[0045] For example, suppose we predict four answer choices, classes A through D, from each of the three LLMs. In this case, the input layer will have 12 units (= 4 × 3), and the output layer will have 4 units.

[0046] Then, the metamodel 13 backpropagates the error so that the "combined likelihood" of each class matches the likelihood of each class in the correct answer. Combined likelihoods for each class output from Metamodel 13: [··, ··, ··, ··] Likelihood of each class when answer A is correct: [1, 0, 0, 0]

[0047] The output layer of the metamodel 13 may, for example, calculate the cross-entropy loss. Using the cross-entropy loss, the model is trained by backpropagation to minimize the difference between probability distributions. For example, the Kullback-Leibler divergence (KL divergence) can be used to measure the difference between probability distributions. P: Combined likelihood for each class Q: What is the likelihood of each class in the correct answer? DKL(P||Q)=Σ i=1 N P(xi)·log(P(xi) / Q(xi)) KL divergence represents the amount of information loss that occurs when approximating a probability distribution P using a probability distribution Q. Other embodiments may include using JS divergence or Wasserstein distance.

[0048] Figure 7 is an explanatory diagram of the metamodel that further inputs embedded representations of the question text.

[0049] According to Figure 7, compared to Figure 6, the training stage further includes an embedded representation conversion unit 14.

[0050] [Embedded Expression Conversion Unit 14] The embedding representation conversion unit 14 converts the classification question text into an embedding vector and inputs that embedding vector into the metamodel 13. Question embedding representations are created by transforming a question into an embedding representation using a pre-trained model such as BERT® or ROBERTA®. Numerous tokens are extracted from the question based on part-of-speech analysis and frequency of occurrence. These tokens are then transformed by a machine learning model into a high-dimensional embedding representation that numerically represents their meaning and characteristics. The embedding representation contains vectorized arrays of semantic features of words and sentences.

[0051] Metamodel 13 takes further question embedding representations as input in parallel and outputs the combined likelihood of each integrated class. The input layer of Metamodel 13 is further equipped with a unit that takes question embedding representations as input. As a result, the likelihoods of each class output from metamodel 13 will reflect the question.

[0052] Figure 8 is a graph showing the accuracy rate against confidence level for LLM in the present invention.

[0053] Figure 8(a) shows a graph representing the accuracy against confidence, after training the metamodel 13 to output the likelihood of each correct class. No advantages can be seen compared to the graphs for each LLM in Figure 4 mentioned earlier. On the other hand, Figure 8(b) shows a graph representing the accuracy rate against confidence level after training the metamodel 13 with confidence weights according to the present invention. The confidence level and accuracy rate are proportional, and it can be seen as a confidence-appropriate model, as shown in Figure 2(c) above.

[0054] <Regarding uncertainty in this invention> According to this invention, uncertainty is not directly trained to improve it. Instead, the likelihood of each class representing the answer is trained as a "confidence weight" for each LLM, thereby controlling the weighted likelihood and ultimately leading to an improvement in uncertainty. In other words, according to the present invention, the overconfidence or underconfidence characteristics of each LLM are suppressed to an appropriate level of confidence by using a metamodel that integrates multiple LLMs and controlling the likelihood input to that metamodel with confidence weights. Ultimately, since uncertainty is calculated from likelihood, making the likelihood appropriate also makes the uncertainty appropriate. According to the present invention, correcting the likelihood with confidence weights results in correcting the uncertainty as well. In classification, the class with the highest likelihood is selected, but this invention does not aim to improve classification performance. The most expected effect of this invention is to make the uncertainty appropriate. To this end, confidence weights are assigned to the likelihood, thereby training the metamodel to become a model with appropriate confidence.

[0055] <<Estimated stage>> Figure 9 is a functional configuration diagram of the estimation device in the estimation stage of the present invention.

[0056] As shown in Figure 9, the LLM with the highest confidence weight is selected based on the training stage described above, and the estimation stage is performed using only that LLM. As shown in Figure 9, for example, let's assume that LLM-Y has the highest confidence weight.

[0057] During the operational phase, attempting to obtain confident and appropriate answers using all multiple LLMs for questions and answers that arise in real time is costly in terms of time and computational resources. Therefore, according to the present invention, only the LLM with the highest confidence weight during the training phase is used during the estimation phase. In other words, the most reliable and useful LLM can be selected from among multiple LLMs.

[0058] As shown in Figure 9, the estimation device 1 receives a question from the user regarding class classification and transmits that question to the LLM-Y from the response collection unit 11. In response, the response collection unit 11 obtains the likelihood of each class as a response from the LLM-Y. Next, the confidence weighting unit 12 outputs a weighted likelihood obtained by multiplying the likelihood of each class of response by the confidence weight of LLM-Y. Next, the metamodel 13 takes the weighted likelihoods of each class in LLM-Y as input in parallel and outputs the combined likelihood of each class. The combined likelihood of each class, as the answer, is then output to the uncertainty calculation unit 15.

[0059] [Uncertainty Calculation Unit 15] The uncertainty calculation unit 15 receives the combined likelihood of each class of responses estimated by LLM for the operational phase and calculates the uncertainty of the LLM. The uncertainty calculation unit 15 calculates the uncertainty using one of the calculation examples 1, 2, or 3 described above, or any other example.

[0060] Furthermore, the uncertainty calculation unit 15 outputs responses with an uncertainty level above a predetermined threshold as an alarm to the application. This is because a high level of uncertainty indicates a greater need to pay attention to the classification of the response.

[0061] For example, in the case of an application such as a customer support chatbot, its alarms could be routed to a human operator. For answers with high uncertainty, the system could be collaborative with a human, with the operator relaying the information to the user. For example, if a user submits a question about a product they wish to purchase, and there is a high degree of uncertainty regarding the LLM-Y answer, the operator may suggest a different product to the user. Of course, as shown in Figure 9, the LLM-Y model, which has a high confidence weight, is selected, resulting in a relatively confident and appropriate model. However, if the LLM-Y model has an overconfident characteristic, it tends to lead to definitive product recommendations, while if it has an underconfident characteristic, it tends to lead to conservative product recommendations. For these reasons, it is preferable to apply not only the predicted probability in classification as an answer, but also uncertainty to actual services.

[0062] As described in detail above, the program, estimation apparatus, and method of the present invention can improve uncertainty when estimating responses based on class classification using multiple large-scale language models.

[0063] Furthermore, this will enable, for example, "improvement of uncertainty in responses in large-scale language models," thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs): "Build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0064] Various changes, modifications, and omissions to the scope of the technical concept and viewpoint of the present invention can be readily made by those skilled in the art with respect to the various embodiments of the present invention described above. The above description is merely illustrative and is not intended to limit the present invention in any way. The present invention is limited only to what is limited by the claims and their equivalents. [Explanation of Symbols]

[0065] 1 Estimation device 11. Response Collection Department 12. Confidence weighting section 13 Metamodel 14 Embedded Expression Conversion Unit 15. Uncertainty Calculation Unit

Claims

1. In a program that operates a computer installed in a device that estimates answers to question sentences using multiple large-scale language models, During the training phase, training data was used that matched class classification questions with the correct class answers. For each large-scale language model, a response collection method is provided that sends the question text of the training data and obtains the likelihood of each class as the response. A confidence weighting means that outputs a weighted likelihood for each large-scale language model by multiplying the likelihood of each class of response by the confidence weight of the large-scale language model, A metamodel that takes the weighted likelihoods of each class in all large-scale language models as parallel inputs, outputs the combined likelihood of each class, and backpropagates the error to the confidence weighting means so that the combined likelihood of each class matches the likelihood of each class in the correct answer. A program characterized by its ability to make a computer function.

2. The metamodel is An input layer that takes the weighted likelihoods of each class in all large-scale language models as parallel inputs and backpropagates the errors to the confidence weighting means, A hidden layer connected to the input layer, An output layer connected to a hidden layer outputs the combined likelihood of each integrated class, and backpropagates the error so that the combined likelihood of each class matches the likelihood of each correct class. The program according to claim 1, characterized in that it causes a computer to function in such a way as to have

3. The response collection means sends the training data question sentences to each large-scale language model, obtains the classification probability for each class as a response, and outputs the likelihood, which is the classification probability normalized by the softmax function, to the confidence weighting means. The program according to claim 1, characterized in that it causes a computer to function in such a way.

4. An embedded expression conversion means that converts the class classification question text into an embedded question text expression, and inputs the embedded question text expression into a metamodel. and make it work The metamodel takes further parallel inputs of embedded question representations and outputs the combined likelihood of each integrated class. The program according to claim 1, characterized in that it causes the computer to function in such a way.

5. Regarding the estimation stage, Using only the large-scale language models with the highest confidence weights in the confidence weighting method, The response collection method involves sending the target question text regarding class classification to a large-scale language model and obtaining the likelihood of each class as a response. The confidence weighting means outputs a weighted likelihood obtained by multiplying the likelihood of each class of response by the confidence weight of the large-scale language model. The metamodel takes the weighted likelihoods of each class in the large-scale language model as input in parallel and outputs the combined likelihood of each class. An uncertainty calculation means inputs the combined likelihood of each class of the estimated response, calculates the uncertainty of the large-scale language model, and outputs responses whose uncertainty exceeds a predetermined threshold as an alarm to the application. The program according to claim 1, characterized in that it further enables the computer to function.

6. The uncertainty calculation method is, Among the likelihoods of all classes, the lower the likelihood of the highest likelihood, the higher the uncertainty. Among all the likelihoods of each class, the smaller the difference between the highest likelihood and the second highest likelihood, the higher the uncertainty. For the likelihood of all classes, higher entropy corresponds to higher uncertainty. The program according to claim 5, characterized in that it causes a computer to function in such a way.

7. Regarding the training phase, An embedded expression conversion means that converts the class classification question text into an embedded question text expression, and inputs the embedded question text expression into a metamodel. and make it work The metamodel takes the embedded representations of the question sentences as input in parallel and outputs the combined likelihood of each integrated class. Regarding the estimation stage, The embedded representation conversion means converts the target question text related to class classification into a question text embedded representation, and inputs the said question text embedded representation into the metamodel. The program according to claim 6, characterized in that it causes the computer to function in such a way.

8. In an estimation device that uses multiple large-scale language models to estimate answers to questions, During the training phase, training data was used that matched class classification questions with the correct class answers. For each large-scale language model, a response collection method is provided that sends the question text of the training data and obtains the likelihood of each class as the response. A confidence weighting means that outputs a weighted likelihood for each large-scale language model by multiplying the likelihood of each class of response by the confidence weight of the large-scale language model, A metamodel that takes the weighted likelihoods of each class in all large-scale language models as parallel inputs, outputs the combined likelihood of each class, and backpropagates the error to the confidence weighting means so that the combined likelihood of each class matches the likelihood of each class in the correct answer. An estimation device characterized by having the following features.

9. In an estimation method for a device that estimates answers to question sentences using multiple large-scale language models, The device is During the training phase, training data was used that matched class classification questions with the correct class answers. For each large-scale language model, the training data question sentences are sent, and the likelihood of each class is obtained as the answer. The first step involves using a learning model to output a weighted likelihood for each large-scale language model, obtained by multiplying the likelihood of each class of responses by the confidence weight of that large-scale language model. The second step involves using a metamodel to input the weighted likelihoods of each class in all large-scale language models in parallel, outputting the combined likelihood of each class, and backpropagating the error to the confidence weighting means so that the combined likelihood of each class matches the likelihood of each class in the correct answer. An estimation method for an apparatus characterized by performing the following.

Images