Model reliability evaluation method and apparatus

Abstract

This evaluates the reliability of the output of models or artificial intelligence, such as generative AI or LLMs. [Solution] The system comprises means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α when the input u is given, means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the input u is given, means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, and if the difference d is not within a predetermined range / is greater than or equal to a predetermined value, means for calculating a value z_in that represents the region in which the internal calculation value exists based on the internal calculation value X_β.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for evaluating the reliability of a model, and more particularly, to a method and apparatus for evaluating the reliability of an output of artificial intelligence.

Background Art

[0002] As the background art of this technology, there is Japanese Patent Application Laid-Open No. 2021-163991 (Patent Document 1). This document describes "a distortion compensation device that executes distortion compensation for a signal amplified by an amplifier in which an internal state that affects distortion characteristics changes, using a distortion compensation model, wherein the distortion compensation model includes a plurality of arithmetic models having distortion compensation characteristics for the amplifier in different internal states, and a synthesizer that synthesizes the plurality of arithmetic models at a synthesis ratio according to the changing internal state. (See [Claim 1])."

[0003] Also, there is Japanese Patent Application Laid-Open No. 2005-267025 (Patent Document 2). This document describes "a threshold determination unit that determines a threshold of a characteristic value of a model having characteristic values according to variables, a characteristic value calculation unit that calculates the characteristic value of a given model based on the variables of the model and writes the variables and characteristic values of the model into a memory map, a distance calculation unit that compares the characteristic value of the model with the threshold and calculates the distance to another model closest to the model in the variable space when the characteristic value of the model exceeds the threshold, a model generation unit that generates a new model within the range of the distance and outputs the variables of the model to the characteristic value calculation unit, and a region extraction unit that extracts a region of a model having a characteristic value exceeding the threshold from the memory map. (See [Claim 1])."

[0004] There is also arXiv:2404.08679 (Non-Patent Literature 1). This document states, "In our paper, we propose that pretrained base models can function as a repository of prior knowledge for OOD data relative to in-distribution data, effectively acting as OOD proxy distributions. Guided by this insight, we discover that the likelihood ratio between the base model and its finetuned counterpart serves as an effective criterion for detecting OOD data. Moreover, for LLM-based question-answering (QA) systems, the same likelihood ratio excels in detecting OOD questions. By identifying and rejecting these OOD questions, we can greatly enhance the robustness of current QA systems. (See Introduction)" [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-163991 [Patent Document 2] Japanese Patent Publication No. 2005-267025 [Non-patent literature]

[0006] [Non-Patent Document 1] arXiv:2404.08679 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the aforementioned prior art (Patent Document 1) synthesizes the outputs of two models with different characteristics and does not evaluate the reliability of the output values ​​of a model that has been trained to specialize in knowledge in a particular field.

[0008] Furthermore, the prior art (Patent Document 2) selects multiple models with parameter values ​​that are within a certain distance range from the parameter values ​​that perform well as candidates for the optimal model when determining the parameter values ​​of the model. It does not evaluate the reliability of the output values ​​of a model that has been trained to specialize in knowledge of a particular field based on distance.

[0009] Furthermore, prior art (Non-Patent Literature 1) involves preparing artificial intelligence A, which has learned general knowledge, and artificial intelligence B, which has additionally learned knowledge in a specific field, and inputting a question related to that specific field into both artificial intelligences. When the difference in the output between the two is small, it is determined that artificial intelligence B has not learned the knowledge in that specific field (it is a question outside of its learning scope, and the reliability of its answer to it is low). However, this requires operating two artificial intelligences, A and B, simultaneously, resulting in a high computational load, and the judgment is based on the difference in output, so the information used for the judgment is relatively small. Therefore, it does not take computational load into consideration, nor does it evaluate reliability with higher accuracy. [Means for solving the problem]

[0010] To solve the above problems, for example, the configuration described in the claims is adopted. This application includes several means for solving the above problems, for example, A means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α given the input u, A means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the aforementioned input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, If the difference d is not within a predetermined range or is greater than or equal to a predetermined value, based on the internal calculation value X_β, A means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, A means for calculating the confidence / certainty λ_v of the output of the model β based on the z_in, It is equipped with.

[0011] Also, for example, In the above configuration, A means for calculating at least one internal calculation value X_βv of the model / AI / artificial intelligence β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, A means for calculating the confidence / certainty λ_v of the output of the model β when the input v is given, based on the difference D_v. It is equipped with.

[0012] Also, for example, A method / apparatus for calculating the reliability / confidence / certainty of the output / answer of a model / artificial intelligence / AI / generative AI / large-scale language model / LLM / transformer(α) using a computer having at least a processor and memory, The aforementioned processor, The parameter values ​​were determined (trained) using training data A. When input u (word / sentence) is input to (model) α (output y_α (word / sentence) is obtained), the values ​​X_α: x1_α, x2_α, ..., xn_α of the internal parameters X: x1, x2, ..., xn_α of the model α are calculated. The parameter values ​​of (model) α were updated (additionally trained) using training data B, and the input u (word / sentence) was input to (model) β (output y_β (word / sentence) was obtained). The values ​​of the internal parameters X:x1,x2,...,xn of model β, X_β:x1_β,x2_β,...,xn_β, were calculated. Based on the foregoing \(X_{\alpha}:x1_{\alpha},x2_{\alpha},\cdots,xn_{\alpha}\), at least one (representative value) \(z_{\alpha1}\) is calculated. Based on the foregoing \(X_{\beta}:x1_{\beta},x2_{\beta},\cdots,xn_{\beta}\), at least one (representative value) \(z_{\beta}\) is calculated. Based on the \(z_{\alpha}\) and the \(z_{\beta}\), the difference \(d\) between the \(z_{\alpha}\) and the \(z_{\beta}\) is calculated. When the difference \(d\) is greater than or equal to a predetermined value, the \(z_{\beta}\) (or a value calculated based on the \(z_{\beta}\)) is set as \(z_{in}\). When the input \(v\) (word / sentence) is input to the (model) \(\beta\) and the output \(y_{\beta}\) (word / sentence) is obtained, the values \(X_{\beta v}:x1_{\beta v},x2_{\beta v},\cdots,xn_{\beta v}\) of the internal parameters \(X:x1,x2,\cdots,xn\) of the model \(\beta\) are calculated. Based on each of the \(X_{\beta v}:x1_{\beta v},x2_{\beta v},\cdots,xn_{\beta v}\) and the \(z_{in}\), differences \(D_{v}:D1_{v},D2_{v},\cdots,Dn_{v}\) are calculated. Based on the differences \(D_{v}:D1_{v},D2_{v},\cdots,Dn_{v}\), the reliability / accuracy \(\lambda_{v}\) of the output \(y_{\beta v}\) of the model \(\beta\) obtained after inputting the input \(v\) (word / sentence) to the model \(\beta\) is calculated. The process is executed.

[0013] Also, for example, The internal calculation values \(X_{\alpha},X_{\beta}\) are internal calculation values when a plurality of \(u1,u2,\cdots,um\) are respectively input. Specifically, The internal calculation value \(X_{\alpha}\) when the \(u_{k}\) is input is \([x1_{\alpha k},x2_{\alpha k},\cdots,xn_{\alpha k}]\). The internal calculation value \(X_{\beta}\) when the \(u_{k}\) is input is \([x1_{\beta k},x2_{\beta k},\cdots,xn_{\beta k}]\). This is characterized by.

[0014] Also, for example, The means for calculating the value \(X_{\alpha}\) of the internal parameter \(X\) of the model \(\alpha\) is The internal parameters \(X1_{\alpha},x2_{\alpha},\cdots,xn_{\alpha}\) when the input \(u\) (word / sentence) is input to the (model) \(\alpha\), and When the input u (word / sentence) is input to the model α and the output y_α (word / sentence) of the model α is being calculated, X_α consisting of both internal parameters of the internal parameters Xn+1_α, xn+2_α, ···, xn+m_α and are calculated, The means for calculating the value X_β of the internal parameter X of the model β is the internal parameters X1_β, x2_β, ···, xn_β when the input u (word / sentence) is input to the (model) β, and X_β consisting of both internal parameters of the internal parameters Xn+1_β, xn+2_β, ···, xn+m_β when the input u1 (word / sentence) is input to the model β and the output y_β (word / sentence) of the model β is being calculated, and are calculated, (means for calculating at least one (representative value) z_α based on the X_α, and means for calculating at least one (representative value) z_β based on the X_β, and means for calculating the difference d between the z_α and the z_β based on the z_α and the z_β, and means for setting the z_β (or a value calculated based on the z_β) as z_in when the difference d is greater than or equal to a predetermined value), the internal parameters X1_βv, x2_βv, ···, xn_βv when the input v (word / sentence) is input to the (model) β, and X_βv consisting of both internal parameters of the internal parameters Xn+1_βv, xn+2_βv, ···, xn+m_βv when the input v1 (word / sentence) is input to the model β and the output y_βv (word / sentence) of the model β is being calculated, and are calculated, means for calculating differences D_v: D1_v, D2_v, ···, Dn_v, Dn+1_v, Dn+2_v, ···, Dn+m_v based on each of the X_βv: x1_βv, x2_βv, ···, xn_βv, xn+1_βv, xn+2_βv, ···, xn+m_βv and the z_in, A means for calculating the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) to the model β based on the difference D_v, and It is equipped with.

[0015] Also, for example, The k of the internal parameter values ​​xk_α, xk_β is, The word number or token number in the aforementioned input u or the aforementioned input v. It is characterized by the following:

[0016] Also, for example, The means for calculating at least one (representative value) z_α or (representative value) z_β is: This is the center vector (mean vector) of x1_α, x2_α, ..., xn_α. It is characterized by the following:

[0017] Also, for example, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. It is characterized by the following:

[0018] Also, for example, The input u1 (word / sentence) is data from the same "field" as training data B. The output y_β (word / sentence) satisfies the predetermined conditions (is correct), Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the aforementioned z_β (or a value calculated based on z_β). It is characterized by the following:

[0019] Also, for example, The aforementioned model β calculates the probability Pj for multiple output candidates, The second confidence / certainty level C_v in the output of model β is obtained by multiplying the confidence / certainty level λ_v of the output of model β by the probability Pj. It is characterized by the following:

[0020] Also, for example, Models α and β have multiple layers and multiple types of internal parameters. The above processing is performed on the multiple types of internal parameters in the multiple layers. It is characterized by the following:

[0021] Also, for example, Model α and Model β are transformers, The aforementioned internal calculation value is at least the attention vector, the magnitude of the attention vector, the parameter value of the FFN, the input / output embedding value, or the output value of each layer. i represents the layer number, j represents the type of internal calculation value, The internal values ​​for the aforementioned pair of inputs uk are xk_α(i,j) and xk_β(i,j), (Representative value) z_α(i,j) based on the above xk_α(i,j) and (Representative value) z_β(i,j) based on the above xk_β(i,j) And, Let the difference d(i,j) be based on the above z_α(i,j) and z_β(i,j). When the difference d(i,j) is greater than or equal to a predetermined value, the z_β1(i,j) (or the value calculated based on z_β(i,j)) is set to z_in(i,j), When the input v (word / sentence) is input to (model) β (output y_β (word / sentence) is obtained), the internal calculation value xk of the model β is x1_βv(i,j), x2_βv(i,j), ..., xn_βv(i,j), Let D1_v(i,j), D2_v(i,j), ..., Dn_v(i,j) be the differences between the above x1_βv(i,j), x2_βv(i,j), ..., xn_βv(i,j) and the above z_in(i,j), Based on the differences D1_v(i,j), D2_v(i,j), ..., Dn_v(i,j), the confidence / accuracy of the output of model β is given as λ_v or λ_v(i,j). It is characterized by the following:

[0022] Also, for example, When λ_v is less than or equal to a predetermined value or C_v is less than or equal to a predetermined value, (Notify / display on screen if you are unsure about an input or answer you have never seen before) It is characterized by the following:

[0023] Also, for example, The value based on the aforementioned D_v, Dj_v, λ_v, or C_v will be displayed on the screen. It is characterized by the following:

[0024] Also, for example, Display the word (token) v_k in the input v that corresponds to a Dk_v of a predetermined value or greater among the D1_v, D2_v, ..., Dn_v. It is characterized by the following:

[0025] Also, for example, This displays the positional relationship between z_in and D1_v, D2_v, ..., Dn_v based on the values ​​of z_in and D1_v, D2_v, ..., Dn_v. It is characterized by the following:

[0026] Also, for example, Models α and β are either LLMs or generative AIs. It is characterized by the following:

[0027] Also, for example, The aforementioned LLM or generative AI is a State Space Model (SSM) based model. It is characterized by the following:

[0028] Also, for example, A model α trained using a predetermined dataset A, Model β is a model that has been further trained on Model α using data (group) B1 selected from a data group B different from the predetermined data group A, A means for determining a representative value z_in of X_β based on the internal state X_β of the model β when the data(group) B1 or data(group) B2 selected from a different data(group) B is input to the model β, A means for calculating the internal state X_βv of the model β when an input v is input to the model β, A means for determining the difference D_v between the internal state X_βv and the representative value z_in, A means for calculating the confidence level λ_v of the input v or the output y of the model β when the input v is given, based on the difference D_v. It is equipped with.

[0029] Also, for example, Model β is a language model, A means for determining a representative value z_in of X_β based on the internal state X_β of the model β when an input u is input to the model β and / or when the output w is calculated when the input u is input, A means for calculating the internal state X_βv of the model β when an input v is input to the model β and / or when the output y is calculated when the input v is input, A means for determining the difference D_v between the internal state X_βv and the representative value z_in, A means for calculating the confidence level λ_v of the input v or the output y of the model β when the input v is given, based on the difference D_v. It is equipped with. [Effects of the Invention]

[0030] In this invention, when the internal calculation value X_α of model α for input u differs from the internal calculation value X_β of model β for input u, it is assumed that model β has acquired some knowledge from model α regarding the field of information that input u possesses, and a knowledge domain is defined based on the internal calculation value X_β. Then, when a new input v is input to model β, and the internal calculation value X_βv at that time is considerably far from the knowledge domain defined above, it is assumed that the input v is seeking information that is not in the knowledge domain acquired by model β, and the reliability of the output of model β at that time is determined to be not necessarily high.

[0031] From the above, according to the present invention, when defining the knowledge domain, although it is necessary to prepare models α and β, they can be operated sequentially and do not need to be operated simultaneously. Furthermore, when evaluating the reliability of the output with respect to the input v, only model β needs to be operated. Therefore, the computational load is lower than a configuration in which models α and β are operated simultaneously to evaluate reliability. In addition, according to the present invention, since reliability is evaluated based on information about the internal state of the model, the amount of information (number of parameters, dimensions) is much larger than that of the model output, and proportionally higher accuracy is expected. From the above, according to the present invention, it is possible to evaluate the reliability of the model output with relatively low computational load and high accuracy. [Brief explanation of the drawing]

[0032] [Figure 1] Block diagram showing the basic structure of the domain representative value z_in calculation. [Figure 2] Block diagram showing the basic configuration of the confidence score λ_v calculation (Block diagram showing the overall configuration in Examples 1, 2, 4, and 5) [Figure 3] System diagrams of reliability evaluation devices in Examples 1-7 [Figure 4] This figure shows the calculation process for the internal calculation value X_α of Model α in Example 1. [Figure 5] This figure shows the calculation process for the internal calculation value X_β of model β in Example 1. [Figure 6] Figure showing the calculation process for the difference d in Examples 1 to 7. [Figure 7] Figure showing the calculation process for the region representative value Z_in in Examples 1 to 7. [Figure 8] This figure shows the calculation process for the internal calculation value X_βv of model β in Example 1. [Figure 9] Figure showing the calculation process for the difference D_v in Examples 1 to 7. [Figure 10] Figure showing the calculation process for the confidence level λ_v in Examples 1 to 7. [Figure 11] This figure shows the calculation process for the internal calculation value X_α of Model α in Examples 2, 3, 6, and 7. [Figure 12] This figure shows the calculation process for the internal calculation value X_β of model β in Examples 2, 3, 6, and 7. [Figure 13] This figure shows the calculation process for the internal calculation value X_βv of model β in Examples 2, 3, 6, and 7. [Figure 14] Block diagram showing the overall configuration in Example 3 [Figure 15] Figure showing the calculation process for the confidence score C_v in Examples 3, 6, and 7. [Figure 16] This figure shows the calculation process for the internal calculation value X_α of Model α in Example 4. [Figure 17] This figure shows the calculation process for the internal calculation value X_β of model β in Example 4. [Figure 18] This figure shows the calculation process for the internal calculation value X_βv of model β in Example 4. [Figure 19] This figure shows the calculation process for the internal calculation value X_α of Model α in Example 5. [Figure 20] This figure shows the calculation process for the internal calculation value X_β of model β in Example 5. [Figure 21] This figure shows the calculation process for the internal calculation value X_βv of model β in Example 5. [Figure 22] Block diagram showing the overall configuration in Example 6 [Figure 23] This figure shows the calculation process for the notification flag f_alert in Example 6. [Figure 24] Block diagram showing the overall configuration in Example 7 [Figure 25] This figure shows the calculation process (part 1) of the calculation means for the display target in Example 7. [Figure 26] This figure shows the calculation process (part 2) of the calculation means for the display target in Example 7. [Figure 27] This figure shows Model α and Model β in the case of Mamba. [Figure 28] This figure shows Model α and Model β for the Mamba2 case. [Figure 29] This figure shows the calculation process for the internal calculation value X when Model α and Model β are Mamba. [Figure 30] This figure shows the calculation process for the internal calculation value X when Model α and Model β are Mamba2. [Modes for carrying out the invention]

[0033] Several embodiments will be described below with reference to the drawings. [Examples]

[0034] In this embodiment, A means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α given the input u, A means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the aforementioned input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, When the difference d is not within the predetermined range / is greater than or equal to the predetermined value, based on the internal calculation value X_β, A means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, A means for calculating the confidence / certainty λ_v of the output of the model β based on the z_in, It is equipped with.

[0035] Also, A means for calculating at least one internal calculation value X_βv of the model / AI / artificial intelligence β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, A means for calculating the confidence / certainty λ_v of the output of the model β when the input v is given, based on the difference D_v. It is equipped with.

[0036] Also, A method / apparatus for calculating the reliability / confidence / certainty of the output / answer of a model / artificial intelligence / AI / generative AI / large-scale language model / LLM / transformer(α) using a computer having at least a processor and memory, The aforementioned processor, The parameter values ​​were determined (trained) using training data A. When input u (word / sentence) is input to (model) α (output y_α (word / sentence) is obtained), the values ​​X_α: x1_α, x2_α, ..., xn_α of the internal parameters X: x1, x2, ..., xn_α of the model α are calculated. The parameter values ​​of (model) α were updated (additionally trained) using training data B, and the input u (word / sentence) was input to (model) β (output y_β (word / sentence) was obtained). The values ​​of the internal parameters X:x1,x2,...,xn of model β, X_β:x1_β,x2_β,...,xn_β, were calculated. Based on the above X_α: x1_α, x2_α, ..., xn_α, at least one (representative value) z_α1 is calculated, Based on the above X_β: x1_β, x2_β, ..., xn_β, at least one (representative value) z_β is calculated, Based on the aforementioned z_α and z_β, the difference d between z_α and z_β is calculated. When the difference d is greater than or equal to a predetermined value, the z_β (or a value calculated based on the z_β) is set to z_in. When the input v (word / sentence) is input to (model) β (output y_β (word / sentence) is obtained), the values ​​of the internal parameters X:x1,x2,...,xn of the model β, X_βv:x1_βv,x2_βv,...,xn_βv, are calculated. Based on the above X_βv: x1_βv, x2_βv, ..., xn_βv and the above z_in, the difference D_v: D1_v, D2_v, ..., Dn_v is calculated, Based on the differences D_v: D1_v, D2_v, ..., Dn_v, the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) into the model β is calculated. Execute the process.

[0037] Also, The aforementioned internal calculation values ​​X_α and X_β are the internal calculation values ​​obtained when multiple u1, u2, ..., um are inputs, and specifically, When uk is input, the internal calculation value X_α is [x1_αk, x2_αk, ..., xn_αk], When uk is input, the internal calculation value X_β is [x1_βk, x2_βk, ..., xn_βk]. It is characterized by the following:

[0038] Also, The means for calculating at least one (representative value) z_α or (representative value) z_β is: This is the center vector (mean vector) of x1_α, x2_α, ..., xn_α. It is characterized by the following:

[0039] Also, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. It is characterized by the following:

[0040] Also, The input u1 (word / sentence) is data from the same "field" as training data B. The output y_β (word / sentence) satisfies the predetermined conditions (is correct), Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the aforementioned z_β (or a value calculated based on z_β). It is characterized by the following:

[0041] Furthermore, in this embodiment, A model α trained using a predetermined dataset A, Model β is a model that has been further trained on Model α using data (group) B1 selected from a data group B different from the predetermined data group A, A means for determining a representative value z_in of X_β based on the internal state X_β of the model β when the data(group) B1 or data(group) B2 selected from a different data(group) B is input to the model β, A means for calculating the internal state X_βv of the model β when an input v is input to the model β, A means for determining the difference D_v between the internal state X_βv and the representative value z_in, A means for calculating the confidence level λ_v of the input v or the output y of the model β when the input v is given, based on the difference D_v. It may also be provided with

[0042] The above configuration is shown below.

[0043] Figure 1 is a block diagram showing the basic configuration of the domain representative value z_in calculation in this embodiment. Model α2 is the base model, and model β3 is a model that has been further trained using data from a predetermined domain on model α2. Based on the internal states X_α and X_β of model α2 and model β3 when input u (here referred to as text / prompt) is input, the difference calculation means d4 calculates the difference d between X_α and X_β. Then, based on this difference d and the internal state X_β, the domain representative value z_in calculation means 5 calculates the domain representative value z_in.

[0044] Figure 2 is a block diagram showing the basic configuration of the confidence λ_v calculation in this embodiment. The process up to the calculation of the region representative value z_in in the upper half of the block diagram is the same as in Figure 1 and is therefore omitted. Based on the internal state X_βv and the region representative value z_in when input v (here referred to as text / prompt) is input to model β3, the difference D_v calculation means 7 calculates the difference D_v. Then, based on this difference D_v, the confidence λ_v calculation means 8 calculates the confidence λ_v.

[0045] Figure 3 is a system diagram of a reliability evaluation device (region representative value z_in calculation device) 1 that implements the processing shown in Figure 1, or a reliability evaluation device (reliability calculation device) 6 that implements the processing shown in Figure 2. The reliability evaluation device (1,6) is provided with an input circuit 26 that processes external signals. External signals here refer to, for example, the aforementioned input u or input v. These external signals pass through the input circuit 26, become input signals, and are sent to the input / output port 27. Each input information sent to the input / output port 27 is written to the RAM 24 via the data bus 25, or stored in the storage device 21. The ROM 23 or storage device 21 contains the processing described later, which is executed by the CPU 22. At that time, calculations are performed using the values ​​written to the RAM 24 or storage device 21 as appropriate. Of the calculation results, the information (values) to be sent to the outside is sent to the input / output port 27 via the data bus 25 and sent to the output circuit 28 as an output signal. The signal from output circuit 28 is output to the outside. This external signal refers to the aforementioned output, etc.

[0046] In other words, the difference calculation means d4, the area representative value z_in calculation means 5, and the confidence level λ_v calculation means 8 shown in Figures 1 and 2, and the difference D_v calculation means 7 shown in Figure 2 are all realized by the CPU 22 shown in Figure 3 executing the processing written to the ROM 23 or storage device 21. Similarly, the various means and each functional unit described later are also realized by the CPU 22 shown in Figure 3 executing the processing written to the ROM 23 or storage device 21.

[0047] The details of each process are explained below.

[0048] <Calculation of Model α and Internal State X_α (Figure 4)> This process performs calculations on the internal calculation value X_α of model α. Specifically, as shown in Figure 4, it extracts the internal calculation values ​​X_α: x1_α, x2_α, ..., xn_α of the MLP (Multilayer Perceptron) for the input u (here, text / prompt).

[0049] Note that X_α can be a single value (x1_α) or multiple values ​​(x1_α, x2_α, ..., xn_α). Also, while it is best to decide which internal calculation value of the MLP to use empirically, you may follow other guidelines if available.

[0050] <Calculation of model β and internal state X_β (Figure 5)> This process calculates the internal calculation value X_β of model β. Specifically, as shown in Figure 5, it extracts the internal calculation values ​​X_β: x1_β, x2_β, ..., xn_β of the MLP for the input u.

[0051] Note that X_β can be a single value (x1_β) or multiple values ​​(x1_β, x2_β, ..., xn_β). Furthermore, while it is best to determine which internal MLP calculation value to use empirically, other guidelines may be followed if available. However, it should be the same variable as X_α shown in Figure 4.

[0052] <Difference d calculation means (Figure 6)> This process calculates the difference d between X_α and X_β. Specifically, the following calculation is performed as shown in Figure 6. - Calculate the center vector (mean vector) C_X_α of X_α... (6-1) - Calculate the center vector (mean vector) C_X_β of X_β... (6-2) - Let the difference d be the L2 distance between C_X_α and C_X_β... (6-3)

[0053] Furthermore, it is also acceptable to use only vectors where the magnitudes (L1 norm, L2 norm, etc.) of X_α and X_β are greater than or equal to a predetermined value for the calculation. In addition, the difference d represents the difference between vectors C_X_α and C_X_β, and any distance between vectors, such as the cosine distance or Mahalanobis distance, may be used.

[0054] <Mechanism for calculating the representative value z_in of the region (Figure 7)> In this process, the region representative value z_in is calculated based on the difference d and the internal state X_β. Specifically, the following calculations are performed as shown in Figure 7. - When d ≥ K_d, let z_in be the center vector of X_β... (7-1)

[0055] The value of K_d should generally be determined empirically. Alternatively, the process described in (7-1) above may be applied only when the output of model β satisfies certain conditions. The conditions for the output of model β to satisfy certain conditions refer, for example, to cases where the output is valid. Whether or not it is valid can be determined by comparing it to objective indicators if available, or by human judgment.

[0056] <Calculation of model β and internal state X_βv (Figure 8)> This process calculates the internal value X_βv of model β. Specifically, as shown in Figure 8, the following calculations are performed. - Extract the MLP's internal calculation values ​​X_βv: x1_βv, x2_βv, ..., xn_βv for the input v (here, text / prompt) ... (8-1)

[0057] Note that X_βv may be a single value (x1_βv) or multiple values ​​(x1_βv, x2_βv, ..., xn_βv). Furthermore, while it is best to determine which internal MLP calculation value to use empirically, other guidelines may be followed if available. However, X_α and X_β shown in Figures 4 and 5 are the same variable.

[0058] <Difference D_v calculation means (Figure 9)> In this process, the difference D_v is calculated based on the internal state X_βv and the region representative value z_in. Specifically, the following calculations are performed as shown in Figure 9. - Dk_βv=||z_in-xk_βv|| 2 Perform the operation on (k: 1, 2, ..., n) ... (9-1) - Let D_v be the mean of Dk_βv...(9-2)

[0059] Note that Dk_βv may be any value representing the distance between vectors, such as the cosine distance or Mahalanobis distance between z_in and xk_βv. Also, D_v may be a value that represents Dk_βv, such as the maximum value of Dk_βv.

[0060] <Reliability λ_v calculation method (Figure 10)> In this process, the confidence score λ_v is calculated based on the difference D_v. Specifically, as shown in Figure 10, the confidence score λ_v is obtained via the function f(D_v) based on the difference D_v. As shown in Figure 10, the function f(D_v) has the characteristic that λ_v decreases as the value of D_v increases. Note that this function f(D_v) may be set empirically or according to the required specifications in a table lookup format, or a suitable mathematical formula may be used if one can be obtained.

[0061] It is preferable that inputs u and v be data from a specific domain used to train model β, or data belonging to that specific domain. Furthermore, although inputs u and v are presented as sentences / prompts, they may consist of multiple sentences, and the same processing can be performed. In this case, the domain representative value z_in may be calculated for each sentence or for all sentences at once; the choice should depend on the desired performance.

[0062] In this configuration, model α is the base model, and model β is a model that has been further trained using data from a predetermined domain. When the data from the predetermined domain used to train model β, or data belonging to that predetermined domain, is taken as input u, and the internal calculation value X_α of model α for that input u is different from the internal calculation value X_β of model β for that input u, and preferably the output of model β is valid, then it is assumed that model β has acquired some additional knowledge from model α in the predetermined domain to which the information contained in input u belongs, and the knowledge domain is defined based on the internal calculation value X_β.

[0063] Then, a new input v, which is data from a predetermined field used to train model β or data belonging to that predetermined field, is input to model β. If the internal calculation value X_βv at that time is considerably far from the knowledge domain defined above, then it is determined that the input v is seeking information that is not in the knowledge domain acquired by model β, and the reliability of the output of model β at that time is not necessarily high.

[0064] Thus, with the configuration shown in this embodiment, when evaluating the reliability of the output with respect to the input v, it is sufficient to operate only the model β. Furthermore, since the reliability is evaluated based on information about the internal state of the model, the amount of information (number of parameters, dimensions) is much greater than when using the model output, and proportionally higher accuracy can be expected.

[0065] Based on the above, this configuration allows for low computational load and high-precision evaluation of the reliability of the model's output. [Examples]

[0066] In this embodiment, A means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α given the input u, A means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the aforementioned input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, When the difference d is not within the predetermined range / is greater than or equal to the predetermined value, based on the internal calculation value X_β, means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, It is equipped with.

[0067] Also, A means for calculating at least one internal calculation value X_βv of the model / AI / artificial intelligence β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, A means for calculating the confidence / certainty λ_v of the output of the model β when the input v is given, based on the difference D_v. It is equipped with.

[0068] Also, A method / apparatus for calculating the reliability / confidence / certainty of the output / answer of a model / artificial intelligence / AI / generative AI / large-scale language model / LLM / transformer(α) using a computer having at least a processor and memory, The aforementioned processor, The parameter values ​​were determined (trained) using training data A. When input u (word / sentence) is input to (model) α (output y_α (word / sentence) is obtained), the values ​​X_α: x1_α, x2_α, ..., xn_α of the internal parameters X: x1, x2, ..., xn_α of the model α are calculated. The parameter values ​​of (model) α were updated (additionally trained) using training data B, and the input u (word / sentence) was input to (model) β (output y_β (word / sentence) was obtained). The values ​​of the internal parameters X:x1,x2,...,xn of model β, X_β:x1_β,x2_β,...,xn_β, were calculated. Based on the above X_α: x1_α, x2_α, ..., xn_α, at least one (representative value) z_α1 is calculated, Based on the above X_β: x1_β, x2_β, ..., xn_β, at least one (representative value) z_β is calculated, Based on the aforementioned z_α and z_β, the difference d between z_α and z_β is calculated. When the difference d is greater than or equal to a predetermined value, the z_β (or a value calculated based on the z_β) is set to z_in. When the input v (word / sentence) is input to (model) β (output y_β (word / sentence) is obtained), the values ​​of the internal parameters X:x1,x2,...,xn of the model β, X_βv:x1_βv,x2_βv,...,xn_βv, are calculated. Based on the above X_βv: x1_βv, x2_βv, ..., xn_βv and the above z_in, the difference D_v: D1_v, D2_v, ..., Dn_v is calculated, Based on the differences D_v: D1_v, D2_v, ..., Dn_v, the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) into the model β is calculated. Execute the process.

[0069] Also, The aforementioned internal calculation values ​​X_α and X_β are the internal calculation values ​​obtained when multiple u1, u2, ..., um are inputs, and specifically, When uk is input, the internal calculation value X_α is [x1_αk, x2_αk, ..., xn_αk], When uk is input, the internal calculation value X_β is [x1_βk, x2_βk, ..., xn_βk]. It is characterized by the following:

[0070] Also, The means for calculating at least one (representative value) z_α or (representative value) z_β is: This is the center vector (mean vector) of x1_α, x2_α, ..., xn_α. It is characterized by the following:

[0071] Also, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. It is characterized by the following:

[0072] Also, The input u1 (word / sentence) is data from the same "field" as training data B. The output y_β (word / sentence) satisfies the predetermined conditions (is correct), Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the aforementioned z_β (or a value calculated based on z_β). It is characterized by the following:

[0073] especially, The k of the internal parameter values ​​xk_α, xk_β is, This is the word number or token number in the aforementioned input u or input v.

[0074] Also, Models α and β are either LLMs or generative AIs.

[0075] The above configuration is shown below.

[0076] Figure 1 is a block diagram showing the basic configuration of the region representative value z_in calculation in this embodiment, but it is the same as in Embodiment 1, so it will not be described in detail.

[0077] Figure 2 is a block diagram showing the basic configuration of the confidence λ_v calculation in this embodiment, and since it is the same as in Embodiment 1, it will not be described in detail.

[0078] Figure 3 is a system diagram of the device that implements the process shown in Figure 1 or Figure 2, but it is the same as in Example 1, so it will not be described in detail.

[0079] The details of each process are explained below.

[0080] <Calculation of Model α and Internal State X_α (Figure 11)> This process calculates the internal value X_α of model α. Specifically, this is shown in Figure 11. In this embodiment, model α is a decoder-based model of a transformer. There is a lot of literature on transformers and their decoders, so they will not be described in detail here. Also, the embedding process, direct terms (residual paths), and regression parts are not relevant to this embodiment, so they are not shown or explained.

[0081] The input u (here, a sentence / prompt) is converted into n tokens (h1, h2, ...hn) through a tokenizer. Based on the tokens, each layer from the 1st to the sth layer is processed using attention layers and FNN layers, and a probability Pk (k: 1, 2, ...f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and Pj is output, along with hj output by the tokenizer.

[0082] At this point, the internal calculation values ​​X_α: x1_α, x2_α, ..., xn_α of the r-th layer are extracted. Here, in this embodiment, xk_α is the internal calculation value corresponding to the k-th token in the input u.

[0083] Note that X_α can be a single value (x1_α) or multiple values ​​(x1_α, x2_α, ..., xn_α). Furthermore, it is best to decide empirically which internal calculation values ​​to choose for the target r-th layer, but you may follow other guidelines if available. It has been pointed out that as the layers get deeper (further away from the input side), the processing shifts from global to detailed, so you may want to refer to such insights. In addition, the internal calculation values ​​for the attention layer's self-attention vector, the magnitude of that vector, and the output of each layer all have variables that clearly correspond to the k-th token, so it is also a good idea to choose such variables as internal variables.

[0084] <Calculation of model β and internal state X_β (Figure 12)> This process calculates the internal value X_β of model β. Specifically, this is shown in Figure 12. In other words, in this embodiment, model β is also a decoder-based model of the transformer. As with Figure 11, processes not related to this embodiment are omitted from illustrations and explanations.

[0085] The input u (here, a sentence / prompt) is converted into n tokens (h1, h2, ...hn) through a tokenizer. Based on the tokens, each layer from the 1st to the sth layer is processed using attention layers and FNN layers, and a probability Pk (k: 1, 2, ...f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and Pj is output, along with hj output by the tokenizer.

[0086] At this point, the internal calculation values ​​X_β: x1_β, x2_β, ..., xn_β of the r-th layer are extracted. Here, in this embodiment, xk_β is the internal calculation value corresponding to the k-th token in the input u.

[0087] Note that X_β can be a single value (x1_β) or multiple values ​​(x1_β, x2_β, ..., xn_β). Also, the target r-th layer and which internal calculation value to choose for the r-th layer should be determined empirically, but it should be the same variable as X_α shown in Figure 11.

[0088] <Difference d calculation means (Figure 6)> This process calculates the difference d between X_α and X_β. Specifically, as shown in Figure 6, it is the same as in Example 1 and will not be described in detail.

[0089] <Mechanism for calculating the representative value z_in of the region (Figure 7)> In this process, the region representative value z_in is calculated based on the difference d and the internal state X_β. Specifically, this is shown in Figure 7, but since it is the same as in Example 1, it will not be described in detail.

[0090] <Calculation of model β and internal state X_βv (Figure 13)> This process calculates the internal value X_βv of model β. Specifically, this is shown in Figure 13. That is, as described above, in this embodiment, model β is a decoder-based model of a transformer. Note that, as with Figures 11 and 12, processes not related to this embodiment are omitted from illustrations and explanations.

[0091] The input v (here, a sentence / prompt) is converted into n tokens (h1, h2, ...hn) through a tokenizer. Based on the tokens, each layer from the 1st to the sth layer is processed by attention layers and FNN layers, and a probability Pk (k: 1, 2, ...f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and Pj is output, along with hj output by the tokenizer.

[0092] At this point, the internal calculation values ​​X_βv: x1_βv, x2_βv, ..., xn_βv of the r-th layer are extracted. Here, in this embodiment, xk_βv is the internal calculation value corresponding to the k-th token in the input v.

[0093] Note that X_βv can be a single value (x1_βv) or multiple values ​​(x1_βv, x2_βv, ..., xn_βv). Also, the target r-th layer and which internal calculation value to choose for the r-th layer should be determined empirically, but they should be the same variables as X_α and X_β shown in Figures 11 and 12.

[0094] <Difference D_v calculation means (Figure 9)> In this process, the difference D_v is calculated based on the internal state X_βv and the region representative value z_in. Specifically, this is shown in Figure 9, but since it is the same as in Example 1, it will not be described in detail.

[0095] <Reliability λ_v calculation method (Figure 10)> In this process, the confidence level λ_v is calculated based on the difference D_v. Specifically, this is shown in Figure 10, but since it is the same as in Example 1, it will not be described in detail.

[0096] Furthermore, it is desirable that inputs u and v be data from a specific field used to train model β, or data belonging to that specific field.

[0097] Furthermore, while inputs u and v are presented as sentences / prompts, they may consist of multiple sentences, and the same processing can be performed regardless. Also, in this case, the region representative value z_in may be calculated for each sentence or for all sentences at once. The choice should depend on the desired performance.

[0098] In this configuration, model α is the base model, and model β is a model that has been further trained using data from a predetermined domain. Both model α and model β are transformer decoder-based models. When the data from the predetermined domain used to train model β, or data belonging to that predetermined domain, is taken as input u, and the internal calculation value X_α of model α for that input u differs from the internal calculation value X_β of model β for that input u, and preferably the output of model β is valid, then it is assumed that model β has acquired some additional knowledge from model α in the predetermined domain to which the information contained in input u belongs, and the knowledge domain is defined based on the internal calculation value X_β.

[0099] Then, a new input v, which is data from a predetermined field used to train model β or data belonging to that predetermined field, is input to model β. If the internal calculation value X_βv at that time is considerably far from the knowledge domain defined above, then it is determined that the input v is seeking information that is not in the knowledge domain acquired by model β, and the reliability of the output of model β at that time is not necessarily high.

[0100] In this embodiment, the internal calculation values ​​xk_α, xk_β, xk_βv are the token numbers in input u or input v, and the internal calculation values ​​correspond to the values ​​of each token in input u or input v, so the correspondence between tokens (words) in the input and the internal variables is explicitly given.

[0101] Furthermore, in this embodiment, reliability was determined using the model's internal calculation values ​​at the time of input. It has been pointed out that LLM performs a kind of in-context learning based on input information, and as in this embodiment, the model's internal calculation values ​​at the time of input are thought to contain information from when in-context learning was being performed, making this a significant method in this case as well.

[0102] Thus, with the configuration shown in this embodiment, when evaluating the reliability of the output for an input v, only model β needs to be run. Furthermore, since the reliability is evaluated based on information about the internal state of the model, the amount of information (number of parameters, dimensions) is far greater than when using the model output, and proportionally higher accuracy is expected. In addition, since models α and β use transformer decoders, which are typical structures of LLMs, they have a wide range of applicability. Moreover, since the correspondence between tokens (words) in the input and internal variables is explicitly given, the processing is highly explainable.

[0103] Based on the above, this configuration makes it possible to evaluate the reliability of the model's output with broad applicability, high explainability, low computational load, and high accuracy. [Examples]

[0104] In this embodiment, A means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α given the input u, A means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the aforementioned input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, When the difference d is not within the predetermined range / is greater than or equal to the predetermined value, based on the internal calculation value X_β, means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, It is equipped with.

[0105] Also, A means for calculating at least one internal calculation value X_βv of the model / AI / artificial intelligence β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, A means for calculating the confidence / certainty λ_v of the output of the model β when the input v is given, based on the difference D_v. It is equipped with.

[0106] Also, A method / apparatus for calculating the reliability / confidence / certainty of the output / answer of a model / artificial intelligence / AI / generative AI / large-scale language model / LLM / transformer(α) using a computer having at least a processor and memory, The aforementioned processor, The parameter values ​​were determined (trained) using training data A. When input u (word / sentence) is input to (model) α (output y_α (word / sentence) is obtained), the values ​​X_α: x1_α, x2_α, ..., xn_α of the internal parameters X: x1, x2, ..., xn_α of the model α are calculated. The parameter values ​​of (model) α were updated (additionally trained) using training data B, and the input u (word / sentence) was input to (model) β (output y_β (word / sentence) was obtained). The values ​​of the internal parameters X:x1,x2,...,xn of model β, X_β:x1_β,x2_β,...,xn_β, were calculated. Based on the above X_α: x1_α, x2_α, ..., xn_α, at least one (representative value) z_α1 is calculated, Based on the above X_β: x1_β, x2_β, ..., xn_β, at least one (representative value) z_β is calculated, Based on the aforementioned z_α and z_β, the difference d between z_α and z_β is calculated. When the difference d is greater than or equal to a predetermined value, the z_β (or a value calculated based on the z_β) is set to z_in. When the input v (word / sentence) is input to (model) β (output y_β (word / sentence) is obtained), the values ​​of the internal parameters X:x1,x2,...,xn of the model β, X_βv:x1_βv,x2_βv,...,xn_βv, are calculated. Based on the above X_βv: x1_βv, x2_βv, ..., xn_βv and the above z_in, the difference D_v: D1_v, D2_v, ..., Dn_v is calculated, Based on the differences D_v: D1_v, D2_v, ..., Dn_v, the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) into the model β is calculated. Execute the process.

[0107] Also, The aforementioned internal calculation values ​​X_α and X_β are the internal calculation values ​​obtained when multiple u1, u2, ..., um are inputs, and specifically, When uk is input, the internal calculation value X_α is [x1_αk, x2_αk, ..., xn_αk], When uk is input, the internal calculation value X_β is [x1_βk, x2_βk, ..., xn_βk]. It is characterized by the following:

[0108] Also, The means for calculating at least one (representative value) z_α or (representative value) z_β is: This is the center vector (mean vector) of x1_α, x2_α, ..., xn_α. It is characterized by the following:

[0109] Also, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. It is characterized by the following:

[0110] Also, The input u1 (word / sentence) is data from the same "field" as training data B. The output y_β (word / sentence) satisfies the predetermined conditions (is correct), Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the aforementioned z_β (or a value calculated based on z_β). It is characterized by the following:

[0111] especially, The k of the internal parameter values ​​xk_α, xk_β is, This is the word number or token number in the aforementioned input u or input v.

[0112] Also, Models α and β are either LLMs or generative AIs.

[0113] Also, The aforementioned model β calculates the probability Pj for multiple output candidates, The second confidence / certainty value C_v in the output of model β is obtained by multiplying the confidence / certainty value λ_v of the output of model β by the probability Pj.

[0114] The above configuration is shown below.

[0115] Figure 14 is a block diagram showing the basic configuration of the confidence score C_v calculation in this embodiment. Based on the confidence score λ_v and the probability Pj, the confidence score C_v is calculated by the confidence score C_v calculation means 9. Other processes are the same as in Figure 2 and will not be described in detail.

[0116] Figure 3 is a system diagram of the device that implements the process shown in Figure 14, but it is the same as in Example 1, so it will not be described in detail.

[0117] The details of each process are explained below.

[0118] <Calculation of Model α and Internal State X_α (Figure 11)> This process calculates the internal value X_α of model α. Specifically, as shown in Figure 11, it is the same as in Example 2 and will not be described in detail.

[0119] <Calculation of model β and internal state X_β (Figure 12)> This process calculates the internal value X_β of model β. Specifically, this is shown in Figure 12, but since it is the same as in Example 2, it will not be described in detail.

[0120] <Difference d calculation means (Figure 6)> This process calculates the difference d between X_α and X_β. Specifically, as shown in Figure 6, it is the same as in Example 1 and will not be described in detail.

[0121] <Mechanism for calculating the representative value z_in of the region (Figure 7)> In this process, the region representative value z_in is calculated based on the difference d and the internal state X_β. Specifically, this is shown in Figure 7, but since it is the same as in Example 1, it will not be described in detail.

[0122] <Calculation of model β and internal state X_βv (Figure 13)> This process calculates the internal value X_βv of model β. Specifically, this is shown in Figure 13, but since it is the same as in Example 2, it will not be described in detail.

[0123] <Difference D_v calculation means (Figure 9)> In this process, the difference D_v is calculated based on the internal state X_βv and the region representative value z_in. Specifically, this is shown in Figure 9, but since it is the same as in Example 1, it will not be described in detail.

[0124] <Reliability λ_v calculation method (Figure 10)> In this process, the confidence score λ_v is calculated based on the difference D_v. Specifically, this is shown in Figure 10, but it is the same as in Example 1, so it will not be described in detail. It is desirable that λ_v be a value that can take on a value between 0 and 1.

[0125] <Reliability C_v calculation method (Figure 15)> In this process, the confidence level C_v is calculated based on the confidence level λ_v and the probability Pj. Specifically, the following calculations are performed as shown in Figure 15. - C_v=Pj×λ_v···(15-1)

[0126] In this configuration, model α is the base model, and model β is a model that has been further trained using data from a predetermined domain. Both model α and model β are transformer decoder-based models. When input u is the data from the predetermined domain used to train model β, or data belonging to that predetermined domain, and the internal calculation value X_α of model α for that input u is different from the internal calculation value X_β of model β for that input u, and preferably the output of model β is valid, then it is assumed that model β has acquired some additional knowledge from model α in the predetermined domain to which the information contained in input u belongs, and the knowledge domain is defined based on the internal calculation value X_β.

[0127] Then, a new input v, which is data from a predetermined field used to train model β or data belonging to that predetermined field, is input to model β. If the internal calculation value X_βv at that time is considerably far from the knowledge domain defined above, then it is determined that the input v is seeking information that is not in the knowledge domain acquired by model β, and the reliability of the output of model β at that time is not necessarily high.

[0128] In this embodiment, the k in the internal calculation values ​​xk_α, xk_β, xk_βv is the token number in input u or input v, and since the internal calculation values ​​correspond to the tokens in input u and v, the correspondence between the tokens (words) in the input and the internal variables is explicitly given.

[0129] Furthermore, the probability Pj is multiplied by the confidence level λ_v to calculate a new confidence level C_v. The probability Pj represents the likelihood of the output token hj, and can be interpreted as a confidence level for the output token hj. By multiplying this by the confidence level λ_v, a more accurate confidence level C_v is obtained.

[0130] Thus, with the configuration shown in this embodiment, when evaluating the reliability of the output for an input v, only model β needs to be run. Furthermore, since the reliability is evaluated based on information about the internal state of the model, the amount of information (number of parameters, dimensions) is far greater than when using the model output, and proportional accuracy is expected to be higher. In addition, since models α and β use transformer decoders, which are typical structures of LLMs, they have a wide range of applicability. Moreover, since the correspondence between tokens (words) in the input and internal variables is explicitly given, the process is highly explainable. Furthermore, by calculating a reliability C_v that considers both the reliability λ_v and the probability Pj, it is possible to calculate a more accurate reliability.

[0131] Based on the above, this configuration makes it possible to evaluate the reliability of the model's output with broad applicability, high explainability, low computational load, and even higher accuracy. [Examples]

[0132] In this embodiment, A means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α given the input u, A means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the aforementioned input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, When the difference d is not within the predetermined range / is greater than or equal to the predetermined value, based on the internal calculation value X_β, means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, It is equipped with.

[0133] Also, A means for calculating at least one internal calculation value X_βv of the model / AI / artificial intelligence β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, A means for calculating the confidence / certainty λ_v of the output of the model β when the input v is given, based on the difference D_v. It is equipped with.

[0134] Also, A method / apparatus for calculating the reliability / confidence / certainty of the output / answer of a model / artificial intelligence / AI / generative AI / large-scale language model / LLM / transformer(α) using a computer having at least a processor and memory, The aforementioned processor, The parameter values ​​were determined (trained) using training data A. When input u (word / sentence) is input to (model) α (output y_α (word / sentence) is obtained), the values ​​X_α: x1_α, x2_α, ..., xn_α of the internal parameters X: x1, x2, ..., xn_α of the model α are calculated. The parameter values ​​of (model) α were updated (additionally trained) using training data B, and the input u (word / sentence) was input to (model) β (output y_β (word / sentence) was obtained). The values ​​of the internal parameters X:x1,x2,...,xn of model β, X_β:x1_β,x2_β,...,xn_β, were calculated. Based on the above X_α: x1_α, x2_α, ..., xn_α, at least one (representative value) z_α1 is calculated, Based on the above X_β: x1_β, x2_β, ..., xn_β, at least one (representative value) z_β is calculated, Based on the aforementioned z_α and z_β, the difference d between z_α and z_β is calculated. When the difference d is greater than or equal to a predetermined value, the z_β (or a value calculated based on the z_β) is set to z_in. When the input v (word / sentence) is input to (model) β (output y_β (word / sentence) is obtained), the values ​​of the internal parameters X:x1,x2,...,xn of the model β, X_βv:x1_βv,x2_βv,...,xn_βv, are calculated. Based on the above X_βv: x1_βv, x2_βv, ..., xn_βv and the above z_in, the difference D_v: D1_v, D2_v, ..., Dn_v is calculated, Based on the differences D_v: D1_v, D2_v, ..., Dn_v, the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) into the model β is calculated. Execute the process.

[0135] Also, The aforementioned internal calculation values ​​X_α and X_β are the internal calculation values ​​obtained when multiple u1, u2, ..., um are inputs, and specifically, When uk is input, the internal calculation value X_α is [x1_αk, x2_αk, ..., xn_αk], When uk is input, the internal calculation value X_β is [x1_βk, x2_βk, ..., xn_βk]. It is characterized by the following:

[0136] Also, The means for calculating at least one (representative value) z_α or (representative value) z_β is: This is the center vector (mean vector) of x1_α, x2_α, ..., xn_α. It is characterized by the following:

[0137] Also, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. It is characterized by the following:

[0138] Also, The input u1 (word / sentence) is data from the same "field" as training data B. The output y_β (word / sentence) satisfies the predetermined conditions (is correct), Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the aforementioned z_β (or a value calculated based on z_β). It is characterized by the following:

[0139] especially, The k of the internal parameter values ​​xk_α, xk_β is This is the word number or token number in the aforementioned input u or input v.

[0140] Also, Models α and β are either LLMs or generative AIs.

[0141] Also, The aforementioned model β calculates the probability Pj for multiple output candidates, The confidence / certainty λ_v of the output of the aforementioned model β is, Let the value obtained by multiplying the aforementioned probability Pj be the second confidence / certainty level C_v in the output of the model β.

[0142] Also, The means for calculating the value X_α of the intrinsic parameter X of the model α is: The internal parameters X1_α, x2_α, ..., xn_α when the input u (word / sentence) is input to the (model) α are, When the input u (word / sentence) is input to the model α and the output y_α (word / sentence) of the model α is calculated, X_α consists of both internal parameters Xn+1_α, xn+2_α, ..., xn+m_α. Calculate, The means for calculating the value X_β of the intrinsic parameter X of the model β is: The internal parameters X1_β, x2_β, ..., xn_β when the input u (word / sentence) is input to the (model) β are: When the input u1 (word / sentence) is input to the model β, and the output y_β (word / sentence) of the model β is calculated, X_β consists of both internal parameters Xn+1_β, xn+2_β, ..., xn+m_β. Calculate, (Means for calculating at least one (representative value) z_α based on the above X_α, A means for calculating at least one (representative value) z_β based on the aforementioned X_β, A means for calculating the difference d between z_α and z_β based on the aforementioned z_α and z_β, (When the difference d is greater than or equal to a predetermined value, the means of setting z_β (or a value calculated based on z_β) to z_in) When input v (word / sentence) is input to the aforementioned (model) β, the internal parameters X1_βv, x2_βv, ..., xn_βv are, When the input v1 (word / sentence) is input to the model β, and the output y_βv (word / sentence) of the model β is calculated, X_βv consists of both internal parameters Xn+1_βv, xn+2_βv, ..., xn+m_βv. Calculate, A means for calculating the difference D_v: D1_v, D2_v, Dn_v, Dn+1_v, Dn+2_v, Dn+m_v based on the aforementioned X_βv: x1_βv, x2_βv, ..., xn_βv, xn+1_βv, xn+2_βv, ..., xn+m_βv and the aforementioned z_in, A means for calculating the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) to the model β based on the difference D_v, and It is equipped with.

[0143] The above configuration is shown below.

[0144] Figure 1 is a block diagram showing the basic configuration of the region representative value z_in calculation in this embodiment, but it is the same as in Embodiment 1, so it will not be described in detail.

[0145] Figure 2 is a block diagram showing the basic configuration of the confidence λ_v calculation in this embodiment, and since it is the same as in Embodiment 1, it will not be described in detail.

[0146] Figure 3 is a system diagram of the device that implements the process shown in Figure 1 or Figure 2, but it is the same as in Example 1, so it will not be described in detail.

[0147] The details of each process are explained below.

[0148] <Calculation of Model α and Internal State X_α (Figure 16)> This process calculates the internal value X_α of model α. Specifically, this is shown in Figure 16. In this embodiment, model α is a decoder-based model of the transformer. Note that the explanation of this figure is given in Figure 11, so any overlapping points will be omitted. In Figure 16, the output regression process, which was omitted in Figure 11, is clearly shown. In the transformer decoder, the output is regressiond and used to infer the next output. There is a lot of literature on this regression process, so it will not be described in detail here. Also, parts such as the embedding process and direct terms (residual paths) are not relevant to this embodiment, so their illustrations and explanations are omitted, as in Figure 11.

[0149] The input u (here, a sentence / prompt) is converted into n tokens (h1, h2, ...hn) through a tokenizer. Furthermore, the output regression is also converted into m tokens (hn+1, hn+2, ...hn+m) through a tokenizer (or the output tokens are directly regressed).

[0150] Based on the tokens, processing is performed in the attention layers and FNN layers of each layer from the 1st to the sth layer, and a probability Pk(k:1,2,····f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and Pj is output, along with hj, which is output by the tokenizer.

[0151] At this point, the internal calculation values ​​X_α: x1_α, x2_α, ..., xn_α, xn+1_α, ..., xn+m_α of the r-th layer are extracted. Here, in this embodiment, xk_α is the internal calculation value corresponding to the k-th token in the input u.

[0152] Note that X_α can be a single value (x1_α) or multiple values ​​(x1_α, x2_α, ..., xn+m_α). Furthermore, it is best to decide empirically which internal calculation values ​​to choose for the target r-th layer, but you may follow other guidelines if available. It has been pointed out that as the layers get deeper (as you move further away from the input), the processing shifts from global to detailed, so you may want to refer to such insights. In addition, for the internal calculation values, the self-attention vector of the attention layer, the magnitude of that vector, and the output of each layer all have variables that clearly correspond to the k-th token, so it is also good to choose such variables as internal variables.

[0153] <Calculation of model β and internal state X_β (Figure 17)> This process calculates the internal value X_β of model β. Specifically, this is shown in Figure 17. In other words, in this embodiment, model β is also a decoder-based model of the transformer. As with Figure 16, processes not related to this embodiment are omitted from illustrations and explanations.

[0154] The input u (here, a sentence / prompt) is converted into n tokens (h1, h2, ...hn) through a tokenizer. Furthermore, the output regression is also converted into m tokens (hn+1, hn+2, ...hn+m) through a tokenizer (or the output tokens are directly regressed).

[0155] Based on the tokens, processing is performed in the attention layers and FNN layers of each layer from the 1st to the sth layer, and a probability Pk(k:1,2,····f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and Pj is output, along with hj, which is output by the tokenizer.

[0156] At this point, the internal calculation values ​​X_β: x1_β, x2_β, ..., xn_β, xn+1_β, ..., xn+m_β of the r-th layer are extracted. Here, in this embodiment, xk_β is the internal calculation value corresponding to the k-th token in the input u.

[0157] Note that X_β may be a single value (x1_β) or multiple values (x1_β, x2_β, ···, xn+m_β). Also, it is advisable to empirically determine which internal operation value of the target r-th layer and the r-th layer to select, and use the same variable as X_β shown in FIG. 16.

[0158] <Difference d calculation means (FIG. 6)> In this process, the difference d between X_α and X_β is calculated. Specifically, as shown in FIG. 6, since it is the same as in Example 1, it will not be described in detail.

[0159] <Region representative value z_in calculation means (FIG. 7)> In this process, the region representative value z_in is calculated based on the difference d and the internal state X_β. Specifically, as shown in FIG. 7, since it is the same as in Example 1, it will not be described in detail.

[0160] <Calculation of model β and internal state X_βv (FIG. 18)> In this process, the internal operation value X_βv of model β is calculated. Specifically, it is shown in FIG. 18. That is, as described above, in this embodiment, model β is a decoder-based model of a transformer. Note that, similar to FIGS. 16 and 17, processes not related to this embodiment are omitted from illustration and description.

[0161] The input v (here it is a sentence / prompt) is converted into n tokens (h1, h2, ···, hn) through a tokenizer. Further, the output regression part is also converted into m tokens (hn+1, hn+2, ···, hn+m) through the tokenizer (or directly regress the tokens on the output side).

[0162] Based on the tokens, processing is performed in the attention layer and FNN layer of each layer from the first layer to the s-th layer, and the probability Pk (k: 1, 2, ···, f) is calculated based on the output of the s-th layer. Here, f is the number of token candidates. The token number j to be output from P1 to Pf is determined, Pj is output, and hj is output by the tokenizer.

[0163] The internal calculation values ​​X_βv: x1_βv, x2_βv, ..., xn_βv, xn+1_βv, ..., xn+m_βv of the r-th layer are extracted. Here, in this embodiment, xk_βv is the internal calculation value corresponding to the k-th token in the input u. X_βv may be a single value (x1_βv) or multiple values ​​(x1_βv, x2_βv, ..., xn+m_βv). The target r-th layer and which internal calculation values ​​of the r-th layer to choose should be determined empirically, but they should be the same variables as X_α and X_β shown in Figures 16 and 17.

[0164] <Difference D_v calculation means (Figure 9)> In this process, the difference D_v is calculated based on the internal state X_βv and the region representative value z_in. Specifically, this is shown in Figure 9, but since it is the same as in Example 1, it will not be described in detail.

[0165] <Reliability λ_v calculation method (Figure 10)> In this process, the confidence level λ_v is calculated based on the difference D_v. Specifically, this is shown in Figure 10, but since it is the same as in Example 1, it will not be described in detail.

[0166] Furthermore, it is desirable that inputs u and v be data from a specific field used to train model β, or data belonging to that specific field.

[0167] Furthermore, while inputs u and v are presented as sentences / prompts, they may consist of multiple sentences, and the same processing can be performed regardless. Also, in this case, the region representative value z_in may be calculated for each sentence or for all sentences at once. The choice should depend on the desired performance.

[0168] In this configuration, model α is the base model, and model β is a model that has been further trained using data from a predetermined domain. Both model α and model β are transformer decoder-based models. When the data from the predetermined domain used to train model β, or data belonging to that predetermined domain, is taken as input u, and the internal calculation value X_α of model α when input u is input and when the output of model α is calculated after input u is input, and the internal calculation value X_β of model β when input u is input and when the output of model β is calculated after input u is input, are different, and preferably the output of model β is valid, then model β is considered to have acquired some additional knowledge from model α in the predetermined domain to which the information contained in input u and the output at that time belongs, and a knowledge domain is defined based on the internal calculation value X_β.

[0169] Furthermore, when a new input v, which is data from a predetermined field used to train model β or data belonging to that predetermined field, is input to model β, and when the internal calculation value X_βv is calculated after input v is input and the output of model β is calculated, if the input v is considerably far removed from the knowledge domain defined above, then it is determined that the input v is seeking information that is not in the knowledge domain acquired by model β, and the reliability of the output of model β at that time is not necessarily high.

[0170] In this embodiment, the k in the internal calculation values ​​xk_α, xk_β, xk_βv is the token number in the input u and its output, or in the input v and its output. Since the internal calculation values ​​correspond to each token in the input u and its output, or in the input v and its output, the correspondence between the tokens (words) in the input and its output and the internal variables is explicitly given.

[0171] Thus, with the configuration shown in this embodiment, when evaluating the reliability of the output for an input v, only model β needs to be operated. Furthermore, since the reliability is evaluated based on information about the internal state of the model, the amount of information (number of parameters, dimensions) is far greater than when using the model output, and proportionally higher accuracy is expected. In addition, since models α and β use transformer decoders, which are typical structures of LLMs, they have a wide range of applicability. Moreover, since not only the internal variables corresponding to the input but also the internal variables used in the output calculation at that time are used, reliability is further enhanced. Furthermore, since the correspondence between tokens (words) in the input and the output at that time and the internal variables is explicitly given, the process is highly explainable.

[0172] Based on the above, this configuration makes it possible to evaluate the reliability of the model's output with broad applicability, high explainability, low computational load, and high accuracy. [Examples]

[0173] In this embodiment, A means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α given the input u, A means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the aforementioned input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, When the difference d is not within the predetermined range / is greater than or equal to the predetermined value, based on the internal calculation value X_β, means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, It is equipped with.

[0174] Also, A means for calculating at least one internal calculation value X_βv of the model / AI / artificial intelligence β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, means for calculating the reliability / accuracy λ_v of the output of the model β at the time of the input v based on the difference D_v is provided with

[0175] Also In a method / apparatus for calculating the reliability / confidence / accuracy of the output / answer of a model / artificial intelligence / AI / generative AI / large language model / LLM / Transformer (α) by a computer including at least a processor and a memory the processor calculates the values X_α:x1_α,x2_α,···,xn_α of the internal parameters X:x1,x2,···,xn of the model α when an input u (word / sentence) is input to the (model) α which has determined (learned) parameter values using learning data A (and an output y_α (word / sentence) is obtained), calculates the values X_β:x1_β,x2_β,···,xn_β of the internal parameters X:x1,x2,···,xn of the (model) β when the (model) β updates (additionally learns) the parameter values of the (model) α using learning data B and the input u (word / sentence) is input to the (model) β (and an output y_β (word / sentence) is obtained), calculates at least one (representative value) z_α1 based on the X_α:x1_α,x2_α,···,xn_α, calculates at least one (representative value) z_β based on the X_β:x1_β,x2_β,···,xn_β, calculates the difference d between the z_α and the z_β based on the z_α and the z_β, when the difference d is greater than or equal to a predetermined value, sets the z_β (or a value calculated based on the z_β) as z_in, calculates the values X_βv:x1_βv,x2_βv,···,xn_βv of the internal parameters X:x1,x2,···,xn of the (model) β when the input v (word / sentence) is input to the (model) β (and an output y_β (word / sentence) is obtained), calculates differences D_v:D1_v,D2_v,···,Dn_v based on each of the X_βv:x1_βv,x2_βv,···,xn_βv and the z_in Based on the differences D_v: D1_v, D2_v, ..., Dn_v, the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) into the model β is calculated. Execute the process.

[0176] Also, The aforementioned internal calculation values ​​X_α and X_β are the internal calculation values ​​obtained when multiple u1, u2, ..., um are inputs, and specifically, When uk is input, the internal calculation value X_α is [x1_αk, x2_αk, ..., xn_αk], When uk is input, the internal calculation value X_β is [x1_βk, x2_βk, ..., xn_βk]. It is characterized by the following:

[0177] Also, The means for calculating at least one (representative value) z_α or (representative value) z_β is: This is the center vector (mean vector) of x1_α, x2_α, ..., xn_α. It is characterized by the following:

[0178] Also, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. It is characterized by the following:

[0179] Also, The input u1 (word / sentence) is data from the same "field" as training data B. The output y_β (word / sentence) satisfies the predetermined conditions (is correct), Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the aforementioned z_β (or a value calculated based on z_β). It is characterized by the following:

[0180] especially, The k of the internal parameter values ​​xk_α, xk_β is, This is the word number or token number in the aforementioned input u or input v.

[0181] Also, Models α and β are either LLMs or generative AIs.

[0182] Also, Models α and β have multiple layers and multiple types of internal parameters. The above processing is performed on the multiple types of internal parameters in the multiple layers.

[0183] Also, Model α and Model β are transformers, The aforementioned internal calculation value is at least the attention vector, the magnitude of the attention vector, the parameter value of the FFN, the input / output embedding value, or the output value of each layer. i represents the layer number, j represents the type of internal calculation value, The internal values ​​for the aforementioned pair of inputs uk are xk_α(i,j) and xk_β(i,j), (Representative value) z_α(i,j) based on the above xk_α(i,j) and (Representative value) z_β(i,j) based on the above xk_β(i,j) And, Let the difference d(i,j) be based on the above z_α(i,j) and z_β(i,j). When the difference d(i,j) is greater than or equal to a predetermined value, the z_β(i,j) (or the value calculated based on z_β(i,j)) is set to z_in(i,j), When the input v (word / sentence) is input to (model) β (output y_β (word / sentence) is obtained), the internal calculation value xk of the model β is x1_βv(i,j), x2_βv(i,j), ..., xn_βv(i,j), Let D1_v(i,j), D2_v(i,j), ..., Dn_v(i,j) be the differences between the above x1_βv(i,j), x2_βv(i,j), ..., xn_βv(i,j) and the above z_in(i,j), Based on the differences D1_v(i,j), D2_v(i,j), ..., Dn_v(i,j), the confidence / accuracy of the output of model β is given as λ_v or λ_v(i,j). It is characterized by the following:

[0184] The above configuration is shown below.

[0185] Figure 1 is a block diagram showing the basic configuration of the region representative value z_in calculation in this embodiment, but it is the same as in Embodiment 1, so it will not be described in detail.

[0186] Figure 2 is a block diagram showing the basic configuration of the confidence λ_v calculation in this embodiment, and since it is the same as in Embodiment 1, it will not be described in detail.

[0187] Figure 3 is a system diagram of the device that implements the process shown in Figure 1 or Figure 2, but it is the same as in Example 1, so it will not be described in detail.

[0188] The details of each process are explained below.

[0189] <Calculation of Model α and Internal State X_α (Figure 19)> This process calculates the internal value X_α of model α. Specifically, this is shown in Figure 19. In this embodiment, model α is a decoder-based model of a transformer. Note that there is a lot of literature on transformers and their decoders, so they will not be described in detail here. Also, the embedding process, direct terms (residual paths), and regression parts are not relevant to this embodiment, so they are not shown or explained.

[0190] The input u (here, a sentence / prompt) is converted into n tokens (h1, h2, ...hn) through a tokenizer. Based on the tokens, each layer from the 1st to the sth layer is processed using attention layers and FNN layers, and a probability Pk (k: 1, 2, ...f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and Pj is output, along with hj output by the tokenizer.

[0191] In this case, we extract the internal calculation values ​​X_α: x1_α(i,j), x2_α(i,j), ..., xn_α(i,j) from multiple layers, not just the r-th layer. Figure 19 shows the case where the internal calculation values ​​of the 1st layer (i=1), the r-th layer (i=r), and the s-th layer (i=s) are calculated. Also, j represents the type of internal parameter, and here, when j=1, it means the attention vector.

[0192] Furthermore, xk_α(i,j) is the internal calculation value corresponding to the k-th token in the input u. X_α can be a single value (x1_α(i,j)) or multiple values ​​(x1_α(i,j), x2_α(i,j), ..., xn_α(i,j)) in each layer.

[0193] It is best to determine empirically which layers to use as internal calculation values ​​and which internal calculation values ​​to choose from within those layers, but if there are other guidelines, you may follow them. It has been pointed out that as the layers get deeper (as you move further away from the input side), the processing shifts from global to detailed, so you may also refer to such insights. In this embodiment, the type of internal calculation value is the self-attention vector of the attention layer, but other variables that clearly correspond to the k-th token exist for the magnitude of the vector and the output of each layer, so you may also choose such variables as internal variables.

[0194] <Calculation of model β and internal state X_β (Figure 20)> This process calculates the internal value X_β of model β. Specifically, this is shown in Figure 20. In other words, in this embodiment, model β is also a decoder-based model of the transformer. As with Figure 19, processes not related to this embodiment are omitted from illustrations and explanations.

[0195] The input u (here, a sentence / prompt) is converted into n tokens (h1, h2, ...hn) through a tokenizer. Based on the tokens, each layer from the 1st to the sth layer is processed using attention layers and FNN layers, and a probability Pk (k: 1, 2, ...f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and Pj is output, along with hj output by the tokenizer.

[0196] In this case, the internal calculation values ​​X_β: x1_β(i,j), x2_β(i,j), ..., xn_β(i,j) of multiple layers are extracted, not just the r-th layer. Figure 20 shows the case where the internal calculation values ​​of the 1st layer (i=1), the r-th layer (i=r), and the s-th layer (i=s) are calculated. Also, j represents the type of internal parameter, and here, when j=1, it means an attention vector.

[0197] Furthermore, xk_β(i,j) is the internal calculation value corresponding to the k-th token in the input u. X_β may be a single value (x1_β(i,j)) or multiple values ​​(x1_β(i,j), x2_β(i,j), ..., xn_β(i,j)) in each layer.

[0198] The layer to be used as an internal calculation value and which internal calculation value to choose from that layer should be determined empirically, but it should be the same variable as X_α shown in Figure 19.

[0199] <Difference d calculation means (Figure 6)> This process calculates the difference d between X_α and X_β. Specifically, as shown in Figure 6, it is the same as in Example 1 and will not be described in detail.

[0200] <Mechanism for calculating the representative value z_in of the region (Figure 7)> In this process, the region representative value z_in is calculated based on the difference d and the internal state X_β. Specifically, this is shown in Figure 7, but since it is the same as in Example 1, it will not be described in detail.

[0201] <Calculation of model β and internal state X_βv (Figure 21)> This process calculates the internal value X_βv of model β. Specifically, this is shown in Figure 21. That is, as described above, in this embodiment, model β is a decoder-based model of a transformer. Note that, as with Figures 19 and 20, processes not related to this embodiment are omitted from illustrations and explanations.

[0202] The input v (here, a sentence / prompt) is converted into n tokens (h1, h2, ...hn) through a tokenizer. Based on the tokens, each layer from the 1st to the sth layer is processed by attention layers and FNN layers, and a probability Pk (k: 1, 2, ...f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and Pj is output, along with hj output by the tokenizer.

[0203] In this case, we extract the internal calculation values ​​X_βv: x1_βv(i,j), x2_βv(i,j), ..., xn_βv(i,j) from multiple layers, not just the r-th layer. Figure 21 shows the case where the internal calculation values ​​of the 1st layer (i=1), the r-th layer (i=r), and the s-th layer (i=s) are calculated. Also, j represents the type of internal parameter, and here, when j=1, it means the attention vector.

[0204] Furthermore, xk_βv(i,j) is the internal calculation value corresponding to the k-th token in the input v. Note that X_βv can be a single value (x1_βv(i,j)) or multiple values ​​(x1_βv(i,j), x2_βv(i,j), ..., xn_βv(i,j)).

[0205] The layers to be used as internal calculation values ​​and which internal calculation values ​​to select for those layers should be determined empirically, but they should be the same variables as X_α and X_β shown in Figures 19 and 20.

[0206] <Difference D_v calculation means (Figure 9)> In this process, the difference D_v is calculated based on the internal state X_βv and the region representative value z_in. Specifically, this is shown in Figure 9, but since it is the same as in Example 1, it will not be described in detail.

[0207] <Reliability λ_v calculation method (Figure 10)> In this process, the confidence level λ_v is calculated based on the difference D_v. Specifically, this is shown in Figure 10, but since it is the same as in Example 1, it will not be described in detail.

[0208] In this configuration, model α is the base model, and model β is a model that has been further trained using data from a predetermined domain. Both model α and model β are transformer decoder-based models. When the data from the predetermined domain used to train model β, or data belonging to that predetermined domain, is taken as input u, and the internal calculation value X_α of model α for that input u differs from the internal calculation value X_β of model β for that input u, and preferably the output of model β is valid, then it is assumed that model β has acquired some additional knowledge from model α regarding the predetermined domain to which the information contained in input u belongs, and the knowledge domain is defined based on the internal calculation value X_β. Note that the internal calculation values ​​X_α and X_β are multiple types of internal calculation values ​​in multiple layers of model α and model β.

[0209] Then, a new input v, which is data from a predetermined field used to train model β or data belonging to that predetermined field, is input to model β. If the internal calculation value X_βv at that time is considerably far from the knowledge domain defined above, then it is determined that the input v is seeking information that is not in the knowledge domain acquired by model β, and the reliability of the output of model β at that time is not necessarily high. Note that the internal calculation value X_βv is a combination of multiple types of internal calculation values ​​in multiple layers of model β.

[0210] In this embodiment, the k in the internal calculation values ​​xk_α, xk_β, xk_βv is the token number in input u or input v, and since the internal calculation values ​​correspond to each token in input u or input v, the correspondence between tokens (words) in the input and the internal variables is explicitly given.

[0211] Thus, with the configuration shown in this embodiment, when evaluating the reliability of the output for an input v, only model β needs to be operated. Furthermore, since the reliability is evaluated based on information about the internal state of the model, the amount of information (number of parameters, dimensions) is far greater than when using the model output, and proportionally higher accuracy is expected. In addition, since models α and β use transformer decoders, which are typical structures of LLMs, they have a wide range of applicability. Moreover, since the correspondence between tokens (words) in the input and internal variables is explicitly given, the processing is highly explainable. Furthermore, since the internal calculation values ​​employ not just one layer but multiple layers and multiple types, a large amount of information is obtained, and even higher reliability is guaranteed.

[0212] Based on the above, this configuration makes it possible to evaluate the reliability of the model's output with broad applicability, high explainability, low computational load, and high accuracy. [Examples]

[0213] In this embodiment, A means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α given the input u, A means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the aforementioned input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, When the difference d is not within the predetermined range / is greater than or equal to the predetermined value, based on the internal calculation value X_β, means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, It is equipped with.

[0214] Also, A means for calculating at least one internal calculation value X_βv of the model / AI / artificial intelligence β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, A means for calculating the confidence / certainty λ_v of the output of the model β when the input v is given, based on the difference D_v. It is equipped with.

[0215] Also, A method / apparatus for calculating the reliability / confidence / certainty of the output / answer of a model / artificial intelligence / AI / generative AI / large-scale language model / LLM / transformer(α) using a computer having at least a processor and memory, The aforementioned processor, The parameter values ​​were determined (trained) using training data A. When input u (word / sentence) is input to (model) α (output y_α (word / sentence) is obtained), the values ​​X_α: x1_α, x2_α, ..., xn_α of the internal parameters X: x1, x2, ..., xn_α of the model α are calculated. The parameter values ​​of (model) α were updated (additionally trained) using training data B, and the input u (word / sentence) was input to (model) β (output y_β (word / sentence) was obtained). The values ​​of the internal parameters X:x1,x2,...,xn of model β, X_β:x1_β,x2_β,...,xn_β, were calculated. Based on the above X_α: x1_α, x2_α, ..., xn_α, at least one (representative value) z_α1 is calculated, Based on the above X_β: x1_β, x2_β, ..., xn_β, at least one (representative value) z_β is calculated, Based on the aforementioned z_α and z_β, the difference d between z_α and z_β is calculated. When the difference d is greater than or equal to a predetermined value, the z_β (or a value calculated based on the z_β) is set to z_in. When the input v (word / sentence) is input to (model) β (output y_β (word / sentence) is obtained), the values ​​of the internal parameters X:x1,x2,...,xn of the model β, X_βv:x1_βv,x2_βv,...,xn_βv, are calculated. Based on the above X_βv: x1_βv, x2_βv, ..., xn_βv and the above z_in, the difference D_v: D1_v, D2_v, ..., Dn_v is calculated, Based on the differences D_v: D1_v, D2_v, ..., Dn_v, the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) into the model β is calculated. Execute the process.

[0216] Also, The aforementioned internal calculation values ​​X_α and X_β are the internal calculation values ​​obtained when multiple u1, u2, ..., um are inputs, and specifically, When uk is input, the internal calculation value X_α is [x1_αk, x2_αk, ..., xn_αk], When uk is input, the internal calculation value X_β is [x1_βk, x2_βk, ..., xn_βk]. It is characterized by the following:

[0217] Also, The means for calculating at least one (representative value) z_α or (representative value) z_β is: This is the center vector (mean vector) of x1_α, x2_α, ..., xn_α. It is characterized by the following:

[0218] Also, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. It is characterized by the following:

[0219] Also, The input u1 (word / sentence) is data from the same "field" as training data B. The output y_β (word / sentence) satisfies the predetermined conditions (is correct), Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the aforementioned z_β (or a value calculated based on z_β). It is characterized by the following:

[0220] especially, The k of the internal parameter values ​​xk_α, xk_β is, This is the word number or token number in the aforementioned input u or input v.

[0221] Also, Models α and β are either LLMs or generative AIs.

[0222] Also, The aforementioned model β calculates the probability Pj for multiple output candidates, The second confidence / certainty value C_v in the output of model β is obtained by multiplying the confidence / certainty value λ_v of the output of model β by the probability Pj.

[0223] Also, When λ_v is less than or equal to a predetermined value or C_v is less than or equal to a predetermined value, (If you are unsure about an input or answer you have never seen before) Notify the system / display information about it on the screen.

[0224] The above configuration is shown below.

[0225] Figure 22 is a block diagram showing the basic configuration of the confidence level C_v calculation in this embodiment. Based on the confidence level λ_v and the confidence level C_v, the notification flag f_alert calculation means 10 calculates the notification flag f_alert. The other processes are the same as those shown in Figure 14 in Embodiment 3, so they will not be described in detail.

[0226] Figure 3 is a system diagram of the device that implements the process shown in Figure 22, but it is the same as in Example 1, so it will not be described in detail.

[0227] The details of each process are explained below.

[0228] <Calculation of Model α and Internal State X_α (Figure 11)> This process calculates the internal value X_α of model α. Specifically, as shown in Figure 11, it is the same as in Example 2 and will not be described in detail.

[0229] <Calculation of model β and internal state X_β (Figure 12)> This process calculates the internal value X_β of model β. Specifically, this is shown in Figure 12, but since it is the same as in Example 2, it will not be described in detail.

[0230] <Difference d calculation means (Figure 6)> This process calculates the difference d between X_α and X_β. Specifically, as shown in Figure 6, it is the same as in Example 1 and will not be described in detail.

[0231] <Mechanism for calculating the representative value z_in of the region (Figure 7)> In this process, the region representative value z_in is calculated based on the difference d and the internal state X_β. Specifically, this is shown in Figure 7, but since it is the same as in Example 1, it will not be described in detail.

[0232] <Calculation of model β and internal state X_βv (Figure 13)> This process calculates the internal value X_βv of model β. Specifically, this is shown in Figure 13, but since it is the same as in Example 2, it will not be described in detail.

[0233] <Difference D_v calculation means (Figure 9)> In this process, the difference D_v is calculated based on the internal state X_βv and the region representative value z_in. Specifically, this is shown in Figure 9, but since it is the same as in Example 1, it will not be described in detail.

[0234] <Reliability λ_v calculation method (Figure 10)> In this process, the confidence score λ_v is calculated based on the difference D_v. Specifically, this is shown in Figure 10, but it is the same as in Example 1, so it will not be described in detail. It is desirable that λ_v be a value that can take on a value between 0 and 1.

[0235] <Reliability C_v calculation method (Figure 15)> In this process, the confidence level C_v is calculated based on the confidence level λ_v and the probability Pj. Specifically, this is shown in Figure 15, but since it is the same as in Example 3, it will not be described in detail.

[0236] <Notification flag f_alert calculation means (Figure 23)> In this process, the notification flag f_alert is calculated based on the confidence levels λ_v and C_v. Specifically, the following calculations are performed as shown in Figure 23. - Set f_alert=1 when λ_v≦K_λ_v or C_v≦K_C_v. - Otherwise, set f_alert=0.

[0237] As shown in Examples 1 and 2, there is also a configuration that uses only λ_v; in that case, f_alert=1 when λ_v is greater than or equal to a predetermined value.

[0238] When f_alert=1, the user should be notified in some way, such as by displaying a message on the screen. K_λ_v and K_C_v should ideally be determined empirically, but other reasonable methods, such as those based on required performance, may also be used.

[0239] In this configuration, model α is the base model, and model β is a model that has been further trained using data from a predetermined domain. Both model α and model β are transformer decoder-based models. When the data from the predetermined domain used to train model β, or data belonging to that predetermined domain, is taken as input u, and the internal calculation value X_α of model α for that input u differs from the internal calculation value X_β of model β for that input u, and preferably the output of model β is valid, then it is assumed that model β has acquired some additional knowledge from model α in the predetermined domain to which the information contained in input u belongs, and the knowledge domain is defined based on the internal calculation value X_β.

[0240] Then, a new input v, which is data from a predetermined field used to train model β or data belonging to that predetermined field, is input to model β. If the internal calculation value X_βv at that time is considerably far from the knowledge domain defined above, then it is determined that the input v is seeking information that is not in the knowledge domain acquired by model β, and the reliability of the output of model β at that time is not necessarily high.

[0241] In this embodiment, the k in the internal calculation values ​​xk_α, xk_β, xk_βv is the token number in input u or input v, and since the internal calculation values ​​correspond to the tokens in input u and input v, the correspondence between the tokens (words) in the input and the internal variables is explicitly given.

[0242] Furthermore, the probability Pj is multiplied by the confidence level λ_v to calculate a new confidence level C_v. The probability Pj represents the likelihood of the output token hj, and can be interpreted as a confidence level for the output token hj. By multiplying this by the confidence level λ_v, a more accurate confidence level C_v is obtained. Furthermore, if the confidence level λ_v is above a predetermined value or the confidence level C_v is below a predetermined value, the system will notify or display on the screen that it is unsure about an unfamiliar input or answer.

[0243] Thus, with the configuration shown in this embodiment, when evaluating the reliability of the output for an input v, only model β needs to be operated. Furthermore, since the reliability is evaluated based on information about the internal state of the model, the amount of information (number of parameters, dimensions) is far greater than when using the model output, and proportionally higher accuracy is expected. In addition, since models α and β use transformer decoders, which are typical structures of LLMs, they have a wide range of applicability. Moreover, since the correspondence between tokens (words) in the input and internal variables is explicitly given, the processing is highly explainable. Furthermore, by calculating a reliability C_v that considers both the reliability λ_v and the probability Pj, it is possible to calculate a more accurate reliability. In addition, since the system notifies when the reliability (λ_v, C_v) is below a predetermined value, the user can easily know this.

[0244] Based on the above, this configuration makes it possible to easily evaluate the reliability of the model's output with broad applicability, high explainability, low computational load, and even higher accuracy. [Examples]

[0245] In this embodiment, A means for calculating at least one internal calculation value X_α of the model / AI / artificial intelligence α given the input u, A means for calculating at least one internal calculation value X_β of the model / AI / artificial intelligence β when the aforementioned input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, When the difference d is not within the predetermined range / is greater than or equal to the predetermined value, based on the internal calculation value X_β, means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, It is equipped with.

[0246] Also, A means for calculating at least one internal calculation value X_βv of the model / AI / artificial intelligence β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, A means for calculating the confidence / certainty λ_v of the output of the model β when the input v is given, based on the difference D_v. It is equipped with.

[0247] Also, A method / apparatus for calculating the reliability / confidence / certainty of the output / answer of a model / artificial intelligence / AI / generative AI / large-scale language model / LLM / transformer(α) using a computer having at least a processor and memory, The aforementioned processor, The parameter values ​​were determined (trained) using training data A. When input u (word / sentence) is input to (model) α (output y_α (word / sentence) is obtained), the values ​​X_α: x1_α, x2_α, ..., xn_α of the internal parameters X: x1, x2, ..., xn_α of the model α are calculated. The parameter values ​​of (model) α were updated (additionally trained) using training data B, and the input u (word / sentence) was input to (model) β (output y_β (word / sentence) was obtained). The values ​​of the internal parameters X:x1,x2,...,xn of model β, X_β:x1_β,x2_β,...,xn_β, were calculated. Based on the above X_α: x1_α, x2_α, ..., xn_α, at least one (representative value) z_α1 is calculated, Based on the above X_β: x1_β, x2_β, ..., xn_β, at least one (representative value) z_β is calculated, Based on the aforementioned z_α and z_β, the difference d between z_α and z_β is calculated. When the difference d is greater than or equal to a predetermined value, the z_β (or a value calculated based on the z_β) is set to z_in. When the input v (word / sentence) is input to (model) β (output y_β (word / sentence) is obtained), the values ​​of the internal parameters X:x1,x2,...,xn of the model β, X_βv:x1_βv,x2_βv,...,xn_βv, are calculated. Based on the above X_βv: x1_βv, x2_βv, ..., xn_βv and the above z_in, the difference D_v: D1_v, D2_v, ..., Dn_v is calculated, Based on the differences D_v: D1_v, D2_v, ..., Dn_v, the confidence / certainty λ_v of the output y_βv of the model β obtained after inputting the input v (word / sentence) into the model β is calculated. Execute the process.

[0248] Also, The aforementioned internal calculation values ​​X_α and X_β are the internal calculation values ​​obtained when multiple u1, u2, ..., um are inputs, and specifically, When uk is input, the internal calculation value X_α is [x1_αk, x2_αk, ..., xn_αk], When uk is input, the internal calculation value X_β is [x1_βk, x2_βk, ..., xn_βk]. It is characterized by the following:

[0249] Also, The means for calculating at least one (representative value) z_α or (representative value) z_β is: This is the center vector (mean vector) of x1_α, x2_α, ..., xn_α. It is characterized by the following:

[0250] Also, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. It is characterized by the following:

[0251] Also, The input u1 (word / sentence) is data from the same "field" as training data B. The output y_β (word / sentence) satisfies the predetermined conditions (is correct), Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the aforementioned z_β (or a value calculated based on z_β). It is characterized by the following:

[0252] especially, The k of the internal parameter values ​​xk_α, xk_β is, This is the word number or token number in the aforementioned input u or input v.

[0253] Also, Models α and β are either LLMs or generative AIs.

[0254] Also, The aforementioned model β calculates the probability Pj for multiple output candidates, The second confidence / certainty value C_v in the output of model β is obtained by multiplying the confidence / certainty value λ_v of the output of model β by the probability Pj.

[0255] Also, The value based on the aforementioned D_v, Dj_v, λ_v, or C_v will be displayed on the screen.

[0256] Also, The word (token) v_k in the input v that corresponds to a Dk_v greater than or equal to a predetermined value among the D1_v, D2_v, ..., Dn_v is displayed.

[0257] Also, The positional relationship between z_in and D1_v, D2_v, ..., Dn_v is displayed based on the values ​​of z_in and D1_v, D2_v, ..., Dn_v.

[0258] The above configuration is shown below.

[0259] Figure 24 is a block diagram showing the basic configuration of the confidence level C_v calculation in this embodiment. Based on the confidence level λ_v, confidence level C_v, input v, and difference D_v, the display target calculation means 11 calculates the information to be displayed. Other processes are the same as those shown in Figure 14 of Embodiment 3 and will not be described in detail.

[0260] Figure 3 is a system diagram of the device that implements the process shown in Figure 24, but it is the same as in Example 1, so it will not be described in detail.

[0261] The details of each process are explained below.

[0262] <Calculation of Model α and Internal State X_α (Figure 11)> This process calculates the internal value X_α of model α. Specifically, as shown in Figure 11, it is the same as in Example 2 and will not be described in detail.

[0263] <Calculation of model β and internal state X_β (Figure 12)> This process calculates the internal value X_β of model β. Specifically, this is shown in Figure 12, but since it is the same as in Example 2, it will not be described in detail.

[0264] <Difference d calculation means (Figure 6)> This process calculates the difference d between X_α and X_β. Specifically, as shown in Figure 6, it is the same as in Example 1 and will not be described in detail.

[0265] <Mechanism for calculating the representative value z_in of the region (Figure 7)> In this process, the region representative value z_in is calculated based on the difference d and the internal state X_β. Specifically, this is shown in Figure 7, but since it is the same as in Example 1, it will not be described in detail.

[0266] <Calculation of model β and internal state X_βv (Figure 13)> This process calculates the internal value X_βv of model β. Specifically, this is shown in Figure 13, but since it is the same as in Example 2, it will not be described in detail.

[0267] <Difference D_v calculation means (Figure 9)> In this process, the difference D_v is calculated based on the internal state X_βv and the region representative value z_in. Specifically, this is shown in Figure 9, but since it is the same as in Example 1, it will not be described in detail.

[0268] <Reliability λ_v calculation method (Figure 10)> In this process, the confidence score λ_v is calculated based on the difference D_v. Specifically, this is shown in Figure 10, but it is the same as in Example 1, so it will not be described in detail. It is desirable that λ_v be a value that can take on a value between 0 and 1.

[0269] <Reliability C_v calculation method (Figure 15)> In this process, the confidence level C_v is calculated based on the confidence level λ_v and the probability Pj. Specifically, this is shown in Figure 15, but since it is the same as in Example 3, it will not be described in detail.

[0270] <Display target calculation means (Figures 25 and 26)> In this process, the information to be displayed is calculated based on confidence levels λ_v, C_v, input v, and difference D_v. This is specifically shown in Figures 25 and 26.

[0271] As shown in Figure 25, Display λ_v, or show it using a gauge or similar. Display C_v, or display it using a gauge or similar.

[0272] Also, as shown in Figure 26, Display the inverse value of D_v, or display it using a gauge or similar method. This displays the positional relationship between z_in and D1_v, D2_v, ..., Dn_v.

[0273] For example, display token k corresponding to Dk_v that exceeds a predetermined value among D1_v, D2_v, ..., Dn_v, or change the display color of token k.

[0274] Note that λ_v and C_v may be displayed as normalized values ​​between 0 and 1. Also, since a larger D_v value indicates a lower confidence level, its reciprocal value is correlated with the confidence level. Similarly, its reciprocal value may also be displayed as a normalized value.

[0275] In this configuration, model α is the base model, and model β is a model that has been further trained using data from a predetermined domain. Both model α and model β are transformer decoder-based models. When the data from the predetermined domain used to train model β, or data belonging to that predetermined domain, is taken as input u, and the internal calculation value X_α of model α for that input u differs from the internal calculation value X_β of model β for that input u, and preferably the output of model β is valid, then it is assumed that model β has acquired some additional knowledge from model α in the predetermined domain to which the information contained in input u belongs, and the knowledge domain is defined based on the internal calculation value X_β.

[0276] Then, a new input v, which is data from a predetermined field used to train model β or data belonging to that predetermined field, is input to model β. If the internal calculation value X_βv at that time is considerably far from the knowledge domain defined above, then it is determined that the input v is seeking information that is not in the knowledge domain acquired by model β, and the reliability of the output of model β at that time is not necessarily high.

[0277] In this embodiment, the k in the internal calculation values ​​xk_α, xk_β, xk_βv is the token number in input u or input v, and since the internal calculation values ​​correspond to the tokens in input u and input v, the correspondence between the tokens (words) in the input and the internal variables is explicitly given.

[0278] Furthermore, the probability Pj is multiplied by the confidence level λ_v to calculate a new confidence level C_v. The probability Pj represents the likelihood of the output token hj, and can be interpreted as a confidence level for the output token hj. By multiplying this by the confidence level λ_v, a more accurate confidence level C_v is obtained. In addition, λ_v, C_v, D_v, and Dk_v are visually displayed on the screen.

[0279] Thus, with the configuration shown in this embodiment, when evaluating the reliability of the output for an input v, only model β needs to be run. Furthermore, since the reliability is evaluated based on information about the internal state of the model, the amount of information (number of parameters, dimensions) is far greater than when using the model output, and proportionally higher accuracy is expected. In addition, since models α and β use transformer decoders, which are typical structures of LLMs, the range of application is wide. Moreover, since the correspondence between tokens (words) in the input and internal variables is explicitly given, the process is highly explainable. Furthermore, by calculating a reliability C_v that considers both the reliability λ_v and the probability Pj, it is possible to calculate a more accurate reliability. In addition, λ_v, C_v, D_v, and Dk_v are displayed visually on the screen, so the user can easily understand them.

[0280] Based on the above, this configuration makes it possible to easily evaluate the reliability of the model's output with broad applicability, high explainability, low computational load, and even higher accuracy.

[0281] In Examples 1-7, a central vector was used to define the knowledge domain acquired by model β and to determine the difference from that knowledge domain. Besides using a central vector, other methods for defining a domain and determining the difference from that domain include LOF (Local Outlier Factor) and kNN (k-Nearest Neighbors). It should be noted that these methods can also be used. Furthermore, to further emphasize the difference between X_α and X_β, it is also possible to project X_α, X_β, and X_βv onto some space and apply the above methods on that space. Examples of projection methods include kernel methods and CL (Contrastive Learning).

[0282] Furthermore, in Examples 1 to 7, it was deemed desirable that inputs u and v be data from a predetermined field used to train model β, or data belonging to that predetermined field. On the other hand, if input u is limited to data from a predetermined field used to train model β, or data belonging to that predetermined field, it is not necessary to calculate the region representative value z_in based on the difference d between the internal calculation value X_α of model α and the internal calculation value X_β of model β. In this case, the region representative value z_in can be calculated based only on X_β.

[0283] Furthermore, the region representative value z_in in Examples 1 to 7 and the above-described case may be calculated sequentially by preparing a new input u. In other words, the region representative value z_in may be added to and grow as needed.

[0284] Furthermore, in language models, model β is not limited to an additionally trained model, but can also be a base model. Also, the input u does not need to be limited to data from a specific domain used to train model β; for any input u, the domain representative value z_in can be calculated based on the internal calculation value X_β of the language model at the time the input u is input or / and at the time the output y is calculated when the input u is input.

[0285] In Examples 1-6, Model α and Model β were MLPs or transformers, but they may also be SSM (State Space Model) based models. Representative SSM-based models include Mamba or Mamba2.

[0286] Figure 27 shows Model α and Model β for Mamba, and Figure 28 shows Model α and Model β for Mamba2.

[0287] Furthermore, Figure 29 shows the internal state calculations (calculation process for the internal calculation value X) when model α and model β are Mamba, and Figure 30 shows the internal state calculations (calculation process for the internal calculation value X) when model α and model β are Mamba2.

[0288] Since there is a wealth of literature on the Mamba series, we will not go into detail here. Furthermore, the embedding process, direct terms (residual paths), and regression sections are not relevant to this embodiment, so their illustrations and explanations are omitted.

[0289] In both Mamba and Mamba2, the input u (here referred to as a sentence / prompt) is converted into n tokens (h1, h2, ...hm) through a tokenizer. Based on the tokens, each layer from the 1st to the sth layer is processed using the State Space Model (SSM) as its core, and a probability Pk (k: 1, 2, ...f) is calculated based on the output of the sth layer, where f is the number of candidate tokens. The token number j to be output is determined from P1 to Pf, and based on the token number j, the tokenizer outputs hj, which is then used as output y. This process is repeated recursively, and the output sentence is constructed by concatenating the outputs y.

[0290] The internal calculation value in both Mamba and Mamba2 is either h(t) (implicit latent state) in the SSM, or the values ​​of each element in matrices A￣ (upper line of A), B￣ (upper line of B), and C, or Δt (step size). Alternatively, a combination of these internal calculation values ​​may be used. Furthermore, internal calculation values ​​from multiple layers or internal calculation values ​​when regression is performed may also be used.

[0291] The Mamba series, based on SSM, is a model that is said to be capable of faster processing than Transformers, and by using Mamba in the model, even faster processing becomes possible.

[0292] Although several embodiments have been described above, these are merely illustrative examples for explaining the present invention and are not intended to limit the scope of the invention to these embodiments only. The present invention can be carried out in various other forms.

[0293] Furthermore, in the above explanation, "RAM24" is one or more memory devices (hereinafter also simply referred to as "memory") which are an example of one or more storage devices, and is typically a main memory device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0294] Furthermore, in the above explanation, "storage device 21" may be one or more persistent storage devices, which are examples of one or more storage devices. Persistent storage devices are typically non-volatile storage devices (e.g., auxiliary storage devices), and specifically, they may be, for example, HDDs (Hard Disk Drives), SSDs (Solid State Drives), NVME (Non-Volatile Memory Express) drives, or SCMs (Storage Class Memory).

[0295] Furthermore, in the above explanation, "storage device 21" may include memory.

[0296] Furthermore, in the above explanation, "CPU22" may refer to one or more processor devices (hereinafter also simply as "processors"). At least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit)22, but may also be other types of processor devices such as a GPU (Graphics Processing Unit). At least one processor device may be single-core or multi-core. At least one processor device may be a processor core. At least one processor device may be a broad-sense processor device such as a circuit that is a collection of gate arrays according to a hardware description language that performs some or all of the processing (e.g., FPGA (Field-Programmable Gate Array), CPLD (Complex Programmable Logic Device), or ASIC (Application Specific Integrated Circuit)).

[0297] Furthermore, although the above description may use the expression "xxx section" to describe functions, a function may be realized by the execution of one or more computer programs by the processor, by one or more hardware circuits (e.g., FPGA or ASIC), or by a combination thereof. When a function is realized by the execution of a program by the processor, the defined processing is carried out using the storage device 21 and / or interface device as appropriate, so the function may be at least a part of the processor. The processing described with a function as the subject may be processing performed by the processor or a reliability evaluation device (1,6) having that processor. The program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable recording medium (e.g., a non-temporary recording medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0298] Furthermore, in the above explanation, the process is sometimes described using "program" as the subject. However, the process described using "program" as the subject may also be a process performed by the processor or a reliability evaluation device (1,6) having that processor. Also, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0299] Furthermore, in the above description, the "reliability evaluation device (1,6)" may be a system composed of one or more physical computers (for example, an on-premise system), or a system implemented on a group of physical computing resources (for example, a cloud infrastructure) (for example, a cloud computing system). The reliability evaluation device (1,6) "displaying" the display information may mean displaying the display information on a display device owned by a computer, or the computer may transmit the display information to a display computer (in the latter case, the display information is displayed by the display computer). [Explanation of symbols]

[0300] 1. Reliability evaluation device (domain representative value z_in calculation device) Model α 3 Model β 4 Difference d calculation means 5. Means for calculating the representative value z_in of the region 6. Reliability evaluation device (reliability calculation device) 7 Difference D_v calculation means 8. Calculation method for confidence level λ_v 9. Confidence level C_v1 calculation method 10. Notification flag f_alert calculation means 11 Display device 21 Storage device for reliability evaluation equipment 22. CPU of the reliability evaluation device 23. ROM of a reliability evaluation device 24. RAM of the reliability evaluation device 25. Data bus of reliability evaluation device 26 Input circuit of reliability evaluation device 27 Input / Output Ports of Reliability Evaluation Device 28 Output circuit of reliability evaluation device

Claims

1. A reliability evaluation device for evaluating the reliability of a model, A computer having at least a processor and memory, A means for calculating at least one internal calculation value X_α of the model α when the input u is given, A means for calculating at least one internal calculation value X_β of the model β when the input u is given, A means for calculating the difference d between the internal calculation value X_α and the internal calculation value X_β based on the internal calculation value X_α and the internal calculation value X_β, If the difference d is not within a predetermined range or is greater than or equal to a predetermined value, then based on the internal calculation value X_β, means for calculating a value z_in that represents the region in which the aforementioned internal calculation value exists, A means for calculating a confidence level λ_v that represents the confidence level or accuracy of the output of the model β based on the z_in, A reliability evaluation device equipped with the following features.

2. In claim 1, A means for calculating at least one internal calculation value X_βv of the model β when the input v is given, A means for calculating the difference D_v based on the aforementioned X_βv and z_in, A means for calculating the confidence level λ_v of the output of the model β when the input v is given, based on the difference D_v. A reliability evaluation device equipped with the following features.

3. In claim 1, The aforementioned model α is at least one of the following: model, artificial intelligence, AI, generative AI, large-scale language model, LLM, or transformer. The reliability, confidence level, or certainty of the output or response of the aforementioned model α is calculated, The aforementioned processor, Using training data A to determine the parameter values, the values ​​of the internal parameters X: x1, x2, ..., xn of model α when input u is input to model α are calculated as X_α: x1_α, x2_α, ..., xn_α. When the input u is input to model β, which has updated the parameter values ​​of model α using training data B, the values ​​of the intrinsic parameters X: x1, x2, ..., xn of model β, X_β: x1_β, x2_β, ..., xn_β, are calculated. Based on the above X_α: x1_α, x2_α, ..., xn_α, at least one representative value z_α1 is calculated, Based on the above X_β: x1_β, x2_β, ..., xn_β, at least one representative value z_β is calculated, Based on the above z_α and z_β, the difference d between z_α and z_β is calculated, When the difference d is greater than or equal to a predetermined value, the value z_β or the value calculated based on z_β is set to z_in. When the input v is input to model β (output y_β is obtained), the values ​​of the internal parameters X: x1, x2, ..., xn of model β, X_βv: x1_βv, x2_βv, ..., xn_βv, are calculated. Based on the above X_βv: x1_βv, x2_βv, ..., xn_βv and the above z_in, the difference D_v: D1_v, D2_v, ..., Dn_v is calculated, Based on the differences D_v: D1_v, D2_v, ..., Dn_v, the confidence level λ_v of the output y_βv of the model β obtained after inputting the input v to the model β is calculated. A reliability evaluation device that performs processing.

4. In claim 3, The aforementioned internal calculation values ​​X_α, X_β are the internal calculation values ​​obtained when each of the multiple u1, u2, ..., um is input. When uk is input, the internal calculation value X_α is [x1_αk, x2_αk, ..., xn_αk], When uk is input, the internal calculation value X_β is [x1_βk, x2_βk, ..., xn_βk]. A reliability evaluation device characterized by the following features.

5. In claim 3, The means for calculating the value X_α of the internal parameter X of the model α is, When the input u is input to the (model) α, the internal parameters X1_α, x2_α, ..., xn_α are, After inputting the input u to the model α, the output y_α of the model α is calculated, and X_α consists of both internal parameters Xn+1_α, xn+2_α, ..., xn+m_α. Calculate, The means for calculating the value X_β of the internal parameter X of the model β is, When the input u is input to the (model) β, the internal parameters X1_β, x2_β, ..., xn_β are, After inputting the input u to the model β, the output y_β of the model β is calculated, and X_β consists of both internal parameters Xn+1_β, xn+2_β, ..., xn+m_β. Calculate, A means for calculating at least one representative value z_α based on the aforementioned X_α, A means for calculating at least one representative value z_β based on the aforementioned X_β, A means for calculating the difference d between z_α and z_β based on the aforementioned z_α and z_β, When the difference d is greater than or equal to a predetermined value, the means of setting z_β or a value calculated based on z_β to z_in When input v is input to the (model) β, the internal parameters X1_βv, x2_βv, ..., xn_βv are, After inputting the input v1 to the model β, the output y_βv of the model β is calculated, and X_βv consists of both internal parameters Xn+1_βv, xn+2_βv, ..., xn+m_βv. Calculate, A means for calculating the difference D_v: D1_v, D2_v, ..., Dn_v, Dn+1_v, Dn+2_v, ..., Dn+m_v based on each of the aforementioned X_βv: x1_βv, x2_βv, ..., xn_βv, xn+1_βv, xn+2_βv, ..., xn+m_βv and the aforementioned z_in, A means for calculating the confidence level λ_v of the output y_βv of the model β obtained after inputting the input v to the model β, based on the difference D_v, A reliability evaluation device equipped with the following features.

6. In claim 3, The internal parameter values ​​xk_α and xk_β are, The word number or token number in the aforementioned input u or the aforementioned input v. A reliability evaluation device characterized by the following features.

7. In claim 3, The means for calculating at least one representative value z_α or representative value z_β is, The central vector of x1_α, x2_α, ..., xn_α A reliability evaluation device characterized by the following features.

8. In claim 3, The means for calculating the difference d based on the aforementioned z_α and z_β is, This is the L2 distance, cosine distance, or Mahalanobis distance between the respective center vectors. A reliability evaluation device characterized by the following features.

9. In claim 3, The input u1 is data from the same field as the training data B. The output y_β satisfies a predetermined condition, Furthermore, when the difference d is greater than or equal to a predetermined value, Let z_in be the value z_β or the value calculated based on z_β. A reliability evaluation device characterized by the following features.

10. In claim 3, The aforementioned model β calculates the probability Pj for multiple output candidates, The value obtained by multiplying the confidence level λ_v of the output of model β by the probability Pj is defined as a second confidence level C_v, which represents the confidence level or certainty of the output of model β. A reliability evaluation device characterized by the following features.

11. In claim 3, Models α and β have multiple layers and multiple types of internal parameters. The above processing is performed on the multiple types of internal parameters in the multiple layers. A reliability evaluation device characterized by the following features.

12. In claim 11, Model α and Model β are transformers, The aforementioned internal calculation value is at least the attention vector, the magnitude of the attention vector, the parameter value of FFN, the input / output embedding value, or the output value of each layer. i represents the layer number, j represents the type of internal calculation value, The internal calculation values ​​for the aforementioned pair of inputs uk are xk_α(i,j) and xk_β(i,j), The representative value z_α(i,j) is based on the above xk_α(i,j). and The representative value z_β(i,j) is based on the above xk_β(i,j). And, Let the difference d(i,j) be based on the above z_α(i,j) and the above z_β(i,j). When the difference d(i,j) is greater than or equal to a predetermined value, the value calculated based on z_β(i,j) or z_β(i,j) is set to z_in(i,j). Let x1_βv(i,j), x2_βv(i,j), ..., xn_βv(i,j) be the values ​​of the internal calculations xk of the model β when the input v is input to the model β. Let D1_v(i,j), D2_v(i,j), ..., Dn_v(i,j) be the differences between each of the above x1_βv(i,j), x2_βv(i,j), ..., xn_βv(i,j) and the above z_in(i,j). Based on the differences D1_v(i,j), D2_v(i,j), ..., Dn_v(i,j), the confidence level of the output of the model β is determined as λ_v or λ_v(i,j). A reliability evaluation device characterized by the following features.

13. In claim 3 or claim 11, When λ_v is less than or equal to a predetermined value or C_v is less than or equal to a predetermined value, Announce that fact, or display information related to it on the screen. A reliability evaluation device characterized by the following features.

14. In claim 3 or claim 11, The value based on the aforementioned D_v, Dj_v, λ_v, or C_v will be displayed on the screen. A reliability evaluation device characterized by the following features.

15. In claim 3, The token v_k represents a word or token in the input v that corresponds to a Dk_v of a predetermined value or greater among the D1_v, D2_v, ..., Dn_v, and displays the token v_k that represents the word or token in the input v corresponding to the Dk_v of a predetermined value or greater. A reliability evaluation device characterized by the following features.

16. In claim 3, This displays the positional relationship between z_in and the D1_v, D2_v, ..., Dn_v values ​​based on the z_in values. A reliability evaluation device characterized by the following features.

17. In claim 3, Models α and β are either LLM or generative AI. A reliability evaluation device characterized by the following features.

18. In claim 17, The aforementioned LLM or generated AI is a State Space Model (SSM) based model. A reliability evaluation device characterized by the following features.

19. A reliability evaluation device for evaluating the reliability of a model, A computer having at least a processor and memory, A model α trained using a predetermined dataset A, Model β is obtained by further training Model α using one or more data B1 selected from a data group B different from the predetermined data group A, A means for determining a representative value z_in of X_β based on the internal state X_β of the model β when one or more data B1 or one or more data B2 selected from the data group B different from the one or more data B1 are input to the model β, A means for calculating the internal state X_βv of the model β when an input v is input to the model β, A means for determining the difference D_v between the internal state X_βv and the representative value z_in, A means for calculating the confidence level λ_v of the input v or the output y of the model β when the input v is given, based on the difference D_v. A reliability evaluation device equipped with the following features.

20. A reliability evaluation device for evaluating the reliability of a model, A computer having at least a processor and memory, Model β, which is a language model, A means for determining a representative value z_in of X_β based on the internal state X_β of the model β when an input u is input to the model β and / or when the output w is calculated when the input u is input, A means for calculating the internal state X_βv of the model β when an input v is input to the model β and / or when the output y is calculated when the input v is input, A means for determining the difference D_v between the internal state X_βv and the representative value z_in, A means for calculating the confidence level λ_v of the input v or the output y of the model β when the input v is given, based on the difference D_v. A reliability evaluation device equipped with the following features.

21. In claim 1, The aforementioned model is at least one of Model / AI / Artificial Intelligence α. A reliability evaluation device characterized by the following features.

22. A reliability evaluation method for evaluating the reliability of a model, In a computer having at least a processor and memory, The aforementioned processor, Calculate at least one internal calculation value X_α of the model α for the input u, At least one internal calculation value X_β of the model β for the input u is calculated, Based on the internal calculation value X_α and the internal calculation value X_β, the difference d between the internal calculation value X_α and the internal calculation value X_β is calculated. If the difference d is not within a predetermined range or is greater than or equal to a predetermined value, then based on the internal calculation value X_β, A value z_in representing the region where the aforementioned internal calculation value exists is calculated, Based on the aforementioned z_in, the confidence level λ_v of the output of the model β is calculated. Reliability evaluation methods.

23. A reliability evaluation method for evaluating the reliability of a model, In a computer having at least a processor, memory, and storage device, The aforementioned storage device is A model α trained using a predetermined dataset A, Model β is a model β that is further trained on Model α using one or more data B1 selected from a data group B different from the predetermined data group A. Remember this, The aforementioned processor, Based on the internal state X_β of the model β when one or more data B1 or one or more data B2 selected from the data group B different from the one or more data B1 are input to the model β, a representative value z_in of X_β is obtained. The internal state X_βv of the model β is calculated when the input v is input to the model β. The difference D_v between the internal state X_βv and the representative value z_in is calculated. Based on the difference D_v, the confidence level λ_v of the input v or the output y of the model β when the input v is given is calculated. Reliability evaluation methods.

24. A reliability evaluation method for evaluating the reliability of a model, In a computer having at least a processor, memory, and storage device, The aforementioned storage device is The language model is Model β Remember this, The aforementioned processor, When an input u is input to the model β and / or when the output w is calculated when the input u is input, a representative value z_in of the X_β is obtained. When input v is input to the model β and / or when output y is calculated when input v is input, the internal state X_βv of the model β is calculated, The difference D_v between the internal state X_βv and the representative value z_in is calculated. Based on the difference D_v, the confidence level λ_v of the input v or the output y of the model β when the input v is given is calculated. Reliability evaluation methods.

Images