Estimation device and estimation method

Abstract

The present invention provides an estimation device and estimation method capable of determining the quality of input data. [Solution] The estimation device comprises an estimation unit that estimates the response accuracy of a language model that outputs response information to user input data based on the input data, and an output unit that outputs information on the response accuracy estimated by the estimation unit.

Description

Technical Field

[0001] The present disclosure relates to an estimation device and an estimation method.

Background Art

[0002] In recent years, the use of systems that output answers to prompts (input data) input by users using generative AI (Artificial Intelligence) has been active. For example, Patent Document 1 discloses a configuration that outputs answers using a large language model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the content of the input data input by the user is inappropriate, it is difficult to obtain an answer expected by the user from the generative AI, so a configuration that can determine the quality of the content of the input data is desired.

[0005] Non-limiting embodiments of the present disclosure contribute to providing an estimation device and an estimation method capable of determining the quality of the content of input data.

Means for Solving the Problems

[0006] The estimation device according to the present disclosure an estimation unit that estimates the answer accuracy of a language model that outputs answer information for user input data based on the input data; an output unit that outputs information on the answer accuracy estimated by the estimation unit; and includes.

[0007] The estimation method according to the present disclosure A method for estimating the response accuracy of a language model that outputs response information for user input data, The accuracy of the response is estimated based on the input data, Outputting information on the estimated accuracy of the answer, It has. [Effects of the Invention]

[0008] According to this disclosure, it is possible to determine whether the content of the input data is of good quality. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the estimation device according to this embodiment. [Figure 2] This figure shows an example of how information on the accuracy of the response obtained by the estimation device is displayed. [Figure 3] This figure shows an example of how information on the accuracy of the response obtained by the estimation device is displayed. [Figure 4] This flowchart shows an example of the operation of the estimation control for answer accuracy in the estimation device. [Figure 5] This figure shows an example of an estimation device related to a modified example. [Figure 6] Figure 5 is a flowchart showing an example of feedback control operation in the estimation device. [Figure 7] This figure shows an example of an estimation device related to a modified example. [Figure 8] Figure 7 is a flowchart showing an example of the operation of the display control of the feedback value in the estimation device. [Modes for carrying out the invention]

[0010] (Embodiment) The embodiments of this disclosure will now be described in detail with reference to the drawings. Figure 1 is a diagram showing the estimation device 100 according to this embodiment.

[0011] As shown in FIG. 1, the estimation device 100 is an information processing device that outputs response information for a prompt (input data) input by a user using a generative AI, and includes, for example, a language model 110. The language model 110 is, for example, a known large language model and is a model trained using a large amount of datasets. With this language model 110, the estimation device 100 generates and outputs a response to the input data input by the user. Note that the configuration for outputting response information for the input data by the language model 110 may be the same as, for example, a known technique (for example, the technique described in Patent Document 1).

[0012] Further, the estimation device 100 is configured to be able to estimate the response accuracy of the language model 110. The estimation device 100 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and input / output circuits (not shown), and estimates the response accuracy of the language model 110 based on a preset program. The estimation device 100 has an estimation unit 120 and a display unit 130.

[0013] The estimation unit 120 is, for example, a language model provided separately from the language model 110 and is trained using a large amount of datasets so that the response accuracy can be estimated. The estimation unit 120 is also a language model that can be generated using a known technique similar to the language model 110.

[0014] The estimation unit 120 is capable of in-context learning (referred to as Few Shot prompting) that improves the response accuracy by giving examples to the language model as prompts. Also, although the estimation unit 120 is provided separately from the language model 110, it may be integrated with the language model 110.

[0015] The estimation unit 120 estimates the answer accuracy of the language model 110 based on the user's input data. The answer accuracy is, for example, a score indicating the degree of user satisfaction with the content of the answer information output by the language model 110, and is the score output by the estimation unit 120, which is a language model, for the input data input by the user so as to estimate the answer accuracy. Note that the answer accuracy may be a value preset for each sentence related to the input data.

[0016] For example, when the user's input data is input, the estimation unit 120 outputs information on the answer accuracy associated with the input data to the display unit 130.

[0017] The display unit 130 is an output unit that displays and outputs the answer accuracy information output by the estimation unit 120 to the user. The answer accuracy is expressed, for example, by a number, color, bar length, etc. The display unit 130 displays the answer accuracy information, for example, below the input data input by the user on the display screen.

[0018] For example, as shown in FIG. 2, assume that the user inputs predetermined input data (the part of reference sign I1). In FIG. 2, "An idea for a patent application is desired" is shown as an example of the input data. In contrast, an input (the part of reference sign I2) is given to the estimation unit 120 to request estimation of the answer accuracy for the input input data. In FIG. 2, "When this data is input, the language model returns some output, but... please estimate with a score from 0 to 1 whether an answer that satisfies the user is output for this input" is shown as an example of the input.

[0019] The input regarding the request for estimation of the answer accuracy may or may not be displayed on the display unit 130. The input regarding the request for estimation of the answer accuracy is automatically generated and input as a prompt when the user's input data is input.

[0020] The estimation unit 120 receives user input data and input regarding the request for estimation of response accuracy, and outputs response accuracy information associated with the user's input data to the display unit 130.

[0021] Figure 2 shows an example of response accuracy information output by the estimation unit 120, where the part labeled O1 is shown. In the example in Figure 2, since the user's input data is relatively simple, 0.4 is shown as the score indicating response accuracy. In addition, in the example in Figure 2, along with the score, the following is output: "...Since no specific theme or field is indicated,...the possibility of a satisfactory answer being output is considered low. The score is estimated to be 0.4..." Thus, the estimation unit 120 may output information regarding the reason for estimating the response accuracy, along with the response accuracy, to the display unit 130.

[0022] The information regarding the reasoning behind the estimation of response accuracy is generated by the estimation unit 120, which is a language model, but it may also be information about a score indicating response accuracy or a set phrase associated with each range of scores.

[0023] For example, Figure 3 shows an example where the user's input data includes specific content. In the example in Figure 3, input such as "Please list using AI..." (part I3) is entered following the content of I1 in Figure 2.

[0024] In this case, since more specific information is input than what is shown in Figure 2, the output (the part with code O2) is "...Sufficient information has been provided for the language model to generate an appropriate output....The score is estimated to be 0.7...." For example, the content shown in Figure 3 has higher response accuracy than the content shown in Figure 2, so the information regarding the reason for the estimated response accuracy is more positive than that shown in Figure 2.

[0025] Furthermore, information regarding the reasons for estimating the accuracy of the response may or may not be displayed on the display unit 130.

[0026] Furthermore, the estimation unit 120 may update the answer accuracy information when, for example, a symbol indicating a sentence delimiter, such as a punctuation mark (comma, period, etc. in the case of English), is entered by the user. For example, each time a symbol indicating a sentence delimiter is entered, the answer accuracy may be estimated from the text entered up to that point, and the answer accuracy information may be output to the display unit 130 each time.

[0027] Furthermore, the estimation unit 120 may predict the next content the user will input based on the user's input data and output it to the display unit 130. For example, suppose the user inputs the content of code I1 shown in Figure 2. The estimation unit 120, which is a language model, uses a known method to predict the next input content, such as code I3, "Utilizing AI...", based on the content input by the user, and outputs it to the display unit 130.

[0028] Furthermore, the estimation unit 120 may also estimate the accuracy of the response based on the predicted content. For example, the estimation unit 120 may estimate the accuracy of the response by assuming that the input content of both code I1 and code I3 is confirmed. In addition, the estimation unit 120 may predict multiple patterns of content that the user will input next. In this case, the estimation unit 120 may estimate the accuracy of the response for each of the multiple patterns.

[0029] Furthermore, the display unit 130 may also display each word included in the input data separately according to its degree of contribution to the answer accuracy, in addition to the answer accuracy information. Information on the degree of contribution of each word may be obtained, for example, by the estimation unit 120 calculating the degree of contribution of each word using a known method, or by inputting a prompt to the estimation unit 120 to output the degree of contribution of each word to the answer accuracy. This prompt is entered automatically, but may also be entered by the user.

[0030] For example, suppose the user inputs the part labeled I1 in Figure 3, "I would like some ideas for intellectual property applications," and the part labeled I3, starting with "using AI...". In this case, suppose the words "software development" and "efficiency" contribute positively to the accuracy of the response, while the word "AI" contributes negatively to the accuracy of the response. In this case, the display unit 130 displays the words "software development" and "efficiency" and the word "AI" in different ways.

[0031] Specifically, for example, words like "software development" and "efficiency" are highlighted in a relatively prominent color, while the word "AI" is highlighted in a relatively dark color.

[0032] This approach makes it easier to obtain hints that will help users get the answers they are looking for.

[0033] Next, an example of the operation of the estimation device 100 of this disclosure will be described. Figure 4 is a flowchart showing an example of the operation of the estimation control for answer accuracy in the estimation device 100. This control is performed as appropriate when a user inputs input data into the estimation device 100.

[0034] As shown in Figure 4, the estimation device 100 acquires user input data (step S101). After step S101, the estimation device 100 inputs the input data to the estimation unit 120 to estimate the response accuracy and outputs information on the estimated response accuracy (step S102).

[0035] After step S102, the estimation device 100 displays the answer accuracy information via the display unit 130 (step S103).

[0036] After step S103, the estimation device 100 determines whether or not to finalize the input data (step S104). The estimation device 100 determines that the input data is finalized if, for example, the user confirms the answer accuracy information displayed on the display unit 130 and then performs an operation such as pressing the Enter key. The estimation device 100 also determines that the input data is not finalized if, for example, the user confirms the answer accuracy information displayed on the display unit 130 and then performs an operation to start inputting again.

[0037] If the determination is made not to finalize the input data (step S104, NO), the process returns to step S101. On the other hand, if the determination is made to finalize the input data (step S104, YES), the estimation device 100 outputs the input data to the language model 110 (step S105).

[0038] After step S105, this control process ends. Note that in step S105, when input data is output to the language model 110, the language model 110 outputs response information for the input data. This response information is displayed, for example, on the display unit 130.

[0039] According to this embodiment configured as described above, the estimation unit 120 estimates the response accuracy of the language model 110 based on the input data and outputs information on the response accuracy. Therefore, the user can easily determine whether the input data is likely to yield the desired response.

[0040] For example, since users can judge the quality of the input data (prompt), it becomes easier to reconsider input data that provides a higher accuracy answer based on that input data. As a result, it becomes easier for users to obtain the desired answer.

[0041] Furthermore, each word included in the response information is displayed separately according to its degree of contribution to the accuracy of the response, making it easier for users to obtain hints to get the desired answer.

[0042] Note that the information regarding the accuracy of the response may be changed according to user evaluations. Figure 5 shows an example of an estimation device 100 related to a modified example.

[0043] As shown in Figure 5, the modified estimation device 100 has a feedback unit 140 in addition to the configuration of the above embodiment.

[0044] The feedback unit 140 feeds back user evaluation information regarding the response accuracy information to the estimation unit 120. For example, suppose the user is presented with response accuracy information (estimated value) by the estimation unit 120, and then the response information for the user's input data is presented by the display unit 130.

[0045] Subsequently, the feedback unit 140 outputs information via the display unit 130 to ask the user whether or not they were satisfied with the response information. The information asked of the user may be, for example, information that allows the user to select one of several response options, such as dissatisfied, neutral, satisfied, or levels 1 to 3.

[0046] When the user provides an answer to the information requested to the user, the feedback unit 140 compares the feedback information with the answer accuracy information.

[0047] For example, suppose the response accuracy information is estimated to be a relatively high score of 0.7, while the feedback information from the user is level 1 (or unsatisfied). In this case, the feedback unit 140 determines that the difference between the value corresponding to level 1 (e.g., 0.3) and the estimated value of 0.7 is greater than a predetermined value, and outputs 0.3 as the feedback value to the estimation unit 120. The estimation unit 120 then retains this information.

[0048] The estimation unit 120, which is capable of in-context learning, includes past feedback values ​​in the prompt when the user inputs a prompt again, and estimates the response accuracy that reflects the past feedback values. Specifically, if there has been one or more feedbacks in the past, the estimation unit 120 reads one or more datasets consisting of combinations of input data to the language model 110 entered by the user at that time and the corresponding feedback values, adds them to the prompt, and determines the response accuracy.

[0049] In the example shown in Figure 3, the estimation unit 120 inserts one or more datasets between the content of code I3 and the content of code I2, and also adds the sentence "Please answer based on past estimation examples" to the end of the content of code I2 to create a prompt. This makes the output of the language model 110's response accuracy closer to the user's intuition.

[0050] In this way, the actual level of user satisfaction with the response information obtained from the language model 110 can be easily reflected in the estimates made by the estimation unit 120.

[0051] Furthermore, if there is no difference between the feedback information and the response accuracy information, the feedback unit 140 does not need to output the feedback value to the estimation unit 120.

[0052] When there is no difference between the feedback information and the response accuracy information, this includes cases where the feedback value and the estimated value corresponding to the response accuracy information are equal, as well as cases where the difference between the feedback value and the estimated value is within a predetermined small range.

[0053] This simplifies the processing of the estimation device 100.

[0054] Next, the control of the estimation device 100 shown in Figure 5 will be described. Figure 6 is a flowchart showing an example of the operation of the feedback control in the estimation device 100 shown in Figure 5. This control is executed as appropriate when the user inputs feedback information into the estimation device 100.

[0055] As shown in Figure 6, the estimation device 100 determines whether the difference between the feedback value and the estimated value is greater than a predetermined value (step S201). The predetermined value is a relatively large value and can be set as appropriate. Alternatively, instead of step S201, the device may be configured to perform feedback only if the user inputs a feedback value, and not perform feedback if the user does not input a feedback value.

[0056] If the result of the determination is that the difference between the feedback value and the estimated value is less than or equal to a predetermined value (step S201, NO), this control is terminated. On the other hand, if the difference between the feedback value and the estimated value is greater than the predetermined value (step S201, YES), the estimation device 100 inputs the feedback value to the estimation unit 120 (step S202).

[0057] The estimation unit 120 then stores the feedback value together with the input data to the language model 110 that was input by the user at that time (step S203). This process is repeated one or more times, and one or more datasets are stored in the estimation unit 120.

[0058] The next time the user enters a prompt, the estimation unit 120, which is capable of in-context learning, will include one or more datasets and the phrase "Please answer based on past estimation examples" in the prompt, and will estimate the response accuracy that reflects past feedback values. This makes the output of the response accuracy of the language model 110 closer to the user's perception.

[0059] Furthermore, in the estimation device 100 shown in Figure 5, the estimation unit 120 holds user feedback values, but it is also possible to allow the user to delete inappropriate feedback values ​​from the held feedback values. Figure 7 shows an example of an estimation device 100 according to a modified example.

[0060] As shown in Figure 7, the modified estimation device 100 has a history management unit 150 and a storage unit 160 in addition to the configuration shown in Figure 5.

[0061] The history management unit 150 stores the feedback values ​​(evaluation information) given to the estimation unit 120 in the storage unit 160 and manages the history of the feedback values ​​stored in the storage unit 160. The storage unit 160 stores one or more datasets consisting of combinations of input data to the language model 110 entered by the user and the corresponding feedback values, through the process shown in Figure 6.

[0062] The history management unit 150 controls the display unit 130 to display one or more datasets stored in the storage unit 160 when requested by the user. This allows the user to easily check the results of their past feedback.

[0063] Furthermore, the history management unit 150 deletes at least a portion of the feedback values ​​stored in the storage unit 160 based on the user's operation.

[0064] For example, suppose a user has previously requested a more detailed answer from the estimation device 100 (language model 110), and has provided feedback to the estimation device 100 indicating a harsh evaluation of that answer.

[0065] However, if, due to the user's circumstances, a relatively rough answer is sufficient, and the feedback value that has been given a strict evaluation by the estimation unit 120 is referenced, an estimated value with an answer accuracy that does not match the user's perception will be output.

[0066] In such cases, the user deletes, for example, a feedback value currently set as an estimate of the response accuracy via an operation unit (not shown). Based on this operation, the history management unit 150 deletes the feedback value.

[0067] This allows the accuracy of the answers to be adjusted to match the user's preferred responses.

[0068] Next, the control of the estimation device 100 shown in Figure 7 will be described. Figure 8 is a flowchart showing an example of the operation of the display control of the feedback value in the estimation device 100 shown in Figure 7. This control is executed as appropriate, for example, when the feedback value is reflected in the estimation unit 120 in the control shown in Figure 6.

[0069] As shown in Figure 8, the estimation device 100 stores a dataset in the storage unit 160 consisting of a combination of a feedback value input to the estimation unit 120 and the corresponding input data entered by the user as a prompt (step S301). After step S301, the estimation device 100 displays the dataset previously stored in the storage unit 160 on the display unit 130 (step S302).

[0070] After step S302, the estimation device 100 determines whether or not there is a user instruction to delete the feedback value (step S303). If the determination is that there is no instruction to delete the feedback value (step S303, NO), this control terminates.

[0071] On the other hand, if there is an instruction to delete the feedback value (step S303, YES), the estimation device 100 deletes the dataset containing the feedback value to be deleted from the storage unit 160 (step S304). After that, this control is terminated.

[0072] Subsequently, when a prompt is entered by the user, the estimation unit 120, which is capable of in-context learning, includes the dataset from which inappropriate feedback values ​​have been removed in the prompt and estimates the response accuracy that reflects the feedback values ​​other than those that have been removed. This makes the output of the response accuracy of the language model 110 closer to the user's perception.

[0073] In the above embodiment, the language model 110 was provided in the estimation device 100, but this disclosure is not limited thereto, and the language model 110 may be provided in another device. In this case, the estimation device 100 may be provided between the input interface for user input data and the other device.

[0074] Furthermore, in the above embodiment, the display unit 130, which is the display medium, was provided on the estimation device 100, but the disclosure is not limited thereto, and the display medium may be provided on a device other than the estimation device. In this case, the estimation device only needs to be configured to have an output unit capable of transmitting data related to the accuracy of the answer to the above device.

[0075] Furthermore, although the output unit in the above embodiment was a display unit 130, this disclosure is not limited to this, and other configurations besides a display unit are also acceptable as long as they can output information on the accuracy of the response, such as audio or printed data.

[0076] Furthermore, the embodiments described above are merely examples of how this disclosure may be implemented, and the technical scope of this disclosure should not be limited by them. Therefore, this disclosure can be implemented in various ways without departing from its essence or its main features. [Industrial applicability]

[0077] The estimation device described herein is useful as an estimation device and estimation method capable of determining the quality of the content of input data. [Explanation of Symbols]

[0078] 100 Estimator 110 Language Models 120 Estimation part 130 Display section 140 Feedback Section 150 History Management Department 160 Storage section

Claims

1. An estimation unit that estimates the response accuracy of a language model that outputs response information for user input data based on the input data, An output unit that outputs information on the accuracy of the response estimated by the estimation unit, An estimation device equipped with the following features.

2. The aforementioned response accuracy is a score indicating the user's level of satisfaction with the content of the response information. The estimation device according to claim 1.

3. The output unit outputs the response accuracy information to a display medium. The estimation unit controls the display medium to distinguish each word included in the response information according to its degree of contribution to the response accuracy. The estimation device according to claim 1.

4. The system further includes a feedback unit that feeds back the user's evaluation information to the estimation unit regarding the information on the accuracy of the response, The estimation unit determines the information regarding the accuracy of the response based on the information fed back from the feedback unit. The estimation device according to claim 3.

5. A storage unit that stores the evaluation information fed back by the feedback unit, A history management unit that manages the history of the evaluation information stored in the memory unit, Furthermore, The history management unit controls the display medium to display past evaluation information stored in the storage unit. The estimation device according to claim 4.

6. The history management unit deletes at least a portion of the evaluation information stored in the storage unit based on the user's operation. The estimation device according to claim 5.

7. The estimation unit is a model different from the language model, which is a trained model that takes the input data as input and outputs the response accuracy information. The estimation device according to claim 1.

8. A method for estimating the response accuracy of a language model that outputs response information for user input data, The accuracy of the response is estimated based on the input data, Outputting information on the estimated accuracy of the answer, An estimation method having

9. The aforementioned response accuracy is a score indicating the user's level of satisfaction with the content of the response information. The estimation method according to claim 8.

10. In outputting the above information, the response accuracy information is output to a display medium. In the estimation process, control is performed to distinguish each word included in the response information according to its degree of contribution to the response accuracy and to display it on the display medium. The estimation method according to claim 8.

11. This further includes providing feedback on the user's evaluation information regarding the accuracy of the response, In making the above estimation, the information on the accuracy of the response is determined based on the feedback information. The estimation method according to claim 10.

12. The evaluation information that has been fed back is stored, To manage the history of the stored evaluation information, It further includes, In managing the history, control is performed to display the stored past evaluation information on the display medium. The estimation method according to claim 11.

13. In managing the history, delete at least a portion of the stored evaluation information based on the user's actions. The estimation method according to claim 12.

14. The estimation of the response accuracy is performed using a model different from the language model, which has been trained to receive the input data and output information on the response accuracy. The estimation method according to claim 8.

Images