Program, information processing system, and information processing method

Abstract

To efficiently make a highly reliable evaluation on a generated text.SOLUTION: A program causes an information processing system to implement: an acquisition unit which acquires input information to be input to a large language model, and response information corresponding to the input information; and an evaluation unit which uses a model to evaluate the response information based on the input information and the response information acquired by the acquisition unit. The model is generated by machine learning using training data including input information, response information corresponding to the input information, and label information obtained by evaluating the response information with arbitrary items.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing system, and an information processing method. [Background technology]

[0002] Technologies are being studied to make the expressions in response texts generated by large-scale language models easier for users to understand and to make the contents of the response texts more in line with the purpose and context of the dialogue.

[0003] For example, Patent Document 1 discloses a technology that modifies a response text generated by a response generation unit, determines whether the expression in the modified response text is appropriate for the user, and whether the content is appropriate for the content of past exchanges, and executes a process to determine the response text to be output from a group of modified response texts based on the obtained results. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-158992 Summary of the Invention [Problem to be solved by the invention]

[0005] When evaluating the quality of text generated by an arbitrary large-scale language model, two methods are possible: evaluation using a larger-scale language model with higher performance, and evaluation by humans. While evaluation using a larger-scale language model with higher performance can efficiently evaluate the quality of the generated text, the reliability of the evaluation tends to be lower than that of human evaluation. On the other hand, human evaluation allows for highly reliable evaluation, but the cost of the evaluation work tends to be high.

[0006] Therefore, an object of the present invention is to provide a program, an information processing system, and an information processing method that can efficiently perform a more reliable evaluation of generated text. [Means for solving the problem]

[0007] A program according to one aspect of the present invention is a program that causes an information processing system to implement an acquisition unit that acquires input information for a large-scale language model and response information corresponding to the input information, and an evaluation unit that evaluates the response information using a model based on the input information and response information acquired by the acquisition unit, wherein the model is a model generated by machine learning of training data that includes the input information, the response information corresponding to the input information, and label information that evaluates the response information using any item.

[0008] An information processing system according to one aspect of the present invention includes an acquisition unit that acquires input information for a large-scale language model and response information corresponding to the input information, and an evaluation unit that evaluates the response information using a model based on the input information and response information acquired by the acquisition unit, wherein the model is a model generated by machine learning of training data that includes the input information, the response information corresponding to the input information, and evaluation information that evaluates the response information on any item.

[0009] An information processing method according to one aspect of the present invention includes an information processing system that acquires input information for a large-scale language model and response information corresponding to the input information, and evaluates the response information using a model based on the acquired input information and response information, wherein the model is a model generated by machine learning of training data that includes the input information, the response information corresponding to the input information, and label information evaluating the response information for any item. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide a program, an information processing system, and an information processing method that can efficiently perform a more reliable evaluation of generated text. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram showing a configuration of an information processing system 100 according to an embodiment of the present invention. [Figure 2A] FIG. 1 is a diagram showing a flow of creating learning data for machine learning in the information processing system 100. [Figure 2B] 1 is a flowchart showing an example of machine learning processing in the information processing system 100. [Figure 3A] FIG. 2 is a diagram illustrating an image of evaluation processing in the information processing system 100. [Figure 3B] 10 is a flowchart showing an example of evaluation processing in the information processing system 100. [Figure 4A] FIG. 10 is a diagram showing another example of the evaluation process in the information processing system 100. [Figure 4B] 10 is a flowchart showing another example of the evaluation process in the information processing system 100. [Figure 5] FIG. 2 is a diagram illustrating an image of evaluation processing in the information processing system 100. DETAILED DESCRIPTION OF THE INVENTION

[0012] One embodiment of the present invention will be described in detail below. Note that the following embodiment is an example for explaining the present invention, and is not intended to limit the present invention to only that embodiment. Furthermore, the present invention can be modified in various ways without departing from the gist of the present invention. Furthermore, those skilled in the art can adopt embodiments in which the elements described below are replaced with equivalents, and such embodiments are also included in the scope of the present invention. Furthermore, to facilitate understanding, the following description will be given taking as an example an embodiment in which the present invention is realized using an information processing device, but as mentioned above, the present invention is not limited to this.

[0013] 1 is a diagram showing an example of the configuration of an information processing system 100 according to one embodiment of the present invention. The information processing system 100 is an information processing system implemented by an information processing program. The information processing system 100 evaluates response information based on input information for a large-scale language model and response information corresponding to the input information, and may be configured using one or more computers.

[0014] In one embodiment of the present invention, a large-scale language model (hereinafter also referred to as "LLM") is a model that is trained from a huge amount of text data and is capable of performing a variety of natural language processing tasks, such as question answering, sentence generation, summarization, translation, and text classification.

[0015] Furthermore, input information to a large-scale language model refers to, for example, input or questions from a user, or instructions or information for the model, and refers to information that serves as a starting point or trigger for the large-scale language model to generate a response sentence. Furthermore, response information refers to information that the large-scale language model outputs in response to an input.

[0016] The information processing system 100 may be communicably connected to a user terminal 200 that uses the response information evaluation service via a network such as the Internet.

[0017] The user terminal 200 is a computer used by a user, such as a personal computer or a mobile terminal. The user terminal 200 can access the information processing system 100, for example, by a web browser or a dedicated application. Although only one user terminal 200 is shown in FIG. 1, in reality, multiple user terminals 200 exist.

[0018] The following describes details of the information processing system 100. The information processing system 100 may include a storage unit 110, an acquisition unit 130, a learning unit 120, an evaluation unit 140, and an output unit 150. Each of these units is realized, for example, by using a storage area such as a memory or by a processor executing a program stored in the storage area.

[0019] The storage unit 110 may store the input information and response information acquired by the acquisition unit 130, or may store learning data when the learning unit 120 performs machine learning. The storage unit 110 may also record the evaluation result of the response information evaluated by the evaluation unit 140 using a model.

[0020] FIG. 2A shows a flow of creating learning data for machine learning in the information processing system 100.

[0021] The training data includes input information, response information corresponding to the input information, and label information evaluating the response information based on arbitrary items. The example shown in Figure 2A illustrates a manner in which label information evaluating based on arbitrary items is attached to response information output in response to certain input information in an arbitrary large-scale language model.

[0022] Here, the label information may be score data evaluated by a human. By using such data as training data, a model capable of performing highly reliable evaluation that reflects human evaluation can be obtained. Furthermore, in one embodiment of the present invention, score data obtained by manual evaluation in the past may be used as part of the training data as the score data evaluated by a human.

[0023] The items included in the label information are not particularly limited, and may include, for example, items such as relevance, accuracy, fluency, and amount of information. Here, relevance is an item that indicates whether the response information is relevant as an answer to the question of the input information, accuracy is an item that indicates the correctness of the response information, fluency is an item that indicates the correctness of the grammar and vocabulary in the response information, the smoothness of the writing, and readability, and amount of information is an item that indicates the amount and usefulness of the information included in the response information.

[0024] The label information may be, for example, an arbitrary score for the evaluation item, such as 1 to 5. In addition to items that can be graded by score, the label information may also include items that classify question categories into arbitrary classes.

[0025] The learning unit 120 generates a model for evaluating the response information based on the input information and the response information by executing machine learning based on the learning data stored in the storage unit 110. The algorithm for the machine learning performed by the learning unit 120 is not particularly limited, but may be, for example, a random forest or an SVM (support vector machine).

[0026] The acquisition unit 130 acquires input information for a large-scale language model and response information corresponding to the input information, and stores them in the storage unit 110. The acquisition unit 130 may acquire input information from the user terminal 200, input the input information to a server of an external organization, for example, a server that provides a service for using a large-scale language model, and acquire one piece of response information corresponding to the input information.

[0027] The acquisition unit 130 may input input information to one large-scale language model and acquire one piece of response information corresponding to the input information, or may input input information to each of multiple large-scale language models and acquire multiple pieces of response information corresponding to the input information from each of the multiple large-scale language models for each large-scale language model.

[0028] Furthermore, as long as the input information and response information are input to the large-scale language model, they do not need to be actual input, and the acquisition unit 130 may acquire input learning information corresponding to the input information and response learning information corresponding to the response information as instruction data to be used for training the large-scale language model.

[0029] The evaluation unit 140 evaluates the response information using the above model based on the input information and response information acquired by the acquisition unit 130, and stores the evaluated information in the storage unit 110. Figures 3A, 4A, and 5 show images of the evaluation process in the information processing system 100.

[0030] As shown in FIG. 3A, the evaluation unit 140 may evaluate the quality of the response information by any item based on the input information and the response information.

[0031] 4A, when the acquisition unit 130 acquires multiple pieces of response information 1 to 3 corresponding to input information from multiple large-scale language models LLM1 to LLM3, the evaluation unit 140 may evaluate each of the multiple pieces of response information 1 to 3 acquired by the acquisition unit 130. This makes it possible to compare the response information output by multiple large-scale language models for the same input information.

[0032] Furthermore, as shown in FIG. 5, when the acquisition unit 130 acquires input learning information corresponding to the input information and response learning information corresponding to the response information as instruction data used for training the large-scale language model, the evaluation unit 140 may evaluate the response learning information using the model based on the input learning information and response learning information acquired by the acquisition unit 130. Here, the evaluation of the response learning information may be information that can be used when using the instruction data as learning data for supervised learning. Furthermore, the evaluation of the response learning information may be, for example, as shown in FIG. 5, a score evaluation of the quality of the response information for any item, similar to the evaluation results shown in FIG. 3A, etc.

[0033] The output unit 150 outputs the evaluation result by the evaluation unit 140, as shown in FIG. 3A.

[0034] Furthermore, in the case where the evaluation unit 140 evaluates each of the plurality of pieces of response information acquired by the acquisition unit 130, as shown in FIG. 4A, the output unit 150 may output at least one of the plurality of pieces of response information based on the evaluation of each of the plurality of pieces of response information.

[0035] Specifically, the output unit 150 may output one piece of response information with the highest evaluation, or may output multiple pieces of response information with evaluations exceeding a certain threshold, or may output response information that combines multiple pieces of response information so as to obtain a higher evaluation.

[0036] Furthermore, when evaluation items to be emphasized, such as emphasizing accuracy as response information, are set in advance, the output unit 150 may output response information that is highly evaluated for accuracy. The evaluation items to be emphasized may be set according to the input information, or the acquisition unit 130 may acquire information on the evaluation items to be emphasized from the user terminal 200 together with the input information.

[0037] This allows the user to obtain more appropriate response information from among the response information of multiple large-scale language models in response to input information.

[0038] FIG. 2B is a flowchart showing an example of machine learning processing in the information processing system 100.

[0039] As shown in Fig. 2B, in the machine learning process, first, response information output by a human in response to certain input information in an arbitrary large-scale language model is assigned label information evaluated using arbitrary items, thereby creating training data including the input information, response information corresponding to the input information, and label information evaluating the response information using arbitrary items, and storing the training data in the storage unit 110 (S21, see Fig. 2A). Then, the learning unit 120 performs machine learning using the training data stored in the storage unit 110 to generate a model (S22). In one embodiment of the present invention, the machine learning may be supervised learning using labeled data.

[0040] FIG. 3B is a flowchart showing an example of the evaluation process in the information processing system 100.

[0041] As shown in FIG. 3B, in the evaluation process, the acquisition unit 130 may acquire input information from the user terminal 200 (S31), input the input information to a server of an external organization, for example, a server that provides a service for using a large-scale language model, and acquire one piece of response information corresponding to the input information (S32).

[0042] Alternatively, if the user terminal 200 has acquired response information from a large-scale language model in advance, the acquisition unit 130 may acquire the input information and the response information from the user terminal 200 (S31, S32).

[0043] Then, the evaluation unit 140 may evaluate the response information using the above model based on the input information and response information acquired by the acquisition unit 130 (S33), and finally, the output unit 150 may transmit the result to the user terminal 200.

[0044] In addition, in steps S31 and S32, if the acquisition unit 130 acquires input learning information corresponding to the input information and response learning information corresponding to the response information as instruction data to be used for training the large-scale language model, the evaluation unit 140 may evaluate the response learning information using the above model based on the input learning information and response learning information acquired by the acquisition unit 130 (S33).

[0045] FIG. 4B is a flowchart showing another example of the evaluation process in the information processing system 100.

[0046] As shown in FIG. 4B, when multiple large-scale language models LLM1 to LLM3 are used, the acquisition unit 130 may acquire input information from the user terminal 200 (S41), input the input information to multiple servers that provide services using the large-scale language models, and acquire multiple pieces of response information for each large-scale language model in response to the input information (S42).

[0047] Then, the evaluation unit 140 evaluates the multiple pieces of response information using a model based on the input information and response information acquired by the acquisition unit 130 (S43), and the output unit 150 may output the results in any format, such as outputting each evaluation result as shown in FIG. 4A (S44).

[0048] Furthermore, evaluation items may be added or changed later. For example, suppose that after creating a model that evaluates based on four items (relevance, accuracy, fluency, and information volume), two additional items, evaluation item A and evaluation item B, are added to create a model that outputs a total of six evaluation items. In this case, training data may be prepared by manually adding evaluation items A and B to the training data used to create the model that evaluates based on the original four evaluation items, and machine learning may be performed using this training data to create a model that outputs a total of six evaluation items. By evaluating the LLM using the model obtained in this way, it becomes possible to evaluate based on the new evaluation items as well.

[0049] Furthermore, in the above example, if evaluation item A is related to an item that has already been evaluated, for example, one of the original four evaluation items, and can be evaluated by calculating the score of such an evaluated item, or if evaluation item A can be evaluated based on the number of words or characters in the response information, and it is possible to evaluate it mechanically rather than manually, the mechanically determined item may be used as the evaluation score for item A.

[0050] An embodiment of the present invention has been described above. According to this embodiment, a model for evaluating response information is generated based on input information and response information by machine learning using training data including input information, response information corresponding to the input information, and label information evaluating the response information for any item. Then, by inputting the input information for a large-scale language model and the response information corresponding to the input information into the model, it is possible to perform an evaluation efficiently and with high reliability. [Explanation of symbols]

[0051] 100...information processing system, 110...storage unit, 120...learning unit, 130...acquisition unit, 140...evaluation unit, 150...output unit, 200...user terminal

Claims

1. Information processing systems, an acquisition unit that acquires input information for a large-scale language model and response information corresponding to the input information; an evaluation unit that evaluates the response information using a model based on the input information and the response information acquired by the acquisition unit, The model is a model generated by machine learning of training data including input information, response information corresponding to the input information, and label information evaluating the response information on any item. program.

2. The label information includes score data obtained by manually evaluating the response information. The program according to claim 1.

3. the large-scale language model includes a plurality of large-scale language models; the acquisition unit acquires a plurality of pieces of response information corresponding to the input information from each of the plurality of large-scale language models; the evaluation unit evaluates each of the plurality of pieces of response information acquired by the acquisition unit, and further realizing an output unit that outputs at least one of the plurality of pieces of response information based on an evaluation of each of the plurality of pieces of response information. The program according to claim 1.

4. the acquisition unit acquires input learning information corresponding to the input information and response learning information corresponding to the response information as instruction data used for training the large-scale language model; the evaluation unit evaluates the response learning information using the model based on the input learning information and the response learning information acquired by the acquisition unit. The program according to claim 1.

5. an acquisition unit that acquires input information for a large-scale language model and response information corresponding to the input information; an evaluation unit that evaluates the response information using a model based on the input information and the response information acquired by the acquisition unit, The model is a model generated by machine learning of learning data including input information, response information corresponding to the input information, and evaluation information evaluating the response information on any item. Information processing system.

6. The information processing system obtaining input information for a large-scale language model and response information corresponding to the input information; and evaluating the response information using a model based on the acquired input information and the acquired response information; The model is a model generated by machine learning of training data including input information, response information corresponding to the input information, and label information evaluating the response information on any item. Information processing methods.

Images