Information processing system

Abstract

To improve the response accuracy of large-scale language models. [Solution] The information processing system (1) includes a storage means (40) that stores the user's evaluation of the answer output from the large-scale language model and the text data input to the large-scale language model in association with each other, when the text data extracted from the database (30) based on the user's question and the question are input to the large-scale language model; a search means (111) that searches the database based on a question entered by a user and outputs search results including text data related to the question and a search score indicating the degree of association between the question and the text data; and an adjustment means (112) that adjusts the search score based on the evaluation associated with the text data stored in the storage means.

Description

Technical Field

[0001] The present invention relates to the technical field of information processing systems.

Background Art

[0002] As a system of this kind, for example, a system has been proposed in which a query data based on a document is generated in a language model, and a pair of the document and the query data is used for learning a search model for a chatbot (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As a chatbot, by combining a large language model (Large Language Models: LLM) and the search of a specific information source (hereinafter, appropriately referred to as "knowledge base"), a chatbot using a mechanism (Retrieval-Augmented Genration: RAG) that gives an independent information source to the large language model has been proposed. In a chatbot using RAG, an independent information source may be given to the large language model by inputting a part of the search results of the knowledge base into the large language model. At this time, information expected by the user may not be input into the large language model. As a result, the answer generated by the large language model may be different from the answer required by the user. Note that a large language model is a language model constructed using a very large dataset and deep learning technology.

[0005] This invention has been made in view of the above-mentioned problems, and aims to provide an information processing system that can improve the response accuracy of large-scale language models. [Means for solving the problem]

[0006] An information processing system according to one aspect of the present invention includes: a storage means that stores text data extracted from a database based on a user's question, and the question itself, in association with the user's evaluation of the answer output from the large-scale language model and the text data input to the large-scale language model; a search means that searches the database based on a question entered by a user and outputs search results including text data related to the question and a search score indicating the degree of association between the question and the text data; and an adjustment means that adjusts the search score based on the evaluation associated with the text data stored in the storage means. [Brief explanation of the drawing]

[0007] [Figure 1] This is a diagram showing the configuration of an information processing system according to an embodiment. [Figure 2] This figure shows an example of a displayed image. [Figure 3] This block diagram shows an example of the configuration of a computing device according to the embodiment. [Modes for carrying out the invention]

[0008] Embodiments of the information processing system will be described with reference to Figures 1 to 3. In Figure 1, the information processing system 1 comprises an information processing device 10, a server 20, a knowledge base 30, and a database 40. The information processing device 10, server 20, knowledge base 30, and database 40 are configured to communicate with each other via a network NW. Server 20 is a server for operating a large-scale language model (LLM). For this reason, server 20 may be referred to as an LLM server. Server 20 may be a cloud server.

[0009] Knowledge Base 30 may contain multiple text data entries. Each text data entry may be fragmented data generated by splitting a single document. Such fragmented data may be referred to as "chunks." Specific examples of methods for splitting a single document include splitting at fixed lengths, splitting at sentence units based on sentence delimiters, and splitting based on structure such as Markdown. Each text data entry may be vectorized text data. In other words, Knowledge Base 30 may be a vector database / vector store.

[0010] (Chatbot) Server 20 and Knowledge Base 30 provide a chatbot service using RAG. For example, user U may use the chatbot service via terminal device 50. In this case, user U may input a question via terminal device 50. Here, "question" is not limited to interrogative sentences. For example, "question" may be a sentence that includes expressions such as requests, instructions, or commands, such as "Tell me about ****" or "Answer me about ****". Therefore, "question" is a concept that includes not only sentences in the form of interrogative sentences, but also sentences that include expressions such as requests, instructions, or commands. In other words, "question" may mean a sentence that seeks an answer from the other party. Note that terminal device 50 may be a personal computer, a tablet terminal, or a smartphone.

[0011] Terminal device 50 may search the knowledge base 30 based on the input question. Terminal device 50 may send a prompt to server 20 that includes the input question and text data as search results from the knowledge base 30. As a result, the prompt is input to the large-scale language model. Server 20 may send the answer to the question generated by the large-scale language model to terminal device 50. Terminal device 50 may display an image showing the answer from the large-scale language model.

[0012] As a result, the terminal device 50 may display the image 51 shown in Figure 2. For example, the image 51 may include an area 511 that displays the question entered by user U and an area 512 that displays the answer from the large-scale language model. Area 512 may display the answer from the large-scale language model, as well as a system evaluation of the answer from the large-scale language model. However, the system evaluation does not have to be displayed in area 512. Even in this case, the system evaluation may be obtained. The system evaluation may be, for example, an evaluation of the validity of the answer from the large-scale language model, and an evaluation of the prompts entered into the large-scale language model, at least one of the above. For example, the system evaluation may be obtained by a method called LLM-as-a-Judge.

[0013] For example, below area 512, buttons 513 and 513 may be placed for inputting user U's evaluation of the large-scale language model's response. In the example shown in Figure 2, user U's evaluation is a two-level rating of "GOOD" and "BAD". However, user U's evaluation is not limited to a two-level rating; it may be a three-level or higher rating.

[0014] User U may operate the terminal device 50 to select button 513 or 514 (in other words, User U may evaluate the response of the large-scale language model). The terminal device 50 may send feedback information to the server 20 indicating User U's evaluation and the text data included in the prompt (i.e., the text data input to the large-scale language model). Upon receiving the feedback information, the server 20 may associate User U's evaluation indicated by the feedback information and the text data included in the prompt with the system evaluation related to the response evaluated by User U (i.e., associate User U's evaluation with the system evaluation with the text data) and register this information in the database 40.

[0015] (Information processing device 10) In Figure 1, the information processing device 10 comprises an arithmetic unit 11, a storage device 12, a communication device 13, an input device 14, and an output device 15. The arithmetic unit 11, storage device 12, communication device 13, input device 14, and output device 15 are connected via a data bus 16. The information processing device 10 may be a personal computer, a tablet terminal, or a smartphone.

[0016] The arithmetic unit 11 may have a processor. The arithmetic unit 11 may have a single processor or multiple processors. In other words, the arithmetic unit 11 may have one or more processors. Furthermore, the processor may be a multi-core processor. If the arithmetic unit 11 has a single processor that is a multi-core processor, then logically, the arithmetic unit 11 can be said to have multiple processors.

[0017] The processor may be at least one of the following: CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and TPU (Tensor Processing Unit).

[0018] The storage device 12 may be, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, a magneto-optical disk drive, a SSD (Solid State Drive), and an optical disk array. That is, the storage device 12 may be realized by a single device or by a plurality of devices.

[0019] The communication device 13 may be capable of communicating with a device external to the information processing device 10 (for example, the server 20). Note that the communication device 13 may perform wired communication or wireless communication.

[0020] The input device 14 is a device capable of receiving input of information to the information processing device 10 from the outside. The input device 14 may include an operating device (for example, a keyboard, a mouse, a touch panel, etc.) that can be operated by a user of the information processing device 10. The input device 14 may include a recording medium reading device capable of reading information recorded on a recording medium detachable from the information processing device 10, such as a USB (Universal Serial Bus) memory. Note that when information is input to the information processing device 10 via the communication device 13 (in other words, when the information processing device 10 acquires information via the communication device 13), the communication device 13 may function as an input device.

[0021] The output device 15 is a device capable of outputting information to the outside of the information processing device 10. The output device 15 may have a display device capable of outputting visual information such as characters and images as the above information. Note that the output device 15 may have a speaker capable of outputting auditory information such as sound as the above information. The output device 15 may have a vibration motor capable of outputting tactile information such as vibration as the above information. The output device 15 may have a printer. The output device 15 may be capable of outputting information to a recording medium detachable from the information processing device 10, such as a USB memory. Note that when the information processing device 10 outputs information via the communication device 13, the communication device 13 may function as an output device.

[0022] The storage device 12 can store desired data. The storage device 12 may store the computer program CP executed by the arithmetic unit 11. When the arithmetic unit 11 is executing the computer program CP, the storage device 12 may temporarily store data temporarily used by the arithmetic unit 11.

[0023] In addition, the computer program CP may be recorded on a computer-readable and non-temporary recording medium. In this case, the computer program CP may be stored in the storage device 12 by reading the recording medium using a recording medium reader (not shown) provided in the information processing apparatus 10. As the recording medium, at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program may be used. In addition, the computer program CP may be acquired from a device (not shown) outside the information processing apparatus 10 via the communication device 13. In other words, the computer program CP may be downloaded from an external device to the storage device 12 of the information processing apparatus 10.

[0024] The arithmetic unit 11 (for example, a processor) may execute the processing to be performed by the information processing apparatus 10 together with the storage device 12 in which the computer program CP is stored (in other words, together with the storage device 12 and the computer program CP stored in the storage device 12). For example, by executing the computer program CP, the arithmetic unit 11 (for example, within the processor) may realize a logical functional block for executing the processing to be performed by the information processing apparatus 10.

[0025] The arithmetic unit 11 of the information processing device 10 has a search unit 111, an adjustment unit 112, and a selection unit 113 for using the chatbot service described above (see Figure 3). The search unit 111, adjustment unit 112, and selection unit 113 may be implemented as the logical functional blocks described above. At least one of the search unit 111, adjustment unit 112, and selection unit 113 may be implemented as a physical processing circuit. At least one of the search unit 111, adjustment unit 112, and selection unit 113 may be implemented in a form that combines a logical functional block and a physical processing circuit.

[0026] When a user of the information processing device 10 inputs a question via the input device 14, the search unit 111 of the arithmetic unit 11 searches the knowledge base 30 based on the question. For example, the search unit 111 may calculate a search score indicating the degree of relevance between the question and the text data registered in the knowledge base 30. The search unit 111 may extract text data with a search score of a predetermined value or higher as text data related to the question. The search unit 111 may output search results including the extracted text data and its search score. The search unit 111 may extract multiple text data related to the question. In other words, the search unit 111 may extract one or more text data related to the question from the knowledge base 30. Various existing methods can be applied to the calculation method of the search score. Therefore, a detailed explanation of the calculation method of the search score is omitted.

[0027] The adjustment unit 112 of the arithmetic unit 11 may determine whether or not the text data included in the search results is registered in the database 40. If the text data included in the search results is registered in the database 40, the adjustment unit 112 obtains the evaluation (i.e., at least one of the evaluation of user U and the system evaluation) associated with the text data (i.e., the text data included in the search results) from the database 40. The adjustment unit 112 adjusts the search score based on the obtained evaluation. For example, if the obtained evaluation is high, the adjustment unit 112 may adjust the search score to increase. If the obtained evaluation is low, the adjustment unit 112 may adjust the search score to decrease.

[0028] The selection unit 113 of the arithmetic unit 11 selects text data to be input into the large-scale language model from the text data included in the search results, based on the adjusted search score. For example, the selection unit 113 may select a predetermined number of text data with relatively high search scores (for example, the top 5 text data with high search scores) as text data to be input into the large-scale language model.

[0029] The arithmetic unit 11 may send a prompt to the server 20 via the communication device 13, which includes the question and the text data selected by the selection unit 113. As a result, the prompt is input to the large-scale language model. The server 20 may send the answer to the question generated by the large-scale language model to the information processing device 10. The arithmetic unit 11 of the information processing device 10 may control the output device 15 to display the answer from the large-scale language model.

[0030] (Technical effects) In this embodiment, the search score for the currently retrieved text data is adjusted based on past evaluations associated with the text data. Here, text data associated with high-rated past evaluations can be said to be text data that contributed to the generation of high-rated answers (for example, answers suitable for the user's question) by the large-scale language model. Therefore, if text data associated with high-rated past evaluations is used, it can be expected that the large-scale language model will generate the answer the user is looking for. In this embodiment, since the search score is adjusted as described above, the likelihood of text data associated with high-rated past evaluations being input into the large-scale language model increases. In other words, according to this embodiment, the large-scale language model can be input text data that is more suitable for generating answers to questions. As a result, the information processing system 1 according to this embodiment can improve the answer accuracy of the large-scale language model.

[0031] Various aspects of the invention derived from the embodiments described above are described below.

[0032] An information processing system according to one aspect of the invention includes: a storage means that stores text data extracted from a database based on a user's question, and the question itself, in association with the user's evaluation of the answer output from the large-scale language model and the text data input to the large-scale language model; a search means that searches the database based on a question entered by a user and outputs search results including text data related to the question and a search score indicating the degree of association between the question and the text data; and an adjustment means that adjusts the search score based on the evaluation associated with the text data stored in the storage means.

[0033] In the above-described embodiment, "Knowledge Base 30" corresponds to an example of a "database," "Database 40" corresponds to an example of a "storage means," "Search Unit 111" corresponds to an example of a "search means," and "Adjustment Unit 112" corresponds to an example of an "adjustment means."

[0034] The information processing system according to the above embodiment may include selection means for selecting text data to be input into the large-scale language model from the text data included in the search results, based on the adjusted search score. In the above embodiment, the "selection unit 113" corresponds to an example of the "selection means".

[0035] The present invention is not limited to the embodiments described above, and can be modified as appropriate without contradicting the gist or idea of ​​the invention as can be read from the claims and specification as a whole. Information processing systems involving such modifications are also included within the technical scope of the present invention. [Explanation of Symbols]

[0036] 1... Information processing system, 10... Information processing device, 20... Server, 30... Knowledge base, 40... Database, 111... Search unit, 112... Adjustment unit, 113... Selection unit

Claims

1. A storage means that stores the user's evaluation of the answer output by the large-scale language model and the text data input into the large-scale language model in relation to each other, by inputting the text data extracted from the database based on the user's question and the question into a large-scale language model. A search means that searches the database based on a question entered by a user and outputs search results including text data related to the question and a search score indicating the degree of relationship between the question and the text data. An adjustment means for adjusting the search score based on the evaluation associated with the text data stored in the storage means, An information processing system equipped with the following features.

2. The system includes a selection means for selecting text data to be input into the large-scale language model from the text data included in the search results, based on the adjusted search score. The information processing system according to claim 1.

Images