Information processing system

Abstract

To improve the accuracy of answers from a large-scale language model. An information processing system includes a storage unit that stores evaluations of answers output from a large-scale language model by a user and text data input to the large-scale language model based on a question sentence of the user, and a search unit that searches a database based on a question sentence input by a user and outputs a search result including text data related to the question sentence and a search score indicating a degree of association between the question sentence and the text data, and an adjustment unit that adjusts the search score based on the evaluations associated with the text data stored in the storage unit.

Description

Technical Field

[0001] This invention relates to the technical field of an information processing system. Background Technology

[0002] As such a system, for example, a system has been proposed that enables a language model to generate document-based query data and uses the pairing of documents and query data for learning the search model of a chatbot (see Patent Document 1).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-076413 Summary of the Invention

[0004] As a chatbot, a mechanism called Retrieval-Augmented Generation (RAG) is proposed: a unique information source is assigned to the Large Language Model (LLM) by combining a large-scale language model (LLM) with a search for a specific information source (hereinafter, appropriately referred to as the "knowledge base"). In chatbots using RAG, a unique information source is assigned to the LLM by inputting a portion of the search results from the knowledge base into the LLM. However, the information the user expects is sometimes not input into the LLM. As a result, the responses generated by the LLM sometimes differ from the responses requested by the user. Furthermore, a large-scale language model refers to a language model built using very large datasets and deep learning techniques.

[0005] The present invention was made in view of the above-mentioned problems, and its objective is to provide an information processing system that can improve the response accuracy of large-scale language models.

[0006] An information processing system according to one aspect of the present invention comprises: a storage unit that inputs text data extracted from a database based on a user's question and the question into a large-scale language model, thereby establishing and storing the user's evaluation of the answer output from the large-scale language model and the text data input into the large-scale language model; a search unit that searches the database based on a user-inputted question and outputs search results, the search results including text data related to the question and a search score indicating the degree of association between the question and the text data; and an adjustment unit that adjusts the search score based on the evaluation associated with the text data stored in the storage unit. Attached Figure Description

[0007] Figure 1This is a diagram showing the structure of the information processing system involved in the implementation method.

[0008] Figure 2 This is a diagram representing an example of an image being displayed.

[0009] Figure 3 This is a block diagram illustrating an example of the structure of the computing device involved in the implementation. Detailed Implementation

[0010] refer to Figures 1 to 3 The implementation methods involved in the information processing system are described. Figure 1 In this system, information processing system 1 includes information processing device 10, server 20, knowledge base 30, and database 40. Information processing device 10, server 20, knowledge base 30, and database 40 are configured via a network NW to enable communication between them. Server 20 is a server used for applying Large Scale Language Model (LLM). Therefore, server 20 can be called an LLM server. Alternatively, server 20 can be a cloud server.

[0011] Multiple text data entries can be registered in knowledge base 30. Each text data entry can be fragmented data generated by segmenting a document. This fragmented data can be called a "block". Furthermore, as specific examples of document segmentation methods, methods that segment by a certain length (in other words, fixed length), methods that segment by sentence based on delimiters, and methods that segment based on markup languages ​​(Markdown) and other structures can be used. Each text data entry can be vectorized text data. That is, knowledge base 30 can be a vector database / vector storage.

[0012] (Chatbot)

[0013] Server 20 and knowledge base 30 provide a chatbot service using RAG. For example, user U can utilize the chatbot service via terminal device 50. In this case, user U can input a question via terminal device 50. Here, "question" is not limited to interrogative sentences. For example, a "question" can be a statement containing expressions such as "Please tell me about ****" or "Please answer about ****". Therefore, a "question" is not limited to interrogative sentences, but includes the concept of statements containing expressions such as requests, instructions, and commands. That is, a "question" can refer to a statement requesting an answer from the other party. In addition, terminal device 50 can be a personal computer, tablet terminal, or smartphone.

[0014] Terminal device 50 can search knowledge base 30 based on the input question. Terminal device 50 can send a prompt, including the input question and text data as the search results of knowledge base 30, to server 20. As a result, the prompt is input into a large-scale language model. Server 20 can send the answer to the question generated by the large-scale language model to terminal device 50. Terminal device 50 can display an image representing the answer from the large-scale language model.

[0015] The result can be displayed on terminal device 50. Figure 2 The image 51 shown is an example. For instance, image 51 may include a region 511 displaying the question entered by user U and a region 512 displaying the response from the large-scale language model. In region 512, in addition to the response from the large-scale language model, a systematic evaluation of the response may also be displayed. The systematic evaluation may also not be displayed in region 512. Even in this case, a systematic evaluation may be required. The systematic evaluation may be, for example, at least one of an evaluation related to the appropriateness of the response from the large-scale language model and an evaluation related to the prompts input into the large-scale language model. For example, the systematic evaluation can be obtained using the LLM-as-a-Judge method.

[0016] For example, buttons 513 and 514 could be configured below area 512 for user U to evaluate the response of the large-scale language model. Figure 2 In the example shown, user U's rating is a two-stage rating of "GOOD" and "BAD". However, user U's rating is not limited to two stages; it can be a rating of three or more stages.

[0017] User U can operate terminal device 50 to select button 513 or 514 (in other words, user U can evaluate the answer from the large-scale language model). Terminal device 50 can send feedback information representing user U's evaluation and the text data included in the above prompt (i.e., the text data input into the large-scale language model) to server 20. Upon receiving the feedback information, server 20 can associate user U's evaluation represented by the feedback information and the text data included in the above prompt with the system evaluation related to the answer evaluated by user U (i.e., associate user U's evaluation, system evaluation, and text data) and register it in database 40.

[0018] (Information processing device 10)

[0019] exist Figure 1The information processing device 10 includes a computing unit 11, a storage unit 12, a communication unit 13, an input unit 14, and an output unit 15. The computing unit 11, storage unit 12, communication unit 13, input unit 14, and output unit 15 are connected via a data bus 16. Furthermore, the information processing device 10 can be a personal computer, a tablet computer, or a smartphone.

[0020] The arithmetic unit 11 may have a processor. Furthermore, the arithmetic unit 11 may have a single processor or multiple processors. That is, the arithmetic unit 11 may have more than one processor. Additionally, the processor may be a multi-core processor. In the case where the arithmetic unit 11 has a single processor that functions as a multi-core processor, it can be said that the arithmetic unit 11 logically has multiple processors.

[0021] The processor may be at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), and a Tensor Processing Unit (TPU).

[0022] Storage device 12 may be at least one of random access memory (RAM), read-only memory (ROM), hard disk drive, magneto-optical disk drive, solid-state drive (SSD), and optical disk array. That is, storage device 12 may be implemented by a single device or by multiple devices.

[0023] The communication device 13 can communicate with external devices of the information processing device 10 (e.g., server 20). Furthermore, the communication device 13 can perform wired communication or wireless communication.

[0024] Input device 14 is a device capable of accepting information input from an external source to information processing device 10. Input device 14 may include user-operable devices (e.g., keyboard, mouse, touch panel, etc.) of information processing device 10. Input device 14 may include, for example, a recording medium reading device capable of reading information recorded on a recording medium removable from information processing device 10, such as a Universal Serial Bus (USB) memory. Furthermore, when information is input to information processing device 10 via communication device 13 (in other words, when information processing device 10 obtains information via communication device 13), communication device 13 may function as an input device.

[0025] Output device 15 is a device capable of outputting information to the outside of information processing device 10. Output device 15 may include a display device capable of outputting visual information such as characters and images. Additionally, output device 15 may include a speaker capable of outputting auditory information such as sound. Output device 15 may include a vibration motor capable of outputting tactile information such as vibration. Output device 15 may include a printer. Output device 15 may output information to a recording medium removable from information processing device 10, such as a USB memory. Furthermore, when information processing device 10 outputs information via communication device 13, communication device 13 may function as an output device.

[0026] Storage device 12 is capable of storing desired data. The computer program CP executed by the arithmetic unit 11 can be stored in storage device 12. When the arithmetic unit 11 executes the computer program CP, storage device 12 can temporarily store data temporarily used by the arithmetic unit 11.

[0027] Furthermore, the computer program CP can be recorded on a computer-readable and non-temporary recording medium. In this case, the computer program CP can be stored in the storage device 12 by reading the recording medium using a recording medium reading device (not shown) included in the information processing device 10. Additionally, at least one of optical discs, magnetic media, magneto-optical discs, semiconductor memory, and any medium capable of storing other programs can be used as the recording medium. Furthermore, the computer program CP can also be obtained from an external (not shown) device outside the information processing device 10 via the communication device 13. In other words, the computer program CP can be downloaded from an external device to the storage device 12 of the information processing device 10.

[0028] The arithmetic unit 11 (e.g., a processor) can perform the processing to be performed by the information processing unit 10 together with the storage device 12 storing the computer program CP (in other words, together with the storage device 12 and the computer program CP stored in the storage device 12). For example, the arithmetic unit 11 can implement logical function blocks for performing the processing to be performed by the information processing unit 10 within the arithmetic unit 11 (e.g., within the processor) by executing the computer program CP.

[0029] In order to utilize the aforementioned chatbot service, the arithmetic unit 11 of the information processing device 10 includes a search unit 111, an adjustment unit 112, and a selection unit 113 (see reference). Figure 3 The search unit 111, adjustment unit 112, and selection unit 113 can be implemented as the aforementioned logic function blocks. Alternatively, at least one of the search unit 111, adjustment unit 112, and selection unit 113 can be implemented as a physical processing circuit. At least one of the search unit 111, adjustment unit 112, and selection unit 113 can be implemented as a combination of logic function blocks and physical processing circuits.

[0030] When a user of the information processing device 10 inputs a question via the input device 14, the search unit 111 of the processing unit 11 searches the knowledge base 30 based on the question. For example, the search unit 111 can calculate a search score indicating the degree of relevance between the question and text data registered in the knowledge base 30. The search unit 111 can extract text data with a search score of a predetermined value or higher as text data related to the question. The search unit 111 can output search results including the extracted text data and its search score. Furthermore, the search unit 111 can extract multiple pieces of text data related to the question. That is, the search unit 111 can extract more than one piece of text data related to the question from the knowledge base 30. In addition, the method for calculating the search score can be applied to various existing methods. Therefore, a detailed description of the method for calculating the search score is omitted.

[0031] The adjustment unit 112 of the computing device 11 can determine whether the text data included in the search results is registered in the database 40. If the text data included in the search results is registered in the database 40, the adjustment unit 112 retrieves the evaluation (i.e., at least one of the user U's evaluation and the system evaluation) associated with the text data (i.e., the text data included in the search results) from the database 40. The adjustment unit 112 adjusts the search score based on the retrieved evaluation. For example, if the retrieved evaluation is high, the adjustment unit 112 can adjust to increase the search score. If the retrieved evaluation is low, the adjustment unit 112 can adjust to decrease the search score.

[0032] The selection unit 113 of the computing device 11 selects text data to be input into the large-scale language model from the text data included in the search results based on the adjusted search scores. For example, the selection unit 113 may select a predetermined number of text data with relatively high search scores (e.g., the top 5 text data with the highest search scores) as the text data to be input into the large-scale language model.

[0033] The processing unit 11 can send a prompt, including a question and text data selected by the selection unit 113, to the server 20 via the communication device 13. As a result, the prompt is input into a large-scale language model. The server 20 can send the answer to the question generated by the large-scale language model to the information processing unit 10. The processing unit 11 of the information processing unit 10 can control the output device 15 to display the answer from the large-scale language model.

[0034] (Technical effect)

[0035] In this embodiment, the search score associated with the text data is adjusted based on past evaluations that are linked to the text data. Here, it can be said that text data associated with past evaluations that are highly rated is text data that helps generate highly rated answers (e.g., answers suitable for the user's question) based on a large-scale language model. Therefore, if text data associated with past evaluations that are highly rated is used, it is expected that the large-scale language model will generate the answer requested by the user. In this embodiment, as described above, the search score is adjusted, thus increasing the likelihood that text data associated with past evaluations that are highly rated will be input into the large-scale language model. That is, according to this embodiment, text data used to enable the large-scale language model to generate answers suitable for the question can be input into the large-scale language model. As a result, the answer accuracy of the large-scale language model can be improved according to the information processing system 1 according to this embodiment.

[0036] Hereinafter, various aspects of the invention derived from the embodiments described above will be described.

[0037] One aspect of the invention relates to an information processing system comprising: a storage unit that extracts text data from a database based on a user's question and inputs the question into a large-scale language model, thereby establishing and storing the user's evaluation of the answer output from the large-scale language model in association with the text data input into the large-scale language model; a search unit that searches the database based on a user-inputted question and outputs search results, the search results including text data related to the question and a search score indicating the degree of association between the question and the text data; and an adjustment unit that adjusts the search score based on the evaluation associated with the text data stored in the storage unit.

[0038] In the above embodiments, "knowledge base 30" is equivalent to "database" in one example, "database 40" is equivalent to "storage unit" in another example, "search unit 111" is equivalent to "search unit" in another example, and "adjustment unit 112" is equivalent to "adjustment unit" in another example.

[0039] The information processing system described above may include a selection unit that selects text data to be input into the large-scale language model from the text data included in the search results based on the adjusted search score. In the above embodiment, "selection unit 113" is an example of a "selection unit".

[0040] This invention is not limited to the embodiments described above, and appropriate modifications can be made without departing from the spirit or concept of the invention as read in its entirety from the claims and description. Information processing systems that accompany such modifications are also included within the technical scope of this invention.

[0041] Symbol Explanation

[0042] 1-Information processing system, 10-Information processing device, 20-Server, 30-Knowledge base, 40-Database, 111-Search unit, 112-Adjustment unit, 113-Selection unit.

Claims

1. An information processing system, characterized in that, have: A storage unit that extracts text data from a database based on a user's question and inputs the question into a large-scale language model, thereby establishing and storing the user's evaluation of the answer output from the large-scale language model and the text data input into the large-scale language model; The search unit searches the database based on a question input by a user and outputs search results, which include text data related to the question and a search score indicating the degree of correlation between the question and the text data. and An adjustment unit adjusts the search score based on the evaluation associated with the text data stored in the storage unit.

2. The information processing system according to claim 1, characterized in that, have: The selection unit selects text data to be input into the large-scale language model from the text data included in the search results, based on the adjusted search score.

Images