Information processing system

Abstract

The objective of this invention is to improve the search accuracy of a knowledge base. An information processing system comprises: a storage unit that stores user evaluations of answers in association with user questions, wherein the answers are input into a large-scale language model and output from the large-scale language model by inputting one or more text data extracted from a database based on the user questions and the user questions; an acquisition unit that acquires a question generated by the large-scale language model based on newly registered text data in the database; an extraction unit that extracts user questions similar to the generated question from the storage unit; and a determination unit that, if the evaluation associated with the extracted user questions is a predetermined evaluation, determines that a text data should not be registered in the database.

Description

Technical Field

[0001] This invention relates to the technical field of an information processing system. Background Technology

[0002] As such a system, for example, a system has been proposed that enables a language model to generate document-based query data and uses the pair of documents and query data for learning a search model for a chatbot (see Patent Document 1).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-076413 Summary of the Invention

[0004] As a chatbot, a mechanism called Retrieval-Augmented Generation (RAG) is proposed: This involves combining a Large Language Model (LLM) with a search for a specific information source (hereinafter, appropriately referred to as the "knowledge base") to give the LLM a unique information source. Here, the knowledge base includes multiple data sets (e.g., documents). For example, a knowledge base may sometimes include a single data set and other data sets whose portions of that data set have been updated. For example, a knowledge base may sometimes include multiple data sets with identical or nearly identical content. In such cases, the search accuracy of the knowledge base may decrease. Furthermore, a Large Language Model refers to a language model constructed using a very large dataset and deep learning techniques.

[0005] The present invention was made in view of the above-mentioned problems, and its objective is to provide an information processing system that can improve the search accuracy of a knowledge base.

[0006] An information processing system according to one aspect of the present invention comprises: a storage unit that stores user evaluations of answers in association with user question texts, wherein the answers are output from a large-scale language model by inputting one or more text data extracted from a database based on the user question texts and the user question texts; an acquisition unit that acquires a question text generated by the large-scale language model based on a text data newly registered in the database; an extraction unit that extracts user question texts similar to the generated question texts from the storage unit; and a determination unit that, if the evaluation associated with the extracted user question texts is a specified evaluation, determines that the text data should not be registered in the database. Attached Figure Description

[0007] Figure 1 This is a diagram showing the structure of the information processing system involved in the implementation method.

[0008] Figure 2 This is an example of a scene related to a chatbot.

[0009] Figure 3 This is a block diagram illustrating an example of the structure of the computing device involved in the implementation.

[0010] Figure 4 It is a flowchart illustrating the operation of the information processing system involved in the implementation method. Detailed Implementation

[0011] refer to Figures 1 to 4 The implementation methods related to the information processing system are described. Figure 1 In this system, information processing system 1 includes information processing device 10, server 20, knowledge base 30, and database 40. Information processing device 10, server 20, knowledge base 30, and database 40 are configured via a network NW to enable communication between them. Server 20 is a server used for applying Large Scale Language Model (LLM). Therefore, server 20 can be called an LLM server. Alternatively, server 20 can be a cloud server.

[0012] (Chatbot)

[0013] Server 20 and knowledge base 30 can provide chatbot services using RAG. For example, user U can utilize the chatbot service via terminal device 50. In this case, user U can operate terminal device 50 to launch an application for utilizing the chatbot service. User U can operate terminal device 50 to enter a question in the input field of the chat application. Here, "question" is not limited to interrogative sentences. For example, "question" can be a sentence containing expressions of entrustment, instruction, or command, such as "Please tell me about ****" or "Please answer about ****". Therefore, "question" is not limited to interrogative sentences, but includes the concept of sentences containing expressions of entrustment, instruction, or command. That is, "question" can refer to a sentence requesting an answer from the other party.

[0014] Terminal device 50 can search knowledge base 30 based on the input query. Terminal device 50 can send first information, including the input query and text data representing the search results from knowledge base 30, to server 20. Server 20 can input the query and text data included in the first information as prompts into a large-scale language model. Server 20 can obtain the answer to the query from the output of the large-scale language model. Server 20 can send second information representing the answer to terminal device 50. Upon receiving the second information, terminal device 50 can display the answer represented by the second information on a screen related to the chat application.

[0015] The result can be displayed on terminal device 50. Figure 2 The screen 51 shown. For example, screen 51 may include an area 511 displaying the question entered by user U and an area 512 displaying the answer from the large-scale language model. For example, buttons 513 and 514 for user U to input their evaluation of the large-scale language model's answer may be configured below area 512. Figure 2 In the example shown, user U's rating is a two-stage evaluation: "GOOD" and "BAD". However, user U's rating is not limited to two stages; it can be three or more stages.

[0016] User U can operate terminal device 50 to select button 513 or 514 (in other words, user U can evaluate the answer of the large-scale language model). Terminal device 50 can display the user U's evaluation and the question text related to the answer evaluated by user U (e.g., shown on the screen). Figure 2 The third information (the question text in region 511) is sent to server 20. Upon receiving the third information, server 20 can associate the user U's evaluation included in the third information with the question text (i.e., user U's question text) and register it in database 40. Additionally, terminal device 50 can be a personal computer, tablet terminal, or smartphone.

[0017] (Information processing device 10)

[0018] exist Figure 1 The information processing device 10 includes a computing unit 11, a storage unit 12, a communication unit 13, an input unit 14, and an output unit 15. The computing unit 11, storage unit 12, communication unit 13, input unit 14, and output unit 15 are connected via a data bus 16. Furthermore, the information processing device 10 can be a personal computer, a tablet terminal, or a smartphone.

[0019] The arithmetic unit 11 may have a processor. Furthermore, the arithmetic unit 11 may have a single processor or multiple processors. That is, the arithmetic unit 11 may have more than one processor. Additionally, the processor may be a multi-core processor. In the case where the arithmetic unit 11 has a single processor that functions as a multi-core processor, it can be said that the arithmetic unit 11 logically has multiple processors.

[0020] The processor may be at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), and a Tensor Processing Unit (TPU).

[0021] Storage device 12 may be at least one of random access memory (RAM), read-only memory (ROM), hard disk drive, magneto-optical disk drive, solid-state drive (SSD), and optical disk array. That is, storage device 12 may be implemented by a single device or by multiple devices.

[0022] The communication device 13 can communicate with external devices of the information processing device 10. In addition, the communication device 13 can perform wired communication or wireless communication.

[0023] Input device 14 is a device capable of accepting information input from an external source to information processing device 10. Input device 14 may include user-operable devices (e.g., keyboard, mouse, touch panel, etc.) of information processing device 10. Input device 14 may include, for example, a recording medium reading device capable of reading information recorded on a recording medium removable from information processing device 10, such as a Universal Serial Bus (USB) memory. Furthermore, when information is input to information processing device 10 via communication device 13 (in other words, when information processing device 10 obtains information via communication device 13), communication device 13 may function as an input device.

[0024] Output device 15 is a device capable of outputting information to the outside of information processing device 10. Output device 15 may include a display device capable of outputting visual information such as characters and images. Additionally, output device 15 may include a speaker capable of outputting auditory information such as sound. Output device 15 may include a vibration motor capable of outputting tactile information such as vibration. Output device 15 may include a printer. Output device 15 may output information to a recording medium removable from information processing device 10, such as a USB memory. Furthermore, when information processing device 10 outputs information via communication device 13, communication device 13 may function as an output device.

[0025] Storage device 12 is capable of storing desired data. The computer program CP executed by the arithmetic unit 11 can be stored in storage device 12. When the arithmetic unit 11 executes the computer program CP, storage device 12 can temporarily store data temporarily used by the arithmetic unit 11.

[0026] Furthermore, the computer program CP can be recorded on a computer-readable and non-temporary recording medium. In this case, the computer program CP can be stored in the storage device 12 by reading the recording medium using a recording medium reading device (not shown) included in the information processing device 10. Additionally, at least one of optical discs, magnetic media, magneto-optical discs, semiconductor memory, and any medium capable of storing other programs can be used as the recording medium. Furthermore, the computer program CP can also be obtained from an external (not shown) device outside the information processing device 10 via the communication device 13. In other words, the computer program CP can be downloaded from an external device to the storage device 12 of the information processing device 10.

[0027] The arithmetic unit 11 (e.g., a processor) can perform the processing that the information processing unit 10 should perform together with the storage unit 12 storing the computer program CP (in other words, together with the storage unit 12 and the computer program CP stored in the storage unit 12).

[0028] For example, the arithmetic unit 11 can implement logical function blocks for performing the processing to be performed by the information processing unit 10 by executing a computer program CP.

[0029] (Knowledge Base 30)

[0030] Multiple text data can be registered in the knowledge base 30. This text data can be data segmented from the text contained in a document. This type of data can be called a "block". Furthermore, methods for segmenting the text contained in a document can include, for example, segmenting by a certain length (in other words, fixed length), segmenting by sentence based on sentence delimiters, and segmenting based on structures such as Markdown. Additionally, the knowledge base 30 can vectorize and register multiple text data separately. That is, the knowledge base 30 can be a vector database / vector storage. Besides text data, image data can also be registered in the knowledge base 30.

[0031] Based on the inventors' research, the following is known: New text data can be registered in the knowledge base 30 at any time. On the other hand, it is possible that multiple pieces of text data with identical or nearly identical content are registered in the knowledge base 30, or that both the original and updated text data are registered. Furthermore, when searching the knowledge base 30, it is possible to extract two or more pieces of text data with duplicate content, or to extract both the original and updated text data. As a result, the search accuracy of the knowledge base 30 may be reduced. That is, in the aforementioned chatbot service, the response accuracy of the large-scale language model may be reduced.

[0032] Therefore, when new text data is registered in the knowledge base 30, the information processing apparatus 10 according to this embodiment determines whether to register the text data in the knowledge base 30. For example... Figure 3 As shown, the arithmetic unit 11 of the information processing apparatus 10 includes an acquisition unit 111, an extraction unit 112, and a determination unit 113 for performing this determination. The acquisition unit 111, extraction unit 112, and determination unit 113 can be implemented as the aforementioned logic function blocks. At least one of the acquisition unit 111, extraction unit 112, and determination unit 113 can be implemented as a physical processing circuit. Alternatively, at least one of the acquisition unit 111, extraction unit 112, and determination unit 113 can be implemented in a manner that combines logic function blocks and physical processing circuits.

[0033] Regarding the above judgment, in addition to reference Figure 1 and Figure 3 In addition, also refer to Figure 4 The flowchart is used for illustration. Figure 4 In step S101, the acquisition unit 111 of the information processing device 10 acquires new text data (i.e., text data newly registered in the knowledge base 30). The processing unit 11 of the information processing device 10 sends the new text data to the server 20 via the communication device 13. At this time, the processing unit 11 also sends information (e.g., a prompt) used to enable the large-scale language model to generate a question based on the new text data to the server 20. As a result, the large-scale language model generates a question based on the new text data. For example, if the new text data is "Asakusa is a popular tourist attraction for foreigners.", the large-scale language model can generate the question "What are some popular tourist attractions in Tokyo for foreigners?".

[0034] Server 20 sends the query text generated by the large-scale language model to information processing device 10. As a result, acquisition unit 111 of information processing device 10 acquires the query text generated by the large-scale language model (step S102). Next, extraction unit 112 of information processing device 10 extracts query texts from database 40 from users (e.g., user U) that are similar to the query text generated by the large-scale language model (step S103). In step S103, extraction unit 112 may extract query texts from database 40 from users similar to the query text generated by the large-scale language model based on the similarity between the query text generated by the large-scale language model and the user's query text. Alternatively, extraction unit 112 may extract query texts from database 40 from users similar to the query text generated by the large-scale language model based on the distance between vectors related to the query text generated by the large-scale language model and vectors related to the user's query text.

[0035] Next, the extraction unit 112 (or the acquisition unit 111) retrieves user evaluations from the database 40 that are associated with the user's question text extracted in step S103 (step S104). Based on the user evaluations obtained in step S104, the determination unit 113 of the information processing device 10 determines whether to register new text data into the knowledge base 30 (step S105).

[0036] like Figure 2 In the example shown, when the user's evaluation (e.g., user U) has two stages of evaluation, namely "good" and "bad", the determination unit 113 can make the following determination. If the user's evaluation obtained in step S104 is "bad", the determination unit 113 can determine to register the new text data into the knowledge base 30. On the other hand, if the user's evaluation obtained in step S104 is "good", the determination unit 113 can determine not to register the new text data into the knowledge base 30.

[0037] As described above, the query text generated by the large-scale language model is based on new text data (i.e., text data newly registered in the knowledge base 30). Furthermore, the user's query text extracted in step S103 is similar to the query text generated by the large-scale language model. Therefore, assuming that new text data is registered in the knowledge base 30, and the user's query text extracted in step S103 is the same as or similar to the query text (in other words, a prompt) input into the large-scale language model, the new text data may also be input into the large-scale language model as at least part of the search results of the knowledge base 30.

[0038] Here, if the user's evaluation obtained in step S104 is "good," it can be said that even if new text data is not registered (or added) to the knowledge base 30, a satisfactory answer (in other words, a relatively accurate answer) can be obtained from the large-scale language model. On the other hand, if the user's evaluation is "bad," it can be said that the large-scale language model cannot generate a satisfactory answer for the user in the current knowledge base 30. Therefore, the determination unit 113 can determine not to register new text data when the user's evaluation is "good," and determine to register new text data when the user's evaluation is "bad."

[0039] Furthermore, in step S103, multiple user queries can be extracted. In this case, the determination unit 113 can determine whether to register new text data into the knowledge base 30 based on the user ratings associated with each user's query. For example, the determination unit 113 can use "good" as "+1" and "bad" as "-1" to calculate the average rating based on the multiple user ratings. If the average rating is less than a threshold, the determination unit 113 can determine whether to register new text data into the knowledge base 30. Additionally, if a user's rating has three or more stages, the determination unit 113 can determine whether to register new text data into the knowledge base 30 based on the numerical value corresponding to the user's rating and a threshold.

[0040] If, in step S105, it is determined that new text data should be registered in the knowledge base 30 (step S105: Yes), the processing unit 11 registers the new text data in the knowledge base 30 (step S106). If, in step S105, it is determined that new text data should not be registered in the knowledge base 30 (step S105: No), the process can end. Figure 4 The actions shown.

[0041] (Technical effect)

[0042] According to the information processing system 1 of this embodiment, it is possible to suppress the registration of multiple text data with identical or nearly identical content, or the registration of text data before and after an update. Therefore, according to the information processing system 1, the search accuracy of the knowledge base 30 can be improved. In addition, the utilization cost of the storage device constituting the knowledge base 30 can be reduced.

[0043] (Modified example)

[0044] In database 40, information such as the user's question, text data representing the search results of knowledge base 30 based on that question, and the user's evaluation of the large-scale language model's answer can be registered in a mutually associated manner. If the user's evaluation obtained in step S104 is "good," the determination unit 113 of the information processing device 10 can compare new text data with the text data associated with the user's evaluation (i.e., the text data representing the search results of knowledge base 30). Based on the comparison result, the determination unit 113 can determine whether to register new text data.

[0045] For example, the new text data can be text data related to a rule. If the rule that serves as the new text data is a revised version of a rule that establishes a connection between the new text data and the user's rating, the determination unit 113 can determine to register the new text data and to delete the text data associated with the user's rating from the knowledge base 30. That is, the determination unit 113 can determine to replace the existing text data with the new text data. This configuration allows the latest data to be reflected in the knowledge base 30.

[0046] The following describes various aspects of the invention derived from the embodiments and variations described above.

[0047] One aspect of the invention relates to an information processing system comprising: a storage unit that stores user evaluations of answers in association with user question texts, wherein the answers are output from a large-scale language model by inputting one or more text data extracted from a database based on the user question texts and the user question texts; an acquisition unit that acquires a question text generated by the large-scale language model based on a newly registered text data in the database; an extraction unit that extracts user question texts similar to the generated question texts from the storage unit; and a determination unit that, if the evaluation associated with the extracted user question texts is a specified evaluation, determines that the text data should not be registered in the database. In the above embodiment, "knowledge base 30" is equivalent to "database" in one example, "database 40" is equivalent to "storage unit" in another example, "acquisition unit 111" is equivalent to "acquisition unit" in another example, "extraction unit 112" is equivalent to "extraction unit" in another example, and "determination unit 113" is equivalent to "determination unit" in another example.

[0048] In the information processing system described above, the extraction unit can extract user questions similar to the generated question text based on the similarity between the generated question text and the user question text. Alternatively, the extraction unit can extract user questions similar to the generated question text based on the distance between vectors related to the generated question text and vectors related to the user question text. Therefore, it is relatively easy to extract user questions similar to those generated by a large-scale language model.

[0049] This invention is not limited to the embodiments described above, and appropriate modifications can be made without departing from the spirit or concept of the invention as read in its entirety from the claims and description. Information processing systems that accompany such modifications are also included within the technical scope of this invention.

[0050] Symbol Explanation

[0051] 1-Information processing system, 10-Information processing device, 20-Server, 30-Knowledge base, 40-Database, 111-Acquisition unit, 112-Extraction unit, 113-Judgment unit.

Claims

1. An information processing system, characterized in that, have: A storage unit that stores user evaluations of answers in association with the user question text, wherein the answer is output from the large-scale language model by inputting one or more text data extracted from the database based on the user question text and the user question text into the large-scale language model; The acquisition unit acquires the query text generated by the large-scale language model based on text data newly registered in the database; An extraction unit extracts user question texts that are similar to the generated question texts from the storage unit; and The determination unit determines that if the evaluation associated with the extracted user question is a specified evaluation, it will not register the text data in the database.

2. The information processing system according to claim 1, characterized in that, The extraction unit extracts user question texts that are similar to the generated question text based on the similarity between the generated question text and the user question text.

3. The information processing system according to claim 1, characterized in that, The extraction unit extracts user question texts similar to the generated question text based on the distance between the vectors related to the generated question text and the vectors related to the user question text.

Images