Language model utilization system and language model utilization method

Abstract

The present invention makes it possible to appropriately perform prompt engineering even when accumulated confidential information or the like is included in knowledge external to an LLM. Provided is a language model utilization system 1 in which a sentence of a prompt 6 designated by a user 5 is input to a language model 4, and an answer 7 obtained from the language model 4 is output to the user 5. The system includes a knowledge holding unit 3 in which information pertaining to knowledge is accumulated. Knowledge related to or similar to the prompt 6 is acquired from the knowledge holding unit 3 on the basis of the prompt 6, and the knowledge and the prompt 6 are both input together to the language model 4 to obtain the answer 7.

Description

Language model utilization system and language model utilization method

[0001] The present invention relates to technology for utilizing generative AI and LLMs (Large Language Models), and in particular to technology that is effective when applied to a language model utilization system and language model utilization method that perform so-called prompt engineering.

[0002] Recently, LLMs, such as ChatGPT (registered trademark), have become available that demonstrate incredible conversational and comprehension abilities. However, if you want an LLM to perform highly specialized conversations or tasks (e.g., summarizing text, creating Q&As, generating text, etc.), it is necessary to learn downstream tasks specific to the target field or domain. Therefore, optimal additional learning is essential for utilizing an LLM in business.

[0003] There are multiple methods for additional training of LLMs, but the methods that can be used depend on the model disclosure method of each LLM. For example, if the model itself is publicly available and commercially available, various additional training methods can be used, but if only the API (Application Programming Interface) is publicly available, the methods that can be used are limited to updating parameters that can be performed or accessed through the API.

[0004] When architectures such as RNN (Recurrent Neural Network) and LSTM (Long Short Term Memory) were mainstream in natural language models, it was common to retrain them from scratch for each downstream task or domain. Later, with the emergence of machine learning models based on so-called Transformer-based natural language processing models such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pretrained Transformer), and BART (Bidirectional Auto-Regressive Transformer), a two-step training method was established: a pre-training step to build a base model with a general understanding of language, and an additional training step (fine-tuning) of the base model for a specific domain.

[0005] However, with the recent rise of LLM, language models have become significantly larger in scale, resulting in an enormous number of parameters to be fine-tuned and an enormous amount of data to be used for learning, making conventional fine-tuning, which updates all parameters, impractical in terms of computational resources.Furthermore, if there is no access right to the parameters, such as if the trained model is private and cannot be accessed via API, fine-tuning that updates all parameters is not possible in the first place.

[0006] In response to this, instead of updating parameters, so-called prompt engineering (in-context learning) is being considered, in which the LLM is given unknown knowledge while it is interacting by optimizing the content of the input (prompt) when having the LLM engage in a dialogue or perform a task, or by inserting information such as additional prompts, parameters, and task examples. This makes it possible, for example, to provide the LLM with the latest knowledge that was not included in the data used during pre-learning, thereby preventing learned knowledge from becoming outdated.

[0007] As a related technology, for example, Non-Patent Document 1 describes a Retriever-Reader model consisting of a Retriever that vectorizes a question from a user and searches for documents related to the question from a large number of documents (including unknown knowledge) based on the similarity of the vectors, and a Reader that links the related documents searched by the Retriever with the question sentence and provides the result as input to a language model to extract or generate an answer.

[0008] Ayaka Yomota and seven others, "Verification of the Retriever-Reader Model for Utilizing Technical Knowledge," Proceedings of the 29th Annual Conference of the Association for Natural Language Processing, March 2023, pp. 2030-2033

[0009] In cases where there is a limit to the length of the prompt (number of tokens) that can be input to an LLM and there is a huge amount of external knowledge that needs to be additionally learned, conventional technology makes it possible to provide unknown knowledge to the LLM through prompt engineering, which involves extracting related documents from the huge amount of external knowledge, linking them to a question, and providing the LLM with this as a prompt, thereby interacting with the LLM and obtaining an answer.

[0010] However, some of the external knowledge accumulated in the LLM includes confidential information within the company that cannot be leaked to the outside, and some of this information is inappropriate to enter into an open LLM that can be used as SaaS (Software as a Service).

[0011] Therefore, an object of the present invention is to provide a language model utilization system and a language model utilization method that can appropriately perform prompt engineering even when the external knowledge of the accumulated LLM contains confidential information, etc.

[0012] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

[0013] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0014] A language model utilization system that is a representative embodiment of the present invention is a language model utilization system that inputs a prompt sentence specified by a user into a language model and outputs an answer obtained from the language model to the user.The system has a knowledge storage unit that accumulates information related to knowledge, and based on the prompt, obtains knowledge related to or similar to the prompt from the knowledge storage unit, and inputs the knowledge and the prompt together into the language model to obtain an answer.

[0015] The present invention can also be applied to a method executed in a language model-based system such as the one described above.

[0016] The effects obtained by the representative inventions disclosed in this application can be briefly explained as follows.

[0017] In other words, according to a representative embodiment of the present invention, it is possible to appropriately perform prompt engineering even if the accumulated external knowledge of the LLM contains confidential information, etc.

[0018] FIG. 1 is a diagram showing an overview of an example of the configuration of a language model utilization system according to an embodiment of the present invention. FIG. 2 is a diagram showing an overview of an example of prompt engineering according to an embodiment of the present invention. FIG. 3 is a diagram showing an overview of another example of prompt engineering according to an embodiment of the present invention. FIG. 4 is a diagram showing an overview of another example of prompt engineering according to an embodiment of the present invention.

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all drawings used to explain the embodiments, the same parts are generally designated by the same reference numerals, and repeated explanations will be omitted. However, parts that have been designated and explained in one drawing may be referred to by the same reference numerals in the explanation of other drawings, although they will not be shown again.

[0020] <Overview> In a language model utilization system according to one embodiment of the present invention, a prompt (such as a question or inquiry) is input by a user to an LLM, and an answer inferred and generated by the LLM is obtained. When inputting the prompt to the LLM, additional information is acquired from knowledge independently accumulated outside the LLM and input together with the prompt. This enables prompt engineering (in-context learning) to be performed, which provides the LLM with knowledge unknown to the LLM in advance, making it easier to obtain the desired answer.

[0021] The independently accumulated knowledge may include, for example, documents related to specialized knowledge, various internal company documents, email and chat data between employees, and other confidential information that should not be leaked to the outside. It may also include articles, comments, posts, etc. on general topics such as sports, fashion, and games that are not a problem if leaked to the outside (including those obtained from outside). Furthermore, the knowledge is not limited to knowledge unknown to the LLM, and may also include known (already learned) information.

[0022] <System Configuration> Figure 1 is a diagram showing an overview of an example configuration of a language model utilization system according to one embodiment of the present invention. The language model utilization system 1 is configured, for example, with server equipment, a virtual server constructed on a cloud computing service, a PC, etc., and realizes various functions related to the use of LLMs and the implementation of prompt engineering by executing middleware such as an OS (Operating System), a DBMS (Database Management System), and a Web server program, which are loaded onto memory from a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and software running on the OS and DBMS, using a central processing unit (CPU) (not shown).

[0023] This language model utilization system 1 includes, for example, a prompt processing unit 2 implemented as software, and a knowledge storage unit 3, which is a database that stores independently accumulated knowledge.

[0024] The prompt processor 2 has an interface function for using the LLM and a function related to prompt engineering at that time, in that it receives a prompt 6 input from a user 5, obtains information on knowledge related or similar to the topic related to the prompt 6 from the knowledge storage unit 3, inputs this information together with the prompt 6 into a language model 4 such as an LLM, and outputs the obtained answer 7 to the user 5. For example, ChatGPT can be used as the language model 4, and the prompt processor 2 can be constructed using a framework for implementing an application that applies a language model such as LlamaIndex or LangChain, but this is not limiting. Details of the processing by the prompt processor 2 will be described later.

[0025] The knowledge storage unit 3 is a vector database that stores independently accumulated knowledge information as vector data, which is a format that is easy to use in machine learning, and various types of databases such as Vector Storage can be used. It is also possible to configure the unit to use a key-value database instead of a vector database.

[0026] The knowledge information stored in the knowledge storage unit 3 is not limited to vector data, but may also be text data. In this case, the prompt processing unit 2 performs a text-based process of extracting knowledge similar to the prompt 6 input by the user 5 from the knowledge storage unit 3, and the extracted similar knowledge and the prompt 6 are input into the language model 4 to obtain an answer. At this time, for example, the similar knowledge, or text data related to the similar knowledge and the prompt 6, may be presented to the user 5 and then input into the language model 4. Furthermore, the configuration may be such that only the knowledge specified by the user 5 among the similar knowledge is input into the language model 4.

[0027] It is desirable that the knowledge storage unit 3 and the prompt processing unit 2 take security measures (such as encryption) to prevent the text data from being obtained by a third party. Alternatively, instead of a database or text data, a language model independent of the language model 4 may be used, which outputs knowledge that has a high similarity (such as cosine similarity) to the content of the prompt 6 entered by the user 5 or that is highly relevant. An example of a language model independent of the language model 4 here is a BERT-based model specialized for Retriever. Such a language model may be combined with two or more of the databases or text data to output knowledge that is highly relevant.

[0028] <Prompt Engineering> Fig. 2 is a diagram showing an outline of an example of prompt engineering in one embodiment of the present invention, showing an example of the basic processing flow in this embodiment.

[0029] First, a user 5 accesses the language model utilization system 1 and inputs a prompt 6 (a question, a request for advice, etc.) (S01). The prompt processing unit 2 of the language model utilization system 1 converts the content (words and phrases) of the input prompt 6 into a vector representation (so-called "embedding"). In this embodiment, a request is made to the language model 4 to embed the content into a vector representation (S02), and the prompt 6 is vectorized using the functions of the language model 4, and a response is obtained (S03). Regardless of the functions of the language model 4, a vectorization tool provided in the data management function of the knowledge holding unit 3, or another tool that vectorizes the words and phrases of the prompt 6 using natural language processing, for example, may be used.

[0030] Thereafter, the prompt processing unit 2 queries the knowledge storage unit 3 for similar knowledge based on the returned embedding (S04), and obtains a response from the knowledge storage unit 3 (S05). Here, a certain number of knowledge vectors similar to the embedding of the prompt 6 are extracted from the knowledge vector data accumulated and stored in the knowledge storage unit 3, and these are extracted as similar knowledge. This makes it possible to extract information related to knowledge on topics similar to the content of the prompt 6.

[0031] As described above, similar knowledge extracted from the knowledge storage unit 3 may be converted into text and presented to the user 5 (well-known and commonly used techniques for converting vector data into text are available, and this may be queried in a language model), and only the knowledge specified by the user 5 may be used in subsequent processing, thereby making it possible to delete unnecessary knowledge and obtain the answer desired by the user 5 from the language model 4. In this case, it is desirable to take security measures (encryption, etc.) to prevent the text data from being obtained by third parties.

[0032] The vector obtained as similar knowledge and the prompt 6 are then combined and input into the language model 4 (S06), and an answer is obtained from the language model 4 (S07). The prompt processor 2 outputs the obtained answer to the user 5 (S08), and the user 5 obtains this as answer 7 (S09). This makes it possible to provide unique knowledge unknown to the language model 4 along with the prompt 6 to the language model 4, and obtain answer 7 based on this. Note that the answer from the language model 4 obtained by the prompt processor 2 in step S07 is re-embedded, and the series of processes from step S04 onwards are recursively repeated, thereby further improving the accuracy of answer 7.

[0033] 3 is a diagram outlining another example of prompt engineering in one embodiment of the present invention. This example shows a case where prompt engineering is performed based on the attribute information of user 5, based on the basic processing shown in the example of FIG. 2 above.

[0034] 2 , user 5 accesses language model utilization system 1 and inputs prompt 6, at which time user 5 also inputs attribute information about user 5 (S11). The attribute information about user 5 includes, for example, information about the authority that user 5 has within an organization, personal information, and other personal information. This attribute information may be stored in the user 5's local environment, such as a cookie, or may be stored in a database on the server side that includes language model utilization system 1.

[0035] Thereafter, as in the example of Figure 2, the prompt processing unit 2 of the language model utilization system 1 requests the language model 4 to embed the content of the input prompt 6 into a vector expression (S12), and receives an embedding response from the language model 4 (S13).

[0036] Thereafter, the prompt processing unit 2 queries the knowledge storage unit 3 for similar knowledge based on the returned embedding (S14) and obtains a response from the knowledge storage unit 3 (S15). In the example of Fig. 3, similar knowledge is obtained based on the attribute information of the user 5 input in step S11. For example, if the attribute information specifies a genre or field of interest as personal information of the user 5, knowledge similar to the prompt 6 is extracted from the vector data of knowledge accumulated and stored in the knowledge storage unit 3 that matches the genre or the like (or has a high similarity (cosine similarity, etc.), or is strongly related).

[0037] Furthermore, if the attribute information specifies the authority of the user 5, the vector data of knowledge accumulated and stored in the knowledge storage unit 3 is extracted from the vector data that is permitted to be accessed by the authority, and that is similar to the prompt 6. The vector database of the knowledge storage unit 3 may be constructed as separate databases (tables) for each genre or field (e.g., fashion, sports, games, etc.) or authority (level of confidentiality), or may be constructed as a single database (table) as a whole. When constructed as a single database as a whole, metadata indicating the genre, authority, etc., may be added as a column to enable identification. On the other hand, when constructed as separate databases for each genre or authority, vector data that is ranked above a certain level is extracted from the returned vector data from each genre or authority database.

[0038] The access authority for the vector data of each piece of knowledge accumulated and stored in the knowledge storage unit 3 is defined as p (p = {0, 1}), and the similarity with the prompt 6 input by the user 5 is defined as r, and p x r is calculated, and a decision as to whether or not to extract the vector data of the knowledge is made based on the calculation result.

[0039] 2, the vector obtained as similar knowledge and the prompt 6 are combined and input into the language model 4 (S16), and an answer is obtained from the language model 4 (S17). The prompt processor 2 outputs the obtained answer to the user 5 (S18), and the user 5 obtains this as answer 7 (S19). As in the example of Fig. 2, the answer from the language model 4 obtained by the prompt processor 2 in step S17 is re-embedded, and the series of processes from step S14 onwards can be repeated to further improve the accuracy of answer 7.

[0040] As described above, by extracting similar knowledge based on the attribute information input by the user 5, it is possible to easily obtain a personalized answer 7 related to the subject of interest of the user 5. Furthermore, by extracting similar knowledge from the range of information that is permitted to be accessed based on the authority of the user 5 (the user has access rights), it is possible to obtain the answer 7 to the prompt 6 while maintaining security, without having to learn confidential information by updating the parameters of the language model 4. Note that from the perspective of maintaining security, it is also possible to use a proprietary, private language model 4 instead of an external one, instead of the method described above.

[0041] 4 is a diagram outlining another example of prompt engineering in an embodiment of the present invention. This example shows how, as a preliminary step before carrying out the processes shown in the examples of FIGS. 2 and 3, the latest information and knowledge unknown to the language model 4 is collected and accumulated, and an answer 7 based on this information can be provided to the user 5.

[0042] The prompt processing unit 2 of the language model utilization system 1 periodically or irregularly accesses an external information source 8 to request the latest information (S21) and acquires the latest information in response (S22). The specific method of requesting and responding to the latest information is not particularly limited. Depending on the specifications and interface of the information source 8, for example, the information may be acquired using a publicly available API or by scraping information from a website. Examples of the information source 8 include various information providing sites, services, and repositories, such as news sites, stock prices, and chat services that can acquire conversation logs that are only available within the company. When acquiring the latest information, information on the acquisition date and time may be added. This allows information on the same topic to be treated as different historical information in chronological order.

[0043] Thereafter, the prompt processing unit 2 of the language model utilization system 1 requests the language model 4 to embed the acquired latest information into a vector expression (S23), and receives an embedded response from the language model 4 (S24). The embedded latest information is then stored in the vector database of the knowledge storage unit 3 (S25, S26). As a result, when an answer 7 to a prompt 6 is output to the user 5 in the processing of the examples in Figures 2 and 3 described above, it becomes possible to output an answer 7 based on latest information that is unknown to the language model 4.

[0044] As described above, when storing the latest information in the knowledge storage unit 3, adding information about the date and time of acquisition allows the information to be treated as historical information for the same topic. In this case, for example, when the user 5 inputs the prompt 6 in the processing of step S01 in the example of FIG. 2 described above, the user 5 specifies the point in time at which the answer 7 is based. Then, when the prompt processing unit 2 queries the knowledge storage unit 3 for similar knowledge in the processing of step S04, the information from the point in time specified by the user 5 is added, and similar knowledge is extracted from the knowledge storage unit 3 using the latest information at the specified point in time. This makes it possible to appropriately answer questions that require an answer based on the latest information at a specific point in the past, for example.

[0045] As described above, according to the language model utilization system 1, which is one embodiment of the present invention, when inputting the prompt 6 to the language model 4, additional information is obtained from knowledge independently accumulated in the knowledge storage unit 3, and this information is input together with the prompt 6. This allows for prompt engineering, which provides the language model 4 with knowledge that is unknown to the language model 4 in advance, making it easier to obtain the desired answer.

[0046] The invention made by the inventor has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. Furthermore, the above embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of the above embodiments with other configurations.

[0047] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a storage device such as a memory, hard disk, or SSD, or on a storage medium such as an IC card, SD card, or DVD.

[0048] In addition, in the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. In reality, it can be assumed that almost all components are interconnected.

[0049] The present invention can be used in a language model utilization system and a language model utilization method for performing prompt engineering.

[0050] 1... Language model utilization system, 2... Prompt processing unit, 3... Knowledge holding unit, 4... Language model, 5... User, 6... Prompt, 7... Answer, 8... Information source

Claims

1. A language model utilization system that inputs the text of a prompt specified by a user into a language model and outputs the response obtained from the language model to the user, the system having a knowledge retention unit that stores information related to knowledge, and based on the prompt, acquiring knowledge related to or similar to the prompt from the knowledge retention unit, and inputting the knowledge and the prompt together into the language model to obtain a response.

2. The language model utilization system according to claim 1, wherein knowledge related to or similar to the prompt is acquired from the knowledge retention unit based on the prompt and the attribute information of the user.

3. The language model utilization system according to claim 2, wherein the attribute information includes information on the genre of the user's area of interest, and when acquiring knowledge related to or similar to the prompt from the knowledge retention unit, knowledge that matches the genre is acquired.

4. The language model utilization system according to claim 2, wherein the attribute information includes information related to the user's authority, and when acquiring knowledge related to or similar to the prompt from the knowledge retention unit, knowledge is acquired from the range accessible based on the authority.

5. The language model utilization system according to claim 2, wherein knowledge is collected from other information sources regularly or irregularly and stored in the knowledge retention unit.

6. The language model utilization system according to claim 5, wherein the knowledge collected from the information source is stored in the knowledge retention unit as a history together with the information on the time when the knowledge was collected, and when acquiring knowledge related to or similar to the prompt from the knowledge retention unit, knowledge is acquired from the history that was the latest at that time based on the information on the time point specified by the user.

7. A method for using a language model by an information processing system that inputs a sentence of a prompt specified by a user into the language model and outputs an answer obtained from the language model to the user, wherein the information processing system performs: a first step of obtaining knowledge related to or similar to the prompt from the knowledge accumulated in the information processing system based on the prompt; and a second step of combining the obtained knowledge and the prompt and inputting them into the language model to obtain an answer.

8. The method for using a language model according to claim 7, wherein in the first step, knowledge related to or similar to the prompt is obtained from the knowledge accumulated in the information processing system based on the prompt and the attribute information of the user.

Images