Language model utilization system and language model utilization method
The system addresses the challenge of confidential information in LLMs by integrating a knowledge storage unit to perform prompt engineering, ensuring secure and accurate knowledge input for LLMs, enhancing their performance.
Patent Information
- Application Number
- JP2024010072
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-01-26
- Publication Date
- 2025-08-07
AI Technical Summary
Existing technologies face limitations in performing prompt engineering with Large Language Models (LLMs) due to the constraints of input length and the presence of confidential information that cannot be leaked, especially when additional knowledge needs to be learned.
A language model utilization system that incorporates a knowledge storage unit to store and manage confidential information separately, allowing for prompt engineering by inputting prompts with additional knowledge from this unit into the LLM, ensuring security and relevance.
Enables effective prompt engineering even with confidential information, providing LLMs with relevant and up-to-date knowledge while maintaining security and improving answer accuracy.
Smart Images
Figure 2025115561000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a technology for utilizing generative AI and LLMs (Large Language Models), and in particular to a technology that is effective when applied to a language model utilization system and language model utilization method that perform so-called prompt engineering. [Background technology]
[0002] Recently, LLMs such as ChatGPT (registered trademark) have become available that demonstrate incredible conversational and comprehension abilities. However, if you want an LLM to perform highly specialized conversations or tasks (e.g., summarizing text, creating Q&As, generating text, etc.), it is necessary to learn downstream tasks that are specific to the target field or domain. Therefore, optimal additional learning is essential for utilizing LLMs in business.
[0003] There are several ways to train an LLM, but the methods that can be used depend on the model publication method of each LLM. For example, if the model itself is published and commercially available, various methods of training can be used, but for LLMs where only the API (Application Programming Interface) is published, the methods that can be used are limited to updating parameters that can be done or accessed through the API.
[0004] When architectures such as RNN (Recurrent Neural Network) and LSTM (Long Short Term Memory) were mainstream in natural language models, it was common to retrain them from scratch for each downstream task or domain. Later, with the emergence of machine learning models based on so-called Transformer-based natural language processing models, such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pretrained Transformer), and BART (Bidirectional Auto-Regressive Transformer), a two-step training method was established: a pre-training step to build a base model with a general understanding of language, and an additional training step (fine-tuning) of this base model for a specific domain.
[0005] However, with the recent rise of LLM, language models have become significantly larger in scale, resulting in an enormous number of parameters to be fine-tuned and the size of the data used for training, making conventional fine-tuning, which updates all parameters, impractical in terms of computational resources.Furthermore, if you do not have access rights to the parameters, such as if the trained model is private and cannot be accessed via API, fine-tuning that updates all parameters is not possible in the first place.
[0006] In response to this, so-called prompt engineering (in-context learning) is being considered, in which, instead of updating parameters, the content of the input (prompts) when having the LLM engage in a dialogue or perform a task is optimized, or additional prompts, parameters, task examples, and other information is inserted, thereby providing the LLM with unknown knowledge while it is engaged in a dialogue. This makes it possible, for example, to provide the LLM with the latest knowledge not included in the data used during pre-learning, thereby preventing learned knowledge from becoming outdated.
[0007] As a related technology, for example, Non-Patent Document 1 describes a Retriever-Reader model consisting of a Retriever that vectorizes a question from a user and searches for documents related to the question from a large number of documents (including unknown knowledge) based on the similarity of the vectors, and a Reader that links the related documents searched by the Retriever with the question text and provides the result as input to a language model to extract or generate an answer. [Prior art documents] [Non-patent literature]
[0008] [Non-Patent Document 1] Ayaka Yomota and seven others, "Verification of the Retriever-Reader Model for Utilizing Technical Knowledge," Proceedings of the 29th Annual Conference of the Association for Natural Language Processing, March 2023, pp. 2030-2033 Summary of the Invention [Problem to be solved by the invention]
[0009] While there is a limit to the length of the prompt (number of tokens) that can be input to an LLM, when there is a huge amount of external knowledge that needs to be additionally learned, conventional technology uses prompt engineering to extract relevant documents from the vast amount of external knowledge, link them to the question text, and provide this to the LLM as a prompt, making it possible to provide unknown knowledge to the LLM and interact with it to obtain an answer.
[0010] However, some of the external knowledge accumulated in the LLM includes confidential internal information that cannot be leaked to the public, and some of this information is inappropriate to enter into an open LLM that is available as SaaS (Software as a Service).
[0011] Therefore, an object of the present invention is to provide a language model utilization system and a language model utilization method that can appropriately perform prompt engineering even when the accumulated external knowledge of the LLM contains confidential information, etc.
[0012] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]
[0013] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.
[0014] A language model utilization system that is a representative embodiment of the present invention is a language model utilization system that inputs a prompt sentence specified by a user into a language model and outputs an answer obtained from the language model to the user.The system has a knowledge storage unit that accumulates information related to knowledge, and based on the prompt, obtains knowledge related to or similar to the prompt from the knowledge storage unit, and inputs the knowledge and the prompt together into the language model to obtain an answer.
[0015] The present invention can also be applied to a method executed in a language model-based system such as the one described above. [Effects of the Invention]
[0016] The effects obtained by the representative inventions disclosed in this application can be briefly explained as follows.
[0017] In other words, according to a representative embodiment of the present invention, it is possible to appropriately perform prompt engineering even if the accumulated external knowledge of the LLM contains confidential information, etc. [Brief explanation of the drawings]
[0018] [Figure 1] 1 is a diagram showing an outline of an example of the configuration of a language model utilization system according to an embodiment of the present invention; [Figure 2] FIG. 1 is a diagram showing an overview of an example of prompt engineering in one embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing an overview of another example of prompt engineering in one embodiment of the present invention. [Figure 4] FIG. 10 is a diagram showing an overview of another example of prompt engineering in one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION
[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all drawings used to explain the embodiments, the same parts are generally designated by the same reference numerals, and repeated explanations will be omitted. However, parts that have been designated and explained in one drawing may be referred to by the same reference numerals in the explanation of other drawings, although they will not be shown again.
[0020] <Summary> In a language model utilization system that is one embodiment of the present invention, a prompt (question, consultation, etc.) entered by a user into an LLM is input, and an answer inferred and generated by the LLM is obtained. When inputting a prompt into the LLM, additional information is obtained from knowledge independently accumulated outside the LLM and input together with the prompt. This enables prompt engineering (in-context learning) to be performed, which provides the LLM with knowledge unknown to it in advance, making it easier to obtain the desired answer.
[0021] Independently accumulated knowledge includes, for example, documents related to specialized knowledge, various internal documents, emails and chat data between employees, and other confidential information that should be prevented from leaking to the outside. It may also include articles, comments, posts, etc. on general topics such as sports, fashion, and games that are not a problem if leaked to the outside (including those obtained from outside). It is also not limited to knowledge unknown to the LLM, but may also include known (already learned) information.
[0022] <System configuration> 1 is a diagram showing an overview of an example configuration of a language model utilization system according to one embodiment of the present invention. The language model utilization system 1 is configured, for example, with server equipment, a virtual server constructed on a cloud computing service, a PC, etc., and realizes various functions related to the use of LLMs and the implementation of prompt engineering by executing middleware such as an OS (Operating System), a DBMS (Database Management System), and a Web server program, which are deployed on memory from a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and software running on the OS and DBMS, using a central processing unit (CPU) (not shown).
[0023] This language model utilization system 1 has, for example, a prompt processing unit 2 implemented as software, and also has a knowledge storage unit 3, which is a database that stores independently accumulated knowledge.
[0024] The prompt processor 2 has an interface function for using LLM and a function related to prompt engineering at that time, by receiving a prompt 6 input from a user 5, obtaining information on knowledge related or similar to the topic related to the prompt 6 from the knowledge storage unit 3, inputting this information together with the prompt 6 into a language model 4 such as an LLM, and outputting the obtained answer 7 to the user 5. For example, ChatGPT can be used as the language model 4, and the prompt processor 2 can be constructed using a framework for implementing applications that apply language models such as LlamaIndex or LangChain, but this is not limiting. Details of the processing by the prompt processor 2 will be described later.
[0025] The knowledge storage unit 3 is a vector database that stores independently accumulated knowledge information as vector data, which is a format that is easy to use in machine learning, and various types of databases such as Vector Storage can be used. It is also possible to configure the unit to use a key-value database instead of a vector database.
[0026] The knowledge information stored in the knowledge storage unit 3 is not limited to vector data, but may also be text data. In this case, the prompt processing unit 2 performs a text-based process of extracting knowledge similar to the prompt 6 input by the user 5 from the knowledge storage unit 3, and the extracted similar knowledge and the prompt 6 are input into the language model 4 to obtain an answer. At this time, for example, the similar knowledge, or text data related to the similar knowledge and the prompt 6, may be presented to the user 5 and then input into the language model 4. Furthermore, the configuration may be such that only the knowledge specified by the user 5 among the similar knowledge is input into the language model 4.
[0027] It is desirable that the knowledge storage unit 3 and the prompt processing unit 2 take security measures (encryption, etc.) to prevent the text data from being obtained by a third party. Alternatively, instead of a database or text data, a language model independent of the language model 4 may be used, which outputs knowledge that has a high similarity (cosine similarity, etc.) to the content of the prompt 6 entered by the user 5 or that is highly relevant. An example of a language model independent of the language model 4 here is a BERT-based model specialized for Retriever. Such a language model may be combined with two or more of the databases or text data to output knowledge that is highly relevant.
[0028] <Prompt Engineering> 2 is a diagram showing an outline of an example of prompt engineering in one embodiment of the present invention, showing an example of the basic processing flow in this embodiment.
[0029] First, a user 5 accesses the language model utilization system 1 and inputs a prompt 6 (a question, a request for advice, etc.) (S01). The prompt processing unit 2 of the language model utilization system 1 converts the content (words and phrases) of the input prompt 6 into a vector representation (so-called "embedding"). In this embodiment, a request is made to the language model 4 to embed the content into a vector representation (S02), and the prompt 6 is vectorized using the functions of the language model 4, and a response is obtained (S03). Regardless of the functions of the language model 4, a vectorization tool provided in the data management function of the knowledge storage unit 3, or another tool that vectorizes the words and phrases of the prompt 6 using natural language processing, for example, may be used.
[0030] The prompt processing unit 2 then queries the knowledge storage unit 3 for similar knowledge based on the returned embedding (S04), and obtains a response from the knowledge storage unit 3 (S05). Here, a certain number of knowledge vectors similar to the embedding of prompt 6 are extracted from the knowledge vector data accumulated and stored in the knowledge storage unit 3, and extracted as similar knowledge. This makes it possible to extract information related to knowledge on topics similar to the content of prompt 6.
[0031] As mentioned above, similar knowledge extracted from the knowledge storage unit 3 may be converted into text and presented to the user 5 (well-known and commonly used techniques for converting vector data into text are available, and this may be queried in a language model), and only the knowledge specified by the user 5 may be used in subsequent processing, thereby making it possible to delete unnecessary knowledge and obtain the answer desired by the user 5 from the language model 4. In this case, it is desirable to take security measures (encryption, etc.) to prevent the text data from being obtained by third parties.
[0032] The vector obtained as similar knowledge and the prompt 6 are then combined and input into the language model 4 (S06), and an answer is obtained from the language model 4 (S07). The prompt processor 2 outputs the obtained answer to the user 5 (S08), and the user 5 obtains this as answer 7 (S09). This makes it possible to provide unique knowledge unknown to the language model 4 along with the prompt 6 to the language model 4, and obtain answer 7 based on this. Note that the answer from the language model 4 obtained by the prompt processor 2 in step S07 is re-embedded, and the series of processes from step S04 onwards are recursively repeated, thereby further improving the accuracy of answer 7.
[0033] Figure 3 is a diagram outlining another example of prompt engineering in an embodiment of the present invention. This example shows a case where prompt engineering is performed based on the attribute information of user 5, based on the basic processing shown in the example of Figure 2 above.
[0034] First, as in the example of Figure 2, user 5 accesses language model utilization system 1 and inputs prompt 6, at which time user 5 also inputs attribute information about user 5 (S11). Attribute information about user 5 includes, for example, information about the authority that user 5 has within an organization, personal information, and other personal information. This attribute information may be stored in the local environment of user 5, such as a cookie, or may be stored in a database on the server side that includes language model utilization system 1.
[0035] Thereafter, as in the example of Figure 2, the prompt processing unit 2 of the language model utilization system 1 requests the language model 4 to embed the content of the input prompt 6 into a vector representation (S12), and receives an embedding response from the language model 4 (S13).
[0036] Thereafter, the prompt processing unit 2 queries the knowledge storage unit 3 for similar knowledge based on the returned embedding (S14) and obtains a response from the knowledge storage unit 3 (S15). In the example of Fig. 3, similar knowledge is obtained based on the attribute information of the user 5 input in step S11. For example, if the attribute information specifies a genre or field of interest as personal information of the user 5, knowledge similar to the prompt 6 is extracted from the vector data of knowledge accumulated and stored in the knowledge storage unit 3 that matches the genre, etc. (or has a high similarity (cosine similarity, etc.), or is strongly related).
[0037] Furthermore, if the attribute information specifies the authority of the user 5, the vector data of knowledge accumulated and stored in the knowledge storage unit 3 is extracted from the vector data that is permitted to be accessed by the authority, and that is similar to the prompt 6. The vector database of the knowledge storage unit 3 may be constructed as separate databases (tables) for each genre or field (e.g., fashion, sports, games, etc.) or authority (level of confidentiality), or may be constructed as a single database (table) as a whole. When constructed as a single database as a whole, metadata indicating the genre, authority, etc., can be added as columns to enable identification. On the other hand, when constructed as separate databases for each genre or authority, vector data that is ranked above a certain level is extracted from the returned vector data from each genre or authority database.
[0038] The access authority for the vector data of each piece of knowledge accumulated and stored in the knowledge storage unit 3 is set to p (p={0,1}), and the similarity with the prompt 6 input by the user 5 is set to r, and p×r is calculated, and a decision may be made based on the calculation result as to whether or not to extract the vector data of the piece of knowledge.
[0039] Thereafter, as in the example of Figure 2, the vector obtained as similar knowledge and prompt 6 are combined and input into language model 4 (S16), and an answer is obtained from language model 4 (S17). The prompt processor 2 outputs the obtained answer to user 5 (S18), and user 5 obtains this as answer 7 (S19). As in the example of Figure 2, the answer from language model 4 obtained by prompt processor 2 in step S17 is re-embedded, and the series of processes from step S14 onwards can be repeated to further improve the accuracy of answer 7.
[0040] As described above, by extracting similar knowledge based on the attribute information input by the user 5, it is possible to easily obtain a personalized answer 7 related to the interest of the user 5. Furthermore, by extracting similar knowledge from the range of information that is permitted to be accessed based on the authority of the user 5 (there is access authority), it is possible to obtain an answer 7 to the prompt 6 while maintaining security, without having to learn confidential information by updating the parameters of the language model 4. Note that from the perspective of maintaining security, it is also possible to use a proprietary, private language model 4 instead of an external one, instead of the method described above.
[0041] Figure 4 is a diagram outlining another example of prompt engineering in an embodiment of the present invention. This example shows how, as a preliminary step before carrying out the processes shown in the examples of Figures 2 and 3, the latest information and knowledge unknown to the language model 4 is collected and accumulated, and an answer 7 based on this information can be provided to the user 5.
[0042] The prompt processing unit 2 of the language model utilization system 1 periodically or irregularly accesses an external information source 8 to request the latest information (S21), and acquires the latest information in response (S22). The specific method of requesting and responding to the latest information is not particularly limited. Depending on the specifications and interface of the information source 8, for example, the information may be acquired using a publicly available API or by scraping information from a website. Examples of information sources 8 include various information-providing sites, services, and repositories, such as news sites, stock prices, and chat services that allow acquisition of conversation logs that are only available within the company. When acquiring the latest information, information on the acquisition date and time may be added. This allows information on the same topic to be treated as different historical information in chronological order.
[0043] Thereafter, the prompt processing unit 2 of the language model utilization system 1 requests the language model 4 to embed the acquired latest information into a vector expression (S23), and receives an embedded response from the language model 4 (S24). The embedded latest information is then stored in the vector database of the knowledge storage unit 3 (S25, S26). As a result, when outputting an answer 7 to the prompt 6 to the user 5 in the processing of the examples in FIGS. 2 and 3 described above, it becomes possible to output an answer 7 based on latest information that is unknown to the language model 4.
[0044] As described above, when storing the latest information in the knowledge storage unit 3, adding information about the date and time of acquisition allows the information to be treated as historical information for the same topic. In this case, for example, when the user 5 inputs the prompt 6 in the processing of step S01 in the example of FIG. 2 described above, the user 5 specifies the point in time at which the answer 7 is based. Then, when the prompt processing unit 2 queries the knowledge storage unit 3 for similar knowledge in the processing of step S04, the information from the point in time specified by the user 5 is added, and similar knowledge is extracted from the knowledge storage unit 3 using the latest information at the specified point in time. This makes it possible to appropriately answer questions that require an answer based on the latest information at a specific point in the past, for example.
[0045] As described above, according to the language model utilization system 1 which is one embodiment of the present invention, when inputting the prompt 6 to the language model 4, additional information is obtained from knowledge independently accumulated in the knowledge storage unit 3, and this information is input together with the prompt 6. This allows for prompt engineering, which provides the language model 4 with knowledge that is unknown to the language model 4 in advance, making it easier to obtain the desired answer.
[0046] The invention made by the inventor has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. Furthermore, the above embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of the above embodiments with other configurations.
[0047] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a storage device such as a memory, hard disk, or SSD, or in a storage medium such as an IC card, SD card, or DVD.
[0048] In addition, in the above figures, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. In reality, it can be assumed that almost all components are interconnected. [Industrial Applicability]
[0049] The present invention can be used in a language model utilization system and a language model utilization method for performing prompt engineering. [Explanation of symbols]
[0050] 1...Language model utilization system, 2...Prompt processing unit, 3...Knowledge storage unit, 4...Language model, 5...User, 6...Prompt, 7...Response, 8...Information source
Claims
1. A language model utilization system that inputs a prompt sentence specified by a user into a language model and outputs an answer obtained from the language model to the user, comprising: a knowledge storage unit that stores information related to knowledge; The language model utilization system obtains knowledge related to or similar to the prompt from the knowledge storage unit based on the prompt, and inputs the knowledge and the prompt together into the language model to obtain an answer.
2. 2. The language model utilization system according to claim 1, A language model utilization system that acquires knowledge related to or similar to the prompt from the knowledge storage unit based on the prompt and attribute information of the user.
3. 3. The language model utilization system according to claim 2, The attribute information includes information about genres of interest of the user, A language model utilization system that acquires knowledge that matches the genre when acquiring knowledge related to or similar to the prompt from the knowledge storage unit.
4. 3. The language model utilization system according to claim 2, the attribute information includes information related to the authority of the user, A language model utilization system that, when retrieving knowledge related to or similar to the prompt from the knowledge storage unit, retrieves knowledge from a range accessible based on the authority.
5. 3. The language model utilization system according to claim 2, A language model utilization system that periodically or irregularly collects knowledge from other information sources and stores it in the knowledge storage unit.
6. 6. The language model utilization system according to claim 5, The knowledge collected from the information source is stored in the knowledge storage unit as a history together with information on the time when the knowledge was collected; A language model utilization system that, when retrieving knowledge related to or similar to the prompt from the knowledge storage unit, retrieves knowledge from the history that was the latest at that time based on information specified by the user.
7. 1. A method for using a language model by an information processing system that inputs a prompt sentence specified by a user into a language model and outputs an answer obtained from the language model to the user, comprising: The information processing system, a first step of acquiring knowledge related to or similar to the prompt from knowledge accumulated in the information processing system based on the prompt; a second step of inputting the acquired knowledge and the prompt together into the language model to obtain an answer.
8. 8. The method of using a language model according to claim 7, In the first step, knowledge related to or similar to the prompt is acquired from knowledge accumulated in the information processing system based on the prompt and attribute information of the user.