Language processing device, language processing method, and language processing program
The language processing device addresses prompt attacks in generative AI by averaging contexts using a weighted initialization method, ensuring the AI retains relevant past information and maintains dialogue continuity, thereby improving its response to changing situations.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2026-03-11
AI Technical Summary
Generative AI systems are vulnerable to prompt attacks that manipulate them into outputting unintended responses, posing a security risk.
A language processing device and method that uses a weak initialization unit to replace multiple contexts in a generation AI's memory with a single average context, calculated through weighted averaging, to mitigate prompt attacks while maintaining the continuity of past dialogues.
The solution effectively counters prompt attacks by allowing the AI to gradually return to a past average value, retaining important context and maintaining the natural flow of conversations, thus enhancing the system's resilience against manipulation.
Smart Images
Figure 2026042390000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a security measure against prompt attacks on generated AI. [Background technology]
[0002] Conventionally, an AI model called LLM (Large Language Model) has been provided that enables natural-sounding sentence creation and summarization by learning from large amounts of text. Furthermore, generative AI that combines an LLM with an interactive user interface (see, for example, Non-Patent Document 1) is also widely used. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] ChatGPT, OpenAI, Internet<https: / / openai.com / chatgpt / > Summary of the Invention [Problem to be solved by the invention]
[0004] Generative AI is designed to respond to prompts input from the user by outputting natural-sounding sentences. However, there are attacks that confuse the generative AI by giving it a series of special prompts to intentionally complicate the situation and make the generative AI output sentences that it should not respond to, which poses a security issue for generative AI.
[0005] An object of the present invention is to provide a language processing device, a language processing method, and a language processing program that are capable of taking measures against prompt attacks while continuing the content of past dialogues. [Means for solving the problem]
[0006] The language processing device of the present invention comprises an input unit that inputs a prompt received from a user to a generation AI, an output unit that outputs a context that expresses a context held by the generation AI and a response generated by the generation AI based on the prompt, a memory unit that stores the context that is updated when the generation AI generates a response as a history each time it is updated, and a weak initialization unit that, at a predetermined timing, replaces multiple contexts stored in the memory unit as the history with a single average context that is obtained by averaging the values of the multiple contexts and replaces the context held by the generation AI with the average context.
[0007] The weak initialization unit may generate the average context at the predetermined timing each time the generation AI generates a predetermined number of responses.
[0008] The weak initialization unit may generate the average context when a predetermined waiting time has elapsed without the input unit receiving a prompt, as the predetermined timing.
[0009] The memory unit may further store a past history of the average context, and the weak initialization unit may replace the context held by the generation AI with a weighted average of the average context and a value obtained by averaging the past history.
[0010] The weak initialization unit may calculate the average context by a weighted average.
[0011] The weak initialization unit may determine weights in the weighted average according to an acceleration of the context value.
[0012] The language processing method of the present invention involves a computer inputting a prompt received from a user to a generation AI via an input unit, outputting a context expressing a context held by the generation AI and a response generated by the generation AI based on the prompt via an output unit, storing the context updated when the generation AI generates a response in a memory unit as history each time it is updated, and replacing, at a predetermined timing via a weak initialization unit, the multiple contexts stored in the memory unit as the history with a single average context obtained by averaging the values of the multiple contexts, and replacing the context held by the generation AI with the average context.
[0013] A language processing program according to the present invention causes a computer to function as the language processing device. [Effects of the Invention]
[0014] According to the present invention, it is possible to take measures against prompt attacks while continuing the content of past dialogues. [Brief explanation of the drawings]
[0015] [Figure 1] 1 is a block diagram showing a functional configuration of a language processing apparatus according to an embodiment; [Figure 2] FIG. 10 is a diagram illustrating input and output of a generation AI in an embodiment. [Figure 3] FIG. 1 is a diagram illustrating an operation procedure of a conventional AI system. [Figure 4] FIG. 2 is a diagram illustrating an operation procedure of the language processing device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0016] An example of an embodiment of the present invention will be described below. The language processing device of this embodiment reduces the impact of prompt attacks by using a method of weakly initializing past contexts held by the generation AI, without completely forgetting prompts and responses up to the present.
[0017] FIG. 1 is a block diagram showing the functional configuration of a language processing device 1 according to this embodiment. The language processing device 1 is an information processing device equipped with a control unit 10 and a memory unit 20, and accepts prompts, which are input to the generation AI from a user's terminal via a network (Internet) or directly from the user, and outputs a response sentence as an answer. The generation AI may be implemented in the language processing device 1, or in another information processing device connected for communication.
[0018] The control unit 10 is a part that controls the entire language processing device 1, and realizes each function in this embodiment by appropriately reading and executing various programs stored in the storage unit 20. The control unit 10 may be a CPU. Specifically, the control unit 10 functions as an input unit 11, an output unit 12, and a weak initialization unit 13 by executing a language processing program stored in the storage unit 20.
[0019] The storage unit 20 is a storage area for various programs for causing the hardware group to function as the language processing device 1, various data, etc., and may be a ROM, RAM, flash memory, hard disk drive (HDD), or the like. In this embodiment, in addition to the language processing program, the storage unit 20 stores the context, which is updated when the generation AI generates a response, as a history each time the context is updated.
[0020] The input unit 11 inputs a prompt received from the user to the generation AI. The output unit 12 outputs a response generated by the generation AI based on a prompt and a context that represents the context up to the present that is held by the generation AI.
[0021] Here, the context updated by the generation AI is expressed as a multidimensional numeric vector, i.e., each element of the vector can be subjected to a numeric calculation. In this embodiment, a single context is newly calculated by performing the below-described numeric calculation for each element of the history of multiple contexts acquired as a numeric vector.
[0022] At a predetermined timing, the weak initialization unit 13 replaces multiple contexts stored in the memory unit 20 as history with a single average context obtained by averaging the values of these multiple contexts, and replaces the context held by the generation AI with the average context.
[0023] In this embodiment, the weak initialization unit 13 generates an average context every time the generation AI generates a predetermined number of responses, but this is not limited to this. For example, the weak initialization unit 13 may generate an average context at a constant time interval or when a predetermined waiting time has elapsed without the input unit 11 receiving a prompt.
[0024] FIG. 2 is a diagram showing input and output of the generation AI in this embodiment. If we look at the processing layer of the generative AI as a function, as shown in Figure 2, it is a function that receives prompts and context from the user as input and returns a response. Conventional AI systems that provide services using generative AI repeatedly use this function, operating, for example, as follows:
[0025] FIG. 3 is a diagram showing the operation procedure of a conventional AI system. Each time the user inputs a prompt, the AI stores the current situation as a result of the processing as a context value. As a result, the context of the conversation up to that point is stored in the context and reflected in the next response. The initial context (Init. Context) is the initial state of the generated AI, and stores information such as the role it will play (for example, the role of a knowledgeable and calm assistant).
[0026] Countermeasures against prompt attacks on the generation AI include improvements to the processing layer itself, as well as measures that often involve resetting the context to its initial value so that the generation AI periodically forgets past prompts and responses, or stopping the generation AI from responding after a certain number of prompts. On the other hand, in this embodiment, countermeasures against prompt attacks are realized by synthesizing context values without making any changes to the processing layer of the generation AI, allowing the generation AI to remember past contexts to a certain extent.
[0027] FIG. 4 is a diagram showing the operation procedure of the language processing device 1 in this embodiment. The language processing device 1 stores in the storage unit 20 at least a predetermined number of contexts that are updated every time the generation AI (the function in FIG. 2) outputs a response. Then, the language processing device 1 calculates the average value (Avg. Context 4) of the context for each span of a certain number of times (for example, four times, or seven times if the magical number 7, known as the human forgetting span, is used), and uses the calculation result as the initial context for the next span.
[0028] In this case, the average value may be calculated as a weighted average, and the weight of each context may be set appropriately, thereby preventing the context value from deviating too much from the initial context. Here, the procedure for calculating the average value of the context can be modified as follows, for example.
[0029] (1) The memory unit 20 further stores the past history of the average context, and the weak initialization unit 13 replaces the context held by the generation AI with a value obtained by further weighting the average context and the value obtained by averaging the past history.
[0030] In this way, by calculating the average value hierarchically and combining it with the initial context of the next span using a weighted average, it is possible to efficiently weakly initialize the context while retaining a moderate amount of past context. Note that there is no limit to the number of layers for which the average value is calculated.
[0031] (2) The weak initialization unit 13 calculates the average context by weighted averaging. For example, the context changes in the latter half of a span are the part that remembers the most recent user input (prompt) and are therefore highly important for continuing the dialogue, while the context changes in the first half are considered to be the part that serves as the prerequisite information. Therefore, a calculation method that leaves the context changes in the latter half and removes the changes in the first half, i.e., gives a higher weight to the latter half, may be used.
[0032] (3) The weak initialization unit 13 may determine the weight of the weighted average in accordance with the acceleration of the context value, rather than making it constant. For example, when calculating the average context CA from contexts C1, C2, C3, and C4, the acceleration based on each context value is (C4-C3)-(C3-C2) Or, the acceleration of the change in the context is defined as |C4-C3|-|C3-C2| and weights may be set according to these values.
[0033] Also, based on the rate of increase of acceleration within the span, (|C4-C3|-|C3-C2|) / (|C3-C2|-|C2-C1|) If is a large positive value (for example, 2 or more), the value according to this positive value will be CA=(C1×1+C2×2+C3×3+C4×4) / 10 As shown above, if the weighted average is taken with emphasis on the latter half, and the rate of increase is small or negative (for example, less than 0.5), CA=(C1×4+C2×3+C3×2+C4×1) / 10 As shown above, a weighted average is taken that emphasizes the first half, and in other cases, CA=(C1+C2+C3+C4) / 4 A simple average may be used as the calculation method.
[0034] According to this embodiment, the language processing device 1 can retain, or remember, the context of past dialogues to a certain extent, compared to conventional countermeasures against prompt attacks, thereby suppressing behavior such as suddenly forgetting all the content of past dialogues and maintaining the naturalness of dialogues. As a result, the language processing device 1 can counter prompt attacks while maintaining the content of past dialogues.
[0035] Prompt attacks on generative AI are a problem for which no adequate countermeasures have been found. Humans may initialize context by forgetting to avoid confusion in complex situations, but it has been difficult to implement a forgetting mechanism in generative AI because it is not possible to instruct a system to forget using a prompt, and adding a prompt would further complicate things. In this embodiment, a method is used in which past contexts are numerically averaged and the system is forced to gradually return to the past average value, making it possible to improve the system's ability to respond to changes in the situation during a conversation while also achieving a forgetting system.
[0036] Furthermore, the language processing device 1 can establish an appropriate forgetting mechanism that places emphasis on important content in a dialogue by appropriately weighting using a weighted average. In particular, the language processing device 1 focuses on the amount of change in the context and determines the weight of the weighted average based on the acceleration, thereby realizing appropriate weighting.
[0037] Furthermore, this embodiment makes it possible to counter prompt attacks on generative AI, for example, and therefore contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation."
[0038] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.
[0039] The language processing method by the language processing device 1 is realized by software. When realized by software, programs constituting this software are installed in an information processing device (computer). These programs may be recorded on removable media such as CD-ROMs and distributed, or may be distributed by being downloaded to a computer via a network. [Explanation of symbols]
[0040] 1 Language Processing Unit 10 Control Unit 11 Input section 12 Output section 13 Weak initialization section 20 Memory section
Claims
1. an input unit that inputs a prompt received from a user to the generation AI; an output unit that outputs a response generated by the generation AI based on a context that represents a context held by the generation AI and the prompt; a storage unit that stores a context that is updated when the generation AI generates a response as a history each time the context is updated; A language processing device comprising: a weak initialization unit that, at a predetermined timing, replaces multiple contexts stored in the memory unit as the history with a single average context obtained by averaging the values of the multiple contexts, and replaces the context held by the generation AI with the average context.
2. The language processing device according to claim 1 , wherein the weak initialization unit generates the average context at the predetermined timing each time the generation AI generates a predetermined number of responses.
3. The language processing device according to claim 1 , wherein the weak initialization unit generates the average context when a predetermined waiting time has elapsed without the input unit receiving a prompt, as the predetermined timing.
4. the storage unit further stores a past history of the average context; The language processing device according to claim 1 , wherein the weak initialization unit replaces the context held by the generation AI with a weighted average of the average context and a value obtained by averaging the past history.
5. The language processing device according to claim 1 , wherein the weak initialization unit calculates the average context by weighted averaging.
6. The language processing device according to claim 5 , wherein the weak initialization unit determines the weight in the weighted average according to an acceleration of the context value.
7. The computer The input unit inputs the prompt received from the user to the generation AI; An output unit outputs a context representing the context held by the generation AI and a response generated by the generation AI based on the prompt; The context updated when the generation AI generates a response is stored in a storage unit as a history each time the context is updated; A language processing method in which a weak initialization unit replaces, at a predetermined timing, multiple contexts stored in the memory unit as the history with a single average context obtained by averaging the values of the multiple contexts, and replaces the context held by the generation AI with the average context.
8. A language processing program for causing a computer to function as the language processing device according to any one of claims 1 to 3.