Safety evaluation device, safety evaluation method, and safety evaluation program

Abstract

To provide a safety evaluation device that can quantitatively evaluate the extent to which performance degrades when an adversarial prompt is entered during search extension generation. [Solution] The security evaluation device 1 includes: a normal accuracy calculation unit 11 that calculates a first value indicating the accuracy of search extension generation by accumulating the semantic similarity between the response generated by the search extension generation to be evaluated for a prompt in the dataset and the normal response expected for the prompt; an attack accuracy calculation unit 12 that calculates a second value indicating the accuracy of search extension generation when attacked by accumulating the semantic similarity between the response generated by the search extension generation to an attack prompt with predetermined modifications and the normal response; and an evaluation result output unit 13 that outputs an evaluation value of the security of search extension generation according to the amount of decrease from the first value to the second value.

Description

[Technical Field]

[0001] This invention relates to a technique for safety evaluation in search extension generation. [Background technology]

[0002] Retrieval Augmented Generation (RAG) is a technique that searches a database for sentences similar to the content of a prompt entered by the user, and generates a response to the prompt based on the retrieved related sentences. Normally, when generating a response to a prompt, generative language models can only understand vocabulary and contexts that they have learned in advance, making it difficult to handle content that was not included in the training, such as knowledge in a specific field. By using Retrieval Augmented Generation, sentences containing knowledge in a specific field are prepared in advance as a database, and prompts are given using the information in this database as hints, the generative language model can generate a response while interpreting knowledge in that specific field.

[0003] In contrast, attacks have been reported that can induce the generation of incorrect responses by intentionally modifying the dataset or prompts used in search extension generation (see, for example, Non-Patent Document 1). In this attack, the attacker aims to generate a false answer to a given prompt. First, the attacker prepares a prompt that is expected to be entered by the user and the answer that the attacker wants to induce the user to generate for that prompt. Next, the attacker generates a sentence containing the answer they want the user to generate. At this time, the attacker optimizes the prepared sentence so that it is included at the top of the search results when the search extension generation searches for relevant information from the database based on the expected prompt. Finally, the attacker inserts the optimized sentence into the search extension generation database. If this attack is successful, when the user enters a specific prompt, the sentence inserted by the attacker is extracted in the search extension generation, and an incorrect answer is generated based on this sentence. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Specification of Japanese Patent Application No. 2024-060196 [Patent Document 2] Specification of Japanese Patent Application No. 2024-060268 [Patent Document 3] Specification of Japanese Patent Application No. 2024-086922 [Non-Patent Document]

[0005] [Non-Patent Document 1] W. Zou et al., "PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models," 2024, arXiv:2402.07867. [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] In the above-mentioned attack, it is necessary to assume the prompt that the user is supposed to input. If the user does not input a prompt with the same content, the attack will not succeed. However, when evaluating the security of retrieval-augmented generation, it is necessary to verify overall whether unexpected outputs are caused without limiting the prompt.

[0007] An object of the present invention is to provide a security evaluation device, a security evaluation method, and a security evaluation program that can quantitatively evaluate how much the performance deteriorates when an adversarial prompt is input in retrieval-augmented generation. [Means for Solving the Problems]

[0008] The security evaluation device according to the present invention comprises: a normal-time accuracy calculation unit that calculates a first value indicating the accuracy of the search extension generation by accumulating the semantic similarity between the response generated by the search extension generation to be evaluated for a prompt in a dataset and the normal response expected for the prompt; an attack-time accuracy calculation unit that calculates a second value indicating the accuracy of the search extension generation when attacked by accumulating the semantic similarity between the response generated by the search extension generation for an attack prompt that has been subjected to a predetermined modification and the normal response; and an evaluation result output unit that outputs an evaluation value of the security of the search extension generation according to the amount of decrease from the first value to the second value.

[0009] The aforementioned predetermined manipulation may be an operation to insert an adversarial string into the prompt, and the system may include a string optimization unit that updates the adversarial string to minimize the similarity between the prompt after the operation and the prompt before the operation was performed.

[0010] The security evaluation device may include a database update unit that receives a pair of a prompt and a corresponding malicious response, and adds the pair of the crafted attack prompt and the malicious response to the database for generating the search extension in advance.

[0011] The security evaluation method according to the present invention involves a computer calculating a first value indicating the accuracy of the search extension generation by accumulating the semantic similarity between the response generated by the search extension generation to be evaluated for a prompt in a dataset and the expected normal response to the prompt using a normal accuracy calculation unit; calculating a second value indicating the accuracy of the search extension generation when under attack by accumulating the semantic similarity between the response generated by the search extension generation for an attack prompt that has been modified with a predetermined technique and the normal response using an attack accuracy calculation unit; and outputting an evaluation result output value of the security of the search extension generation corresponding to the decrease in value from the first to the second.

[0012] The safety evaluation program according to the present invention is for causing a computer to function as the safety evaluation device. [Effects of the Invention]

[0013] According to the present invention, it is possible to quantitatively evaluate the extent to which performance degrades when an adversarial prompt is entered during search extension generation. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram showing the functional configuration of a safety evaluation device in an embodiment. [Figure 2] This is a flowchart illustrating the procedure for the safety evaluation method in the embodiment. [Figure 3] This is a flowchart illustrating the procedure for optimizing the crafting process in the embodiment. [Figure 4] This flowchart illustrates the procedure for updating the database as a pre-processing step for safety evaluation in the embodiment. [Modes for carrying out the invention]

[0015] An example of an embodiment of the present invention will be described below. The security evaluation device of this embodiment quantitatively evaluates the security against adversarial attacks in a system using search extension generation by manipulating the prompts entered by the user, such as by inserting adversarial strings, to see how much the accuracy (correctness rate) of the generated response decreases compared to before the manipulation, i.e., the security against adversarial attacks.

[0016] In this embodiment, the following steps are assumed for generating search extensions, and various definitions are shown. Let S be the database used for generating search extensions. For the i-th piece of information in S, prompt p (i) and the response to it (i) Information composed of s (i) =(p (i) ,r (i)Let it be set as such. Also, let the function that returns the similarity between two texts used in search expansion generation be σ. The search expansion generation system is for the prompt p input by the user user to obtain the p for which the similarity σ(p user , p (i) ) is the highest. The information (s (i) and the accompanying information) obtained at this time is set as the context c. (i) Let the operation of creating a new prompt that instructs to respond to the prompt input by the user based on the context c be f. Then, the created prompt can be expressed as p

[0017] = f(c, p RAG = f(c, p user ). When the generative language model is set as m, the response r RAG using search expansion generation is generated as r RAG = m(p RAG ).

[0018] Summarizing the above, when the operation of search expansion generation is RAG, the operation of generating the response r user for the prompt p input by the user based on the database S is expressed by the following formula. RAG r r RAG = RAG(p user ; S)

[0019] Note that when constructing the context c in search expansion generation, there may be multiple pieces of information obtained from the database. That is, the context c may be constructed by combining multiple prompts and responses.

[0020] Figure 1 is a block diagram showing the functional configuration of the safety evaluation device 1 in the present embodiment. The safety evaluation device 1 is an information processing device including a control unit 10, a storage unit 20, and various input / output interfaces. Also, the safety evaluation device 1 is communicatively connected to a generative AI system that uses the generative language model to be evaluated.

[0021] The control unit 10 is the part that controls the entire safety evaluation device 1, and realizes each function in this embodiment by appropriately reading and executing various programs and data stored in the storage unit 20. The control unit 10 may be a CPU. The memory unit 20 is a storage area for various programs and data necessary for the hardware group to function as a safety evaluation device 1, and may be ROM, RAM, flash memory, or a hard disk drive (HDD).

[0022] The control unit 10 functions as a normal accuracy calculation unit 11, an attack accuracy calculation unit 12, an evaluation result output unit 13, a string optimization unit 14, and a database update unit 15 by executing the software (safety evaluation program) stored in the memory unit 20.

[0023] The normal accuracy calculation unit 11 calculates a first value indicating the accuracy of the search extension generation by accumulating the semantic similarity between the response generated by the search extension generation being evaluated for a prompt in the dataset and the expected normal response for that prompt.

[0024] The attack accuracy calculation unit 12 calculates a second value indicating the accuracy of search extension generation when an attack occurs by accumulating the semantic similarity between the response generated by search extension generation to an attack prompt that has been subjected to a predetermined crafting, and a normal response. The specified manipulation may, for example, be an operation to insert an adversarial string into the prompt.

[0025] The evaluation result output unit 13 outputs an evaluation value for the safety of search extension generation, corresponding to the amount of decrease from a first value indicating accuracy under normal conditions to a second value indicating accuracy during an attack.

[0026] The string optimization unit 14 updates the adversarial string to minimize the similarity between the pre-modified prompt and the prompt before the modification.

[0027] The database update unit 15 receives pairs of prompts and corresponding malicious responses, and updates the database for search extension generation by pre-adding pairs of attack prompts (which have been modified) and malicious responses.

[0028] Figure 2 is a flowchart illustrating the procedure for the safety evaluation method in this embodiment. Here, the prompt p that the user enters user In response, the attacker can apply a certain modification g to the attack prompt p adv =g(p user It is assumed that this will generate ). Note that the trick g is not particularly limited, but examples of tricks include concatenating an adversarial string to the end of the prompt, as described later.

[0029] Here, let D be the dataset used for safety evaluation. For the j-th piece of information in D, prompt p (j) And the expected normal response to this prompt is r (j) Information composed of d (j) =(p (j) ,r (j) Let ) be defined as follows. Also, the function ρ is a function that determines whether two sentences have the same meaning and returns a similarity value from 0 to 1, returning 1 if they have the same meaning. The safety evaluation device 1 performs safety evaluation processing using the database S, the dataset D, and the function ρ in the following procedure.

[0030] In step S1, the control unit 10 stores a variable a that indicates the accuracy of the response to the prompt. benign Set to 0, and store a variable a that stores a value indicating the accuracy of the response to the attack prompt. adv Initialize the variable to 0 and the dataset index j to 1.

[0031] In step S2, the normal accuracy calculation unit 11 performs normal search extension generation, and for the j-th data of dataset D, r (j) benign =RAG(p(j) ;S) generates a response, a benign ←a benign +ρ(r (j) ,r (j) benign Update it as ).

[0032] In step S3, the attack accuracy calculation unit 12 generates an adversarial search extension by applying the following to the j-th data point of dataset D: (j) adv =RAG(g(p (j) );S) generates a response, a adv ←a adv +ρ(r (j) ,r (j) adv Update it as ).

[0033] In step S4, the control unit 10 changes the index j from j ← j+1. In step S5, the control unit 10 determines whether j exceeds the size of dataset D, that is, whether all the data in dataset D has been processed. If the determination is YES, the process moves to step S6; if the determination is NO, the process returns to step S2.

[0034] In step S6, the evaluation result output unit 13 determines each variable by |D|, based on the number of data points in dataset D. benign ←a benign / |D|, a adv ←a adv It is normalized as / |D|. Then, the evaluation result output unit 13 is a benign and a adv The difference (a benign -a adv ), or the percentage of decrease ((a benign -a adv ) / a benign The value output represents the safety of the generative AI system using search extension generation, which is the subject of evaluation. Here, the smaller the difference or reduction, the more resistant and safer the system is considered to be against adversarial attacks.

[0035] a obtained in the safety evaluation processbenign This represents the accuracy of search extension generation when no attack is occurring. The security evaluation device 1 is a benign In contrast, a in the event of an attack adv By checking how much it has decreased, we can evaluate the safety of the RAG operation for generating search extensions based on database S.

[0036] By optimizing the crafting method g, it becomes possible to conduct a proper security assessment that anticipates more powerful attacks. The following implementation is a concrete example of the manipulation g. Here, we consider an operation in which an attacker inserts an adversarial string t into the prompt entered by the user. Also, when calculating similarity in search extension generation, we obtain the string embedding, and let the embedding model be E.

[0037] Figure 3 is a flowchart illustrating the procedure for optimizing the crafting method g in this embodiment. In step S11, the string optimization unit 14 determines, for example, a random initial value for the adversarial string t used in the attack. The string optimization unit 14 also initializes the index j of the dataset to 1.

[0038] In step S12, the string optimization unit 14 inserts the adversarial string t at any location in the j-th prompt of the dataset. Here, as an example, it is inserted at the end of the prompt, p (j) adv =[p (j) Let ||t] represent the concatenation of two strings.

[0039] In step S13, the string optimization unit 14 calculates the similarity σ(p (j) ,p (j) adv The optimization method is optimized to minimize ). The optimization method is not particularly limited and known methods can be applied, but as a concrete example of implementation, when embedding a string within the function σ, the string optimization unit 14 calculates the gradient with respect to the model E and optimizes t using backpropagation.

[0040] In step S14, the string optimization unit 14 changes the index j from j ← j + 1. In step S15, the string optimization unit 14 determines whether j exceeds the size of dataset D, that is, whether all data in dataset D has been processed. If the determination is YES, the process moves to step S16; if the determination is NO, the process returns to step S12.

[0041] In step S16, the string optimization unit 14 constructs a crafted g using the updated adversarial string t, with g(·)=[·||t].

[0042] Furthermore, steps S12 to S15 may be performed in batches, which are groups of data points from the dataset. Also, the methods proposed in Patent Documents 1 to 3 may be applied to these steps.

[0043] Furthermore, since an attacker may modify database S, a specially crafted dataset g may be used to create a dataset to be included in database S in order to assess its security against this poisoning attack.

[0044] Figure 4 is a flowchart illustrating the procedure for updating the database as a pre-processing step for safety evaluation in this embodiment. In step S21, the database update unit 15 prompts any prompt p poison And an invalid response to this prompt r poison Specifically, the database update unit 15 accepts input such as multiple sample data or data files.

[0045] In step S22, the database update unit 15 uses the trick g to p poison ′=g(p poison ) update the prompt as, and data s=(p poison ',r poison Add ) to database S.

[0046] In step S23, the database update unit 15 determines whether it has processed all the data prepared in step S21. If the determination is YES, the process moves to step S24; if the determination is NO, the process returns to step S22.

[0047] In step S24, the control unit 10 uses the updated database S' as the database for search extension generation and executes the safety evaluation process (Figure 2).

[0048] According to this embodiment, the security evaluation device 1 prepares a normal response to a prompt as test data, calculates the similarity between the response generated by search extension to this prompt and the response in the test data, and similarly calculates the similarity after applying a predetermined modification to this prompt. By accumulating these results with multiple test data, a first value and a second value indicating the accuracy under normal conditions and the accuracy when attacked are calculated, respectively. Therefore, the safety evaluation device 1 can quantitatively evaluate how much performance deteriorates when an adversarial prompt is input during search extension generation, based on a safety evaluation value corresponding to the amount of decrease from the first value to the second value.

[0049] The security evaluation device 1 can anticipate more powerful attacks by generating adversarial strings used to manipulate prompts in a way that minimizes the similarity between the prompt and the target prompt before and after the manipulation. This allows the security evaluation device 1 to more appropriately evaluate the security of search extension generation. For example, by optimizing adversarial strings using backpropagation with gradients applied to a model that embeds strings, a suitable string can be obtained as an adversarial string. Furthermore, identifying adversarial strings in prompts in this way can lead to measures such as strengthening the system's robustness.

[0050] The security evaluation device 1 accepts a dataset consisting of pairs of prompts and malicious responses, and after manipulating the prompts, adds them to the database for search extension generation in advance. This allows for a more appropriate evaluation of the security of search extension generation, anticipating more powerful attacks.

[0051] Furthermore, this embodiment makes it possible to evaluate the security of adversarial attacks against search extension generation, for example, and thus contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0052] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments described above are merely a list of the most preferred effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0053] The safety evaluation method by the safety evaluation device 1 is implemented by software. When implemented by software, the programs constituting this software are installed on an information processing device (computer). These programs may be distributed to users by being recorded on removable media such as a CD-ROM, or by being downloaded to the user's computer via a network. Furthermore, these programs may be provided to the user's computer as a web service via a network without being downloaded. [Explanation of symbols]

[0054] 1. Safety evaluation device 10 Control Unit 11 Normal accuracy calculation section 12. Accuracy calculation unit during attack 13. Evaluation Result Output Unit 14 String Optimization Section 15. Database Update Section 20 Memory section

Claims

1. A normal accuracy calculation unit calculates a first value indicating the accuracy of the search extension generation by accumulating the semantic similarity between the response generated by the search extension generation to be evaluated for a prompt in the dataset and the expected normal response for the prompt, An attack accuracy calculation unit calculates a second value indicating the accuracy of the search extension generation when an attack occurs, by accumulating the semantic similarity between the response generated by the search extension generation and the normal response to an attack prompt that has been subjected to a predetermined modification of the prompt, A safety evaluation device comprising: an evaluation result output unit that outputs a safety evaluation value of the search extension generation corresponding to the amount of decrease from the first value to the second value.

2. The aforementioned predetermined manipulation is an operation to insert an adversarial string into the prompt, The safety evaluation device according to claim 1, further comprising a string optimization unit that updates the adversarial string to minimize the similarity between the prompt on which the operation was performed and the prompt before the operation was performed.

3. The security evaluation device according to claim 1 or claim 2, further comprising a database update unit that receives a pair of a prompt and a corresponding malicious response, and adds the pair of the attack prompt, which has been modified as described above, and the malicious response to the database for generating the search extension in advance.

4. Computers The normal accuracy calculation unit calculates a first value indicating the accuracy of the search extension generation by accumulating the semantic similarity between the response generated by the search extension generation to be evaluated for a prompt in the dataset and the expected normal response to the prompt. The attack accuracy calculation unit calculates a second value indicating the accuracy of the search extension generation when an attack is performed, by accumulating the semantic similarity between the response generated by the search extension generation and the normal response to an attack prompt that has been subjected to a predetermined modification, A safety evaluation method that outputs a safety evaluation value of the search extension generation corresponding to the amount of decrease from the first value to the second value, using an evaluation result output unit.

5. A safety evaluation program for causing a computer to function as a safety evaluation device according to claim 1 or claim 2.

Images