Preventing attacks on generative models

Abstract

A computer-implemented method of preventing immediate injection attacks against generative models is provided. Input data is received, and a first cue portion is generated from the input data. First, second, and third instructions are received that respectively instruct the generative model to perform a task based on the first cue portion, notify the generative model of a boundary of the first cue portion, and instruct the generative model to ignore any instructions present in the first cue portion. A hint for the generative model is generated from the first hint portion and the first instruction, the second instruction, and the third instruction.

Description

Background Technology

[0001] Large Language Models (LLMs), or “foundational models,” are gaining popularity. These models are examples of generative models and typically employ a transformer architecture. LLMs are trained on enormous amounts of data, including a wide variety of datasets. For example, GPT-4 (Generative Pretrained Transformer 4), developed by OpenAI®, has 1.76 trillion parameters and is trained on approximately 13 trillion tokens. An LLM receives text input (called “tips”) in the form of natural language instructions and generates text as a response. The sheer volume of training data means that LLMs can be used for a wide range of tasks, including code generation, automatic generalization, question answering, and more.

[0002] In some cases, users interact directly with the LLM by writing and submitting their own prompts. For example, the ChatGPT model has a web-based interface that allows users to input prompts, or users can access the LLM via a suitable application programming interface (API) endpoint. However, an increasing number of LLMs and other generative models are being integrated into other applications, referred to in this paper as “indirect prompts.”

[0003] Such applications can utilize LLM for a wide range of tasks. Typically, the application stores template prompts (called “meta prompts”) that include instructions for the LLM, and then populates these prompts with data accessible to the application. For example, a chat application integrated into a web browser can summarize web pages opened in other tabs of the browser by retrieving text from a website and including it in a meta prompt. In another example, a chat application integrated into a banking application could, for instance, retrieve transaction data from a transaction database and provide it to the LLM for question-and-answer or summarization. In yet another example, a virtual assistant integrated into an email client can help summarize emails or draft responses by sending data extracted from emails as part of a prompt to the LLM. Beyond providing data to the LLM, such applications can also act based on the LLM’s responses—for example, automatically sending emails based on context, initiating banking transactions, and so on. Summary of the Invention

[0004] According to one aspect of this disclosure, a computer-implemented method is provided, comprising: receiving input data; generating a first prompt portion from the input data; receiving a first instruction instructing a generative model to perform a task based on the first prompt portion; receiving a second instruction notifying the generative model of the boundaries of the first prompt portion; receiving a third instruction instructing the generative model to ignore any instructions present in the first prompt portion, the first instruction, the second instruction, and the third instruction forming a second prompt portion; and generating a prompt for the generative model, the prompt including the first prompt portion and the second prompt portion.

[0005] A first hint portion, corresponding to the unsafe part of the prompt, can be generated from the input data in several ways. In some examples, transformations are applied to the input data to generate the first hint portion. Examples of transformations include interleaving special characters in the input data (e.g., replacing whitespace characters) and encoding the input data. A second instruction can interpret the type of transformation applied, allowing the generative model to determine which part of the prompt is unsafe. In other examples, the first hint portion is enclosed in delimiting characters that signal the start and end of the first hint portion, and the second hint instruction describes the delimiting characters. The second instruction can form part of a template hint retrieved from memory.

[0006] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The claimed subject matter is also not limited to implementations that address any or all of the shortcomings mentioned herein. Attached Figure Description

[0007] To aid in understanding this disclosure and to illustrate how embodiments may be implemented, reference is made to the accompanying drawings by way of example, in which: Figure 1 It is a schematic block diagram of an example environment that includes various aspects of the system according to this disclosure.

[0008] Figure 2 This is a diagram of the template prompt.

[0009] Figure 3 It is a schematic diagram based on various aspects of this disclosure, in which input data is marked using special characters.

[0010] Figure 4 Based on Figure 3 The following is a schematic flowchart illustrating the method for generating example prompts.

[0011] Figure 5 This is another schematic diagram illustrating various aspects of this disclosure, in which input data is encoded.

[0012] Figure 6 Based on Figure 5 The following is a schematic flowchart illustrating the method for generating example prompts.

[0013] Figure 7 This is another illustration based on various aspects of this disclosure, in which input data is enclosed in delimiters.

[0014] Figure 8A This is a graph showing the impact of delimiters on the attack success rate in a generalization task.

[0015] Figure 8B This is a chart showing the impact of data labeling on the attack success rate in the overview task.

[0016] Figure 8C This is a graph showing the impact of data tagging on the success rate of attacks in document question-answering tasks.

[0017] Figure 8D It is a graph illustrating the impact of data labeling on task accuracy across a range of natural language processing tasks. Figure 8E This is a graph showing the impact of encoded input data on the success rate of attacks in generalization and document question answering.

[0018] Figure 9 This is a schematic block diagram of another example system based on various aspects of this disclosure.

[0019] Figure 10 This is a schematic block diagram of an example computing system. Detailed Implementation

[0020] While generative models (such as LLMs) can be fed into a wide range of legitimate tasks, the nature of their training data means they can also provide responses unrelated to legitimate or legal activities, such as providing recipes for anesthetics, generating malicious code, generating phishing emails, and generating racist or other offensive outputs. Typically, LLMs will have appropriate security measures to moderate the generated content and prevent them from returning content that does not comply with policies or restrictions that were part of the LLM's original design.

[0021] However, attackers attempt to bypass these restrictions. A common technique is to "jailbreak" the LLM by including (or "injecting") information into a prompt that bypasses security measures. For example, an attacker could include instructions in the prompt to make the LLM role-play as a malicious actor, etc. This attack works in the context of direct prompts, and attacks in this context are referred to below as direct prompt injection.

[0022] This type of attack can also be applied in the context of indirect hints, known as indirect hint injection. While attackers cannot access the text of the meta hint in this case, they can access application-accessible data that is to be included in the meta hint. For example, attackers can insert malicious instructions into the content of a webpage, the text of an email, etc., causing the application to retrieve them and include them in the meta hint to jailbreak the LLM. In cases where the application acts based on the LLM output, the attack may expose sensitive data outside the application or allow the application to take malicious actions. In such situations, the LLM user may be unaware of the attack and be an innocent bystander. As LLM systems become more flexible in terms of plugins, skills, and capabilities, the dangers of indirect hint injection become increasingly severe.

[0023] Generative models respond to prompts, which consist of instructions in natural language. Similarly, the data that generative models operate on is also provided in natural language within the prompts. For example, a prompt might include instructions summarizing text, followed by data in the form of summarized text. From the generative model's perspective, there is no clear boundary between instructions and data—they are both part of the natural language input to the model. This is a structural constraint of generative models, which operate on an unbounded stream of lexical terms to generate responses.

[0024] Therefore, if the data itself contains instructions (including malicious ones), the model can act on those instructions. In the context of indirect prompts, data from the prompt is retrieved from a data source, which could be an external source (e.g., a publicly accessible website) or another source within the application (e.g., the text of an email). Either way, the data is susceptible to being injected with malicious instructions. In other cases, the data may harmlessly contain instructions; for example, the text of an email may include instructions directed at the recipient. Generative models acting on these harmless instructions can have unintended consequences and cause harm to the user.

[0025] In summary, the examples disclosed herein provide techniques for enabling generative models to identify portions of prompts based on input data from potentially insecure sources and to ignore any instructions that may be present in those portions of the prompts. These techniques provide defense against malicious instructions inserted into the input data, thereby preventing jailbreak attacks and other attacks against generative models.

[0026] Figure 1 An example of an environment in which this disclosure may operate is shown 1.

[0027] Environment 1 includes Large Language Model (LLM) 201, which is an example of a generative model. LLM 201 is a trained language model based on a transformer deep learning network. LLM 201 is trained on a very large corpus (e.g., on the order of billions of lexical units) and can generate text or data in response to input in the form of prompts.

[0028] Suitable LLM 201 examples include OpenAI's General Pre-trained Transformer (GPT) models, such as GPT-3, GPT-3.5 turbo, or GPT-4. However, various LLM 201 alternatives are available.

[0029] The LLM 201 operates within a suitable computer system 200. For example, the LLM 201 may be stored in a suitable data center and / or as part of a cloud computing environment or other distributed environment. The LLM 201 may be accessed via a suitable API (Application Programming Interface), such as through a network N. The network may include any suitable link, including wired and wireless links, as well as local area networks (LANs) and wide area networks (WANs).

[0030] Environment 1 also includes a computer system 100 configured to interact with the LLM 201. System 100 is configured to generate appropriate prompts 202 and submit them to the LLM 201 via a network. Furthermore, system 100 is configured to receive responses 203 (also referred to as "complete") from the LLM 201.

[0031] Computer system 100 also includes a controller 110 and a storage device 120. The controller 110 includes a processor or other computing unit configured to execute instructions stored in the storage device 120 to perform operations and processes discussed further in detail herein. The storage device 120 may include volatile and non-volatile memory. System 100 may further include a suitable user interface 130.

[0032] Computer system 100 includes application 140, which is configured to generate prompts 202. In other words, application 140 is an LLM-integrated application. Therefore, system 100 is an example of an indirect prompting system. Application 140 may be one or more software programs (e.g., stored in storage device 120) that include instructions executable by controller 110.

[0033] Application 140 can utilize template prompts 121 as the basis for prompts 202 provided to LLM 201. Each template prompt 121 may include predefined instructions to guide LLM 201 in performing a task. Examples of tasks include summarizing, question answering, translation, code generation, etc. Template prompts 121 have fields populated with data by application 140, referred to below as slots. The data is the subject of the task to be performed. For example, in the context of summarizing, the data may be the text to be summarized.

[0034] Figure 2 Template prompt 300 is illustrated in a schematic form to facilitate understanding of indirect prompting techniques.

[0035] Hint 300 is used for generalization. Hint 300 includes instructions 310 that guide the LLM 201 by interpreting the task. Instructions 310 are pre-prepared in the sense that they are instructions prepared in advance by a human (e.g., a system developer or hint engineer). Because they have been prepared in advance by a trusted party, these instructions are known not to be malicious. In other words, the instructions are the trusted or secure part of the hint.

[0036] In addition to explaining the relevant tasks, instruction 310 can provide other relevant guidance to LLM 201. For example, instruction 310 can specify the expected format of the output, relevant factors or information that LLM 201 should consider when generating response 203, and information that should not be included in response 203.

[0037] Slot 320, represented by placeholder {{text}}, is the portion of the prompt filled by application 140.

[0038] The data used to populate slot 320 originates from a data source. Example data sources include sources external to system 100, such as resources accessible via a network. This includes, for example, web pages available via the World Wide Web. In other examples, the data source could be user U, who provides input via user interface 130 (e.g., the user interacting with a chatbot). In other examples, the data source is application data 122, such as data from a database associated with application 140.

[0039] Application 140 is configured to retrieve data from a data source and insert it into a slot. For example, application 140 may query a suitable database, retrieve a web page, or receive user input to be inserted into the slot.

[0040] Inserting data into slot 320 may involve retrieving one or more strings from storage device 120, such as the string including instruction 310. It may also involve generating one or more strings, for example, by converting data retrieved from a data source into one or more strings. The resulting strings can then be concatenated or otherwise combined to form a prompt. For example, each string may be loaded into memory and combined to form a larger string including the prompt. This prompt is then stored in memory (e.g., in volatile memory) before being sent, for example, via an API call to LLM 201.

[0041] In response to the transmission of prompt 300, LLM 201 provides response 203 that forms the result of the task. For example, response 203 includes a summary of the data included in slot 320.

[0042] Based on the response, application 140 may take any appropriate action depending on the context. For example, it may display the result of the task to user U via UI 130, and / or perform other actions. Examples of other actions include storing data, transferring data (including sending emails), performing bank transactions, executing code generated as a result of the task, etc. It should be understood that the relevant actions will depend on the purpose of application 140.

[0043] Examples of this disclosure will now be discussed, which provide a means of distinguishing between the data in the fill slot and the instructions in the prompt.

[0044] Figure 3 A prompt 400 is shown as an example according to this disclosure. Prompt 400 generally follows the structure of prompt 300 because it contains instructions 410 and includes slots 420 for inserting data retrieved from a potentially untrusted data source.

[0045] More specifically, prompt 400 includes a first instruction 411, which generally corresponds to instruction 310 discussed above. That is, the first instruction 411 is provided to instruct the LLM 201 to perform a task.

[0046] Furthermore, the prompt 400 includes a second instruction 412. The second instruction 412 helps the LLM 201 determine which parts of the prompt correspond to safe instructions and which parts correspond to potentially unsafe data. In other words, the second instruction informs the generative model of the boundaries of the unsafe parts of the prompt.

[0047] Instruction 412 explains the interleaving of data with special characters, in this case, the caret character ("^") between words in the text to be summarized. This technique can be called the "data marking" method.

[0048] Furthermore, prompt 400 includes a third instruction 413, which instructs LLM 201 to ignore any instructions present in the unsafe portion of the prompt. For example, the third instruction 413 states that model 201 should not change its task or objective in response to text present in the unsafe portion of prompt 400.

[0049] The data in slot 420 is interspersed with special characters, as discussed in more detail below. It has been found that the generative model can interpret the data in slot 420, and despite the presence of special characters, it has almost no impact on the execution of the task specified in instruction 410. See below for reference. Figure 8A -E discusses the experiments that prove this point. It should be understood that... Figure 3 Only a portion of the data in slot 420 is shown to illustrate the concept of interleaved special characters.

[0050] Instructions 411-413 can be provided in any order in prompt 400. For example, in Figure 3 In this example, a third instruction 413 is provided before the second instruction 412. In other examples, instructions may be provided after the data. Furthermore, the instructions do not need to be consecutive parts of the prompt 400; for example, the first instruction 411 may be interleaved with the second instruction 412 or the third instruction 413, or some parts of the instructions may be repeated.

[0051] Figure 4 An example prompt generation method is shown, where prompt 400 is filled with data interleaved with special characters.

[0052] In step S401, input data is received. As described above, data can be retrieved from a suitable data source.

[0053] In step S402, a prompt portion is generated from the input data. In this case, the input data is transformed by interleaving lexical units (i.e., words or other semantic units that can be interpreted by the LLM201) with special characters.

[0054] For example, the input data can be lexicalized using appropriate lexicalization techniques to generate an ordered list of words or other similar data structures stored in memory. The string is then generated by alternately retrieving words from the list, concatenating the retrieved words with special characters, and repeating this process until all words in the list are appended to the string. Alternatively, a find and replace process can be performed on the input data to replace whitespace characters with special characters.

[0055] In examples where the input data was not initially stored in a text-based format, the input data can be converted to text before being interleaved with special characters. For example, tabular data received from a database can be converted to a comma-separated format or some other appropriate text-based format. Other similar conversions can be applied to other kinds of structured data, such as graphical data stored in a graph database.

[0056] exist Figure 3 In the example, the special character is the caret, but it should be understood that any appropriate special character can be chosen. It is advantageous to choose a special character that is unlikely to appear normally in the input data.

[0057] In steps S403, S404, and S405, first, second, and third instructions 411-413 are received. For example, instruction 410 may form part of a template prompt stored in non-volatile memory, which is then retrieved and stored in volatile memory.

[0058] In step S406, instruction 410 and the transformed input data are combined to generate a prompt. Specifically, the string of instruction 410 and the string of transformed input data are concatenated and stored in memory.

[0059] The prompt can then be sent to LLM 201, and a response can be received as described above.

[0060] Figure 5 Another example prompt 500 is shown, which differs from prompt 400 in its application of transformations to the input data included in prompt 400. In particular, it has been found that many LLMs (including GPT 3, GPT 3.5 turbo, and GPT-4) are capable of natively decoding data. That is, an LLM can receive encoded data in a prompt and implicitly interpret that encoded data in the same way as unencoded plaintext text.

[0061] Hint 500 is structurally similar to Hint 400. Therefore, only the differences will be discussed in detail.

[0062] The prompts include instructions 511 and 513, similar to instructions 411 and 413. However, instructions 512, which specify the boundaries of the data 520 forming the insecure portion of the prompt 500, differ from instructions 412 in that they explain to model 201 that data 520 has been encoded. Figure 5 In the example, the instruction declares that the data is encoded in base64. Then, the data 520 in the prompt is provided in base64 encoding.

[0063] Figure 6An example prompt generation method is shown, where prompt 500 is filled with encoded data. This method includes elements corresponding to those mentioned above. Figure 4 Steps S401 and S403-406 discussed here are steps S601 and S603-S606, and therefore will not be discussed in detail hereafter.

[0064] According to step S602, the unsafe portion of the prompt is generated from the input data by encoding the input data. The received input data is encoded into a target encoding scheme using an appropriate algorithm for encoding the data. As mentioned above, if the data is not in a text-based format, it can be converted to text before encoding.

[0065] exist Figure 5 In the example, the encoding is base64. The Base64 encoding scheme represents binary data as a 24-bit sequence, with each sequence corresponding to four 6-bit base64 digits. Input text data is stored in formats such as ASCII or Unicode, where the encoding is applied to the binary representation of the text in this format.

[0066] It should be understood that various encoding formats are within the scope of this disclosure. For example, data can be encoded in different bases, such as hexadecimal, octal, or binary. In other examples, encryption techniques can be applied to encode data. For example, a substitution cipher can be used to encode data. The mapping between input bits and encoded bits can be described in instruction 512. In some examples, a substitution cipher with a known mapping can be used, allowing LLM 201 to infer the mapping without needing to describe it in the instructions. Such an example is the ROT13 cipher, which involves replacing each letter character with the letter 13 positions after it in the alphabet. Preferably, the encoding format prevents attackers from easily injecting data into the input data, as the input data will form plaintext instructions once encoded. For example, in the ROT13 example, malicious instructions already encoded in ROT13 can be inserted, and when they are further encoded to be included in the prompt, they will be presented in plaintext.

[0067] Figure 7 Another example prompt, 600, is shown, which differs from prompts 400 and 500 in its handling of the input data included within it. Specifically, in this example, the input data is enclosed in special characters that act as separators. For example, the character "<<" is used to indicate the beginning of the input data, and the character ">>" is used to indicate the end of the input data.

[0068] Hint 600 is structurally similar to hints 400 and 500. Therefore, only the differences will be discussed in detail.

[0069] The prompts include instructions 611 and 613, which are similar to the corresponding instructions in the previous prompts. However, the instructions 612, which specify the boundary of the unsafe portion 620 of the prompt 600, differ in that they explain to the model 201 that the data is enclosed in delimiting characters.

[0070] The process of generating error message 600 is usually similar to Figure 4 and Figure 6 The processes described herein. However, the portion of Prompt 600 that includes insecure input data can be generated by including delimiters in a template prompt around the slot, and then inserting the input data into the slot. That is, there is no need for a separate operation involving additional delimiters.

[0071] Despite Figure 7 In the example, the delimiting special characters are "<<" and ">>", but it should be understood that any appropriate symbol or sequence of symbols can be used. For example, XML type tags (e.g., "...") can be used. <start>"and" <end>"or" <data> "and"< / data> (”).

[0072] Turn now Figure 8A -E, published several experiments demonstrating the effectiveness of the techniques discussed in this paper.

[0073] Typically, in experiments, the input data is injected with an attack payload that includes instructions designed to make the LLM 201 return only predetermined keywords. That is, in experiments, the input data includes instructions that urge the model to forget its previous instructions and respond by returning only a single specific keyword. Attack Success Rate (ASR) represents the percentage of successful attacks, where an attack is considered successful if the model returns only the specific keyword. This concept is further illustrated in the table below.

[0074]

[0075] In the third example, the model provides both a valid response to the generalization task and mentions keywords. In this case, the model "notices" the attack text but does not "fall for" the attack, so this is not considered a success in the experiments explained below.

[0076] As for the test data, the synthetic dataset of 1,000 documents contains a hint injection attack designed to make LLM 201 return a predetermined keyword.

[0077] Figure 8A The following ASRs are shown: a baseline prompt, which does not provide measures to prevent prompt injection attacks; an "instruction" prompt, where the prompt only includes instructions to ignore any other instructions present in the data; and a prompt regarding... Figure 7 The discussion uses delimiter hints. In this case, the underlying model is GPT 3.5 Turbo, and the task is a generalization task. The use of delimiters reduces ASR by approximately half.

[0078] Figure 8B Baseline tips and applications are shown above. Figure 3 and Figure 4 A comparison of suggestions for data labeling methods is discussed. The task in this experiment is again a generalization task. The experiment shows that using GPT 3.5 Turbo reduces ASR to 3.10%, and using the GPT-3-text-003 model reduces it to 0.00%.

[0079] Figure 8C Provided with Figure 8B A similar comparison is shown, but in this example, the task is a document question answering task. This method significantly reduces ASR in GPT 3.5 Turbo, GPT-4, and GPT-3-text-003 models.

[0080] Figure 8D This demonstrates that the data labeling method does not significantly affect the LLM's ability to perform low-level tasks. Specifically, Figure 8D The data labeling method is shown to produce similar accuracy to baseline prompts on four different tasks and associated datasets. These are the Stanford Question Answering Dataset (SQuAD) question answering task (see https: / / rajpurkar.github.io / SQuAD-explorer / ), the IMDB sentiment analysis task using a large movie review dataset (see https: / / ai.stanford.edu / ~amaas / data / sentiment / ), the SuperGLUE word task in context, and the SuperGLUE BoolQ task (see https: / / super.gluebenchmark.com / tasks).

[0081] at last, Figure 8E The above shows about Figure 5 and Figure 6 The performance of baseline hints, data tagging methods, and encoding methods is discussed. Using GPT3.5-Turbo, the encoding techniques resulted in the lowest ASR across different tasks. In document summarization, ASR was reduced to 0.0%, and in question answering, ASR was reduced to 1.8%.

[0082] Figure 9 A further application of the techniques discussed herein is illustrated. In this example, system 1100 is configured to receive input instructions from user U, rather than forming an indirect prompting system that includes application 140 performing a pre-defined task using template prompts. This makes system 1100 an example of a direct prompting system, although, as discussed in more detail below, unadulterated input instructions are not directly passed to LLM 201.

[0083] System 1100 includes components similar to those in system 100, wherein the reference numerals of the corresponding components are relative to those in the attached figures. Figure 2 The reference numerals in the accompanying drawings are increased by 1000. That is, system 1100 includes a controller 1110, a storage device 1120, and a user interface 1130 that are consistent with similar components as system 100. Similar components will not be discussed in detail, but the focus will be on the differences between system 100 and 1100.

[0084] exist Figure 9 In the example, user U can provide input instructions 701. User U can, for example, enter instructions into a suitable dialog box in user interface 1130. Additionally, user U can provide input data 721. For example, the user can use user interface 1140 to select a data source, such as by selecting a file using a suitable dialog box, dragging a file into interface 1140, or entering a webpage URL.

[0085] For example, input data 721 can be a file (e.g., structured data such as a spreadsheet or text document), and input instructions 701 can be user questions about the file content, instructions for reformatting or rewriting the file content, etc.

[0086] In response to receiving input data 721 and input instruction 701, system 1100 constructs a prompt 700. The prompt 700 includes the input instruction 701 and second and third instructions 712 and 713, similar to those discussed above, which respectively explain the boundaries of the input data and the instructions that model 201 should not follow in the input data. Part 720 of the prompt 700 is generated from the input data 721 according to one of the techniques discussed herein.

[0087] Therefore, attacks based on malicious instructions present in the input data 721 can be avoided, while still allowing users U to flexibly specify their own input-related instructions.

[0088] Within the scope of this disclosure, a range of modifications and variations can be made to the examples discussed above. While the techniques described above involve highlighting input data by making data boundaries clear, these techniques can alternatively be applied to instructions. That is, instructions can be defined, data-tagged, encoded, or otherwise transformed so that the LLM is guided to follow only the instructions in the prompt section where such transformations have been applied.

[0089] In the example above, this technique was applied to an LLM that receives text prompts and responds with text. However, the technique is not limited to this type of model and can be applied to models that operate with different input and output modalities, including multimodal models.

[0090] For example, these techniques can be readily applied to models that receive text input and produce output in another modality (e.g., images, videos, audio, mechanical control signals, etc.). In such examples, it might be desirable to mitigate attacks that cause the model to provide aggressive or dangerous outputs in the relevant modality.

[0091] Alternatively or additionally, these techniques can be applied to models that receive cues that include inputs other than text, such as structured data, images, videos, audio, etc. For example, such a model can receive cues that include some text components and some inputs of one of the aforementioned modalities. These techniques can be applied to the text components, or, where applicable, to the non-text components. For example, delimiters can be provided around non-text content. Therefore, the term "cue" as used herein can be broadly considered as a reference to the input of the generative model that enables it to produce output, and is not limited to text cues.

[0092] While the examples above involve prompts with slots to be filled with input data, it should be understood that in other examples, prompts may include multiple slots that can be filled with data from different data sources. A wide range of user input techniques are envisioned, and this disclosure is not limited to the user input components discussed above.

[0093] The described techniques can be combined; for example, text can be data-tagged before encoding and then inserted between delimiters.

[0094] Advantageously, the above examples provide a means to prevent cue injection attacks by highlighting portions of the input that include untrusted data. Therefore, generative models are better able to ignore any malicious instructions included in untrusted data. Figure 8A As shown in -E, these techniques significantly reduce the success rate of attacks while maintaining effectiveness relevant to the task at hand.

[0095] Figure 10 A non-limiting example of a computing system 1200 capable of performing one or more of the methods and processes described above is schematically illustrated. The computing system 1200 is shown in a simplified form. The computing system 1200 may be the computer system 100 or 1100 described above, or any other computer device discussed herein. The computing system 1200 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices (such as smartwatches and head-mounted augmented reality devices).

[0096] The computing system 1200 includes a logic processor 1202, volatile memory 1204, and non-volatile storage device 1206. The computing system 1200 may optionally include a display subsystem 1208, an input subsystem 1210, a communication subsystem 1212, and / or... Figure 10 Other components not shown.

[0097] The logic processor 1202 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0098] A logic processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of logic processor 1202 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic processor may optionally be distributed across two or more separate devices, which may be located remotely and / or configured for coordinated processing. Aspects of the logic processor may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In this context, it will be understood that these virtualized aspects run on different physical logic processors on various different machines.

[0099] The non-volatile storage device 1206 includes one or more physical devices configured to store instructions executable by a logic processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 1206 can be changed, for example, to store different data.

[0100] Non-volatile storage device 1206 may include removable and / or built-in physical devices. Non-volatile storage device 1206 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or magnetic memory (e.g., hard disk drive) or other mass storage technologies. Non-volatile storage device 1206 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that non-volatile storage device 1206 is configured to retain instructions even when the non-volatile storage device 1206 is powered off.

[0101] Volatile memory 1204 may include physical devices, including random access memory. Volatile memory 1204 is typically used by logic processor 1202 to temporarily store information during the processing of software instructions. It should be understood that volatile memory 1204 typically does not continue storing instructions when power is lost.

[0102] The logic processor 1202, volatile memory 1204, and non-volatile storage device 1206 can be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SoCs), and complex programmable logic devices (CPLDs).

[0103] The terms "module," "program," and "engine" can be used to describe aspects of a computing system 1200 typically implemented in software by a processor to perform specific functions using portions of volatile memory, involving transformation processing specifically configured for the processor to perform these functions. Thus, a module, program, or engine can be instantiated by executing instructions held by non-volatile storage device 1206 via a portion of volatile memory 1202 using the logic processor 1202. It should be understood that different modules, programs, and / or engines can be instantiated from the same applications, services, code blocks, objects, libraries, routines, APIs, functions, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass individuals or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0104] When included, the display subsystem 1208 can be used to present a visual representation of data stored by the non-volatile storage device 1206. The visual representation may take the form of a graphical user interface (GUI). Because the methods and processes described herein change the data stored by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 1208 can also be transformed to visually represent changes in the underlying data. The display subsystem 1208 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 1202, the volatile memory 1204, and / or the non-volatile storage device 1206 in a shared housing, or such display devices may be peripheral display devices.

[0105] When included, the input subsystem 1210 may include one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller, or interface with them. In some embodiments, the input subsystem may include or interface with selected Natural User Input (NUI) components. Such components may be integrated or peripheral, and the translation and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0106] When included, the communication subsystem 1212 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 1212 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem may allow the computing system 1200 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0107] Additional exemplary features of this disclosure are described below.

[0108] According to a first aspect of this disclosure, a computer-implemented method is provided, comprising: receiving input data; generating a first prompt portion from the input data; receiving a first instruction instructing a generative model to perform a task based on the first prompt portion; receiving a second instruction notifying the generative model of the boundaries of the first prompt portion; receiving a third instruction instructing the generative model to ignore any instructions present in the first prompt portion, the first instruction, the second instruction, and the third instruction forming a second prompt portion; and generating a prompt for the generative model, the prompt including the first prompt portion and the second prompt portion.

[0109] The following describes other additional optional features of the first aspect. Each of these optional features may be presented as dependent on any other optional feature. That is, if presented as a dependent claim, the following features will depend on any of the preceding claims.

[0110] Generating the first prompt may include applying a transformation to the input data to generate transformed input data. The second instruction may include an interpretation of the transformation.

[0111] Applying the transformation may include interleaving the input data with special characters. Interleaving the input data with special characters may include lexicalizing the input data into multiple tokens and forming a string comprising each token separated by a special character. The special character may be a caret.

[0112] Application transformations can include encoding the input data. The input data can be encoded using an encoding scheme. The encoding scheme can be a basic transformation encoding scheme, such as base64. The encoding scheme can also be encryption.

[0113] Generating the first prompt may include enclosing the input data within a delimiter. The second instruction can interpret the delimiter.

[0114] The method may include: retrieving a template prompt that includes at least a second instruction and a third instruction, where appropriate including a second prompt portion. The method may also include: populating fields in the template prompt using a first prompt portion.

[0115] The method may include: receiving user input including a first instruction.

[0116] The method may include retrieving input data from a database. The method may also include retrieving input data from a webpage.

[0117] The task can be one of the following: summarizing, question answering, translation, or code generation.

[0118] The method may include: providing a prompt as input to a generative model. The method may include: receiving a response from the generative model, the response including the result of the task. The method may include: displaying the result of the task on a user interface. The method may include: performing an action based on the result of the task. The action may be one of the following: sending an email, executing a bank transaction, or executing code included in the task result.

[0119] Generative models can be configured to receive text prompts. Generative models can be configured to output text responses. Generative models can be large language models. Generative models can be multimodal models.

[0120] The first part of the prompt can indicate the unreliable portion of the prompt. The second part of the prompt can indicate the reliable portion of the prompt.

[0121] According to a second aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores a template hint for a generative model, the template hint comprising: a field for receiving a first hint portion generated from input data; a first instruction instructing the generative model to perform a task based on the first hint portion; a second instruction informing the generative model of the boundaries of the first hint portion; and a third instruction instructing the generative model to ignore any instructions present in the first hint portion.

[0122] The following describes additional optional features of the second aspect. Each of these optional features may be presented as dependent on any other optional feature. That is, if presented as a dependent claim, the following features will depend on any of the preceding claims.

[0123] The second instruction may include an interpretation of the transformations applied to the input data to generate the first prompt section. The transformation may be interleaving the input data with special characters. The transformation may be encoding the input data. The second instruction may interpret the separators used to close the first prompt section. The template prompt may include separators.

[0124] Furthermore, any feature of the first aspect can be combined with the second aspect in any combination.

[0125] According to another aspect of this disclosure, a computer system is provided, including a processor and a memory, the memory storing instructions that, when executed by the processor, cause the system to perform any method defined herein.

[0126] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided having instructions recorded thereon that, when executed by a computer device, cause the computer device to perform any of the methods set forth herein.

[0127] According to another aspect of this disclosure, a computer program product is provided, which includes instructions that, when executed by a computer device, cause the computer device to perform any of the methods set forth herein.

[0128] Although at least some aspects of the embodiments described herein with reference to the accompanying drawings include computer processes executed in a processing system or processor, aspects of the invention are also extended to computer programs, particularly computer programs on or in a carrier, suitable for practicing aspects of the invention. The program can be in the form of non-transitory source code, object code, intermediate source code, and object code, such as in a partially compiled form, or in any other non-transitory form suitable for implementing processes according to aspects of the invention. The carrier can be any entity or device capable of carrying the program. For example, the carrier can include storage media such as solid-state drives (SSDs) or other semiconductor-based RAM; ROMs such as CD ROMs or semiconductor ROMs; magnetic recording media such as floppy disks or hard disks; general optical storage devices; and so on.

[0129] The examples described herein should be understood as illustrative examples of embodiments of the invention. Further embodiments and examples are contemplated. Any feature described with respect to any example or embodiment may be used alone or in combination with other features. Furthermore, any feature described with respect to any example or embodiment may also be used in combination with one or more features of any other example or embodiment, or any combination of any other example or embodiment. Moreover, equivalents and modifications not described herein may be employed within the scope of the examples of the invention as defined in the claims. The labels "first," "second," "third," etc., used herein are only for distinguishing similar elements and are not intended to imply a specific order.< / end> < / start>

Claims

1. A computer-implemented method, comprising: Receive input data; A first prompt section is generated from the input data; The receiving instruction generative model executes a first instruction based on the first prompt portion to perform a task; Receive a second instruction to notify the generative model of the boundary of the first prompt portion; Receive a third instruction instructing the generative model to ignore any instructions present in the first prompting section, wherein the first instruction, the second instruction, and the third instruction form the second prompting section; as well as Generate a prompt for the generative model, the prompt including a first prompt portion and a second prompt portion.

2. The method of claim 1, wherein generating the first prompt portion includes applying a transformation to the input data to generate transformed input data, and the second instruction includes an interpretation of the transformation.

3. The method of claim 2, wherein applying the transformation includes interleaving the input data with special characters.

4. The method according to any of the preceding claims, wherein applying the transformation includes encoding the input data.

5. The method according to any preceding claim, wherein generating the first prompt portion includes enclosing the input data in a delimiter, and wherein the second instruction interprets the delimiter.

6. The method according to any preceding claim, comprising: Retrieve the template prompt that includes the first instruction, the second instruction, and the third instruction.

7. The method according to any preceding claim, comprising: Receive user input including the first instruction.

8. The method according to any of the preceding claims, comprising: Retrieve the input data from the database.

9. The method according to any preceding claim, comprising: Retrieve the input data from the webpage.

10. The method according to any of the preceding claims, wherein the task is one of: summarizing, question answering, translation, or code generation.

11. The method according to any preceding claim, comprising: The prompt is provided as input to the generative model; as well as Receive a response from the generative model, the response including the result of the task.

12. The method of claim 11, comprising: The results of the task are displayed on the user interface.

13. The method of claim 11, comprising: Perform an action based on the result of the task, wherein the action is one of the following: sending an email, executing a bank transaction, or executing code included in the result of the task.

14. A non-transitory computer-readable medium storing template hints for a generative model, the template hints comprising: A field used to receive the first prompt portion generated from the input data; The generative model is instructed to perform a first instruction based on the first prompt portion; A second instruction is given to the generative model to notify the boundary of the first prompt portion; as well as A third instruction instructs the generative model to ignore any instructions present in the first prompt section.

15. A system comprising: A processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, cause the system to perform the method according to any one of claims 1 to 13.

Images