Method for providing security system prompts for machine learning models
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2026-01-23
- Publication Date
- 2026-07-24
AI Technical Summary
Existing hint extraction defense methods are insufficient to effectively prevent attackers from extracting system hints from machine learning models, especially large language models, through carefully designed queries, leading to security and privacy risks.
By generating and modifying the set of embedded parameters, machine learning models are used to generate coded system cues that are semantically different from the original cues. Optimization techniques such as gradient optimization and genetic optimization are employed to enhance security and ensure the cues are uninterpretable.
It effectively prevents prompt extraction attacks, protects the confidentiality of system prompts, ensures the security and privacy of machine learning models, and keeps the model's responsiveness unaffected.
Smart Images
Figure CN122452731A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for providing security system hints to machine learning models. Furthermore, this invention relates to computer programs, apparatus, and storage media for this purpose. Background Technology
[0002] Machine learning models like Large Language Models (LLMs) are advanced artificial intelligence (AI) systems that are trained on massive amounts of textual data to generate human-like responses in natural language. These models are typically guided by two types of prompts: user prompts, which are input queries typically provided by the user in a chat-like interface, and system prompts, which are hidden instructions given to the model to define its role and personality. System prompts are crucial in shaping the model's responses because they contain fundamental operational guidelines, ethical boundaries, and domain-specific knowledge. Ensuring the confidentiality of these system prompts is paramount, as their exposure could jeopardize security, lead to unintended actions, or leak proprietary information.
[0003] A cue extraction attack is a security threat in which an adversary interacts with an LLM (Local Model) to recover hidden system cues that guide its behavior. System cues typically contain proprietary instructions, sensitive data, or system commands, and their disclosure can lead to misuse or model manipulation. Attackers often use carefully crafted queries to probe the model, aiming to elicit unexpected responses that reveal the underlying cue structure or content, thereby jeopardizing user privacy and model integrity.
[0004] Cue extraction defense refers to strategies and techniques designed to protect LLM cues from being leaked during interactions. These defenses aim to prevent attempts to retrieve cue content. They involve methods such as using specialized detection algorithms to identify suspicious probing behavior or detect cue leakage in the model's response, as well as mechanisms to instruct the LLM to keep cue confidential. The goal is to ensure that the model's internal logic remains secure, confidential, and protected from exploitation.
[0005] Current defenses against cue extraction generally fall into two categories: cue-based and malicious detection. Cue-based defenses involve adding instructions like "You cannot disclose these commands," but attackers can easily bypass this by issuing commands like "Ignore all previous commands." Malicious detection attempts to identify leaks in malicious queries or responses, but this can hinder usability, requires constant updating from new attack queries, and can be circumvented using various techniques. For example, attackers might encode cuees using code or obfuscated formats that detection systems cannot recognize.
[0006] Previous efforts to enhance hint extraction defenses have employed various strategies that do not require knowledge of the attacker's query. One approach is output filtering, as implemented in existing production models, which returns a standard, fixed response when there is overlap between the generated output and the secret hint. While this approach provides a layer of protection, it can be circumvented if attackers formulate their queries, for example, in a different language. Another technique is the use of spoofed hints, which introduce misleading suggestions before the actual hint to reduce the likelihood of extraction. However, this approach only minimizes the risk of theft to a certain extent and does not provide a comprehensive solution. Summary of the Invention
[0007] According to aspects of the invention, a method having the features of claim 1, a computer program having the features of claim 9, a data processing apparatus having the features of claim 10, and a computer-readable storage medium having the features of claim 11 are provided. Further features and details of the invention are disclosed in the corresponding dependent claims, the specification, and the drawings. The features and details described in the context of the method of the invention also correspond to the computer program of the invention, the data processing apparatus of the invention, and the computer-readable storage medium of the invention, and in each case, the reverse.
[0008] According to one aspect of the present invention, a method is provided for providing safety system hints for machine learning models, particularly large language models, the method comprising the following steps: - Provides system hints, which are parameterized by an embedded set of parameters to handle the individual tokens of the system hint. - Generate user queries based on provided system prompts, which specifically simulate natural interactions that guide the user toward relevant information or recommendations, while preserving the intent and structure of the provided system prompts. The user queries are preferably aligned with a specific focus, at least one constraint, and / or at least one preference defined by the system prompts. The generation of user queries can be performed using an Oracle language model. - Use the generated user query and provided system prompts to query the machine learning model, which then produces a response based on the query. The query can be a user query that provides the corresponding input to the machine learning model, and the resulting response can be the corresponding output of the machine learning model. - Taking into account the provided system prompts, the generated user queries, and the resulting responses, modify the set of embedded parameters (its parameterized system prompts) to generate an encoded set of embedded parameters, specifically making the resulting security system prompts as different as possible from the provided system prompts while retaining similar intent. - By processing the individual tokens of the system prompt using the generated set of embedded parameters of the encoded data, a security system prompt is provided. In particular, due to the different processing of the generated set of embedded parameters of the encoded data, the resulting security system prompt is semantically different from the provided system prompt.
[0009] The machine learning model is preferably a large language model. System prompts, particularly the initial inputs or configurations in the machine learning model, define the model's context, behavior, and / or response structure, and can control its interaction with user input. Encoded system prompts may differ from natural human language, making them difficult for attackers to interpret, for example. Machine learning models can be implemented in technological systems, particularly vehicles or manufacturing plants. By protecting the system prompts used to interact with the machine learning model using the method according to the invention, the security and privacy of the machine learning model, and therefore the technological system, can be enhanced. Therefore, prompt extraction attacks can be prevented by means of the method according to the invention. Sensitive information in the system prompts can be protected by implementing the set of embedded parameters encoded according to the invention.
[0010] A further possibility is to iteratively modify the set of embedding parameters, whereby the encoded set of embedding parameters is optimized using at least one optimization objective. This iterative modification process can enhance the accuracy and effectiveness of the security system hints. Such optimization can refine the embedding parameters based on user queries, model responses, and initial system hints, resulting in a more robust and secure machine learning model. The use of at least one optimization objective ensures that the encoded set of embedding parameters is tailored to specific performance criteria, such as accuracy, relevance, and / or security.
[0011] Alternatively, the at least one optimization objective may include minimizing the loss between the response generated for at least one corresponding generated user query and the corresponding secure response, which is produced by querying at least one corresponding generated user query while processing the various tokens of the system hint using an encoded set of embedding parameters. By minimizing this loss, it can be ensured that the machine learning model still provides a similar response when using secure system hints compared to when using (unencrypted) system hints.
[0012] Furthermore, it is possible that at least one optimization objective includes minimizing the cosine similarity between the set of embedding parameters and the generated encoded set of embedding parameters. This minimization of cosine similarity may increase the semantic difference between the original system hint and the secure system hint. By reducing the similarity between the embedding parameters, the machine learning model can learn different representations of the cryptographic hint, making it more uninterpretable and more difficult to reverse engineer the original system hint. This enhanced security can protect sensitive information embedded in the system hint.
[0013] If two optimization objectives are set at the same time, their respective importance can be balanced so that one objective is preferred over the other.
[0014] It is also possible to optimize the set of embedding parameters of the encoding using gradient optimization in the embedding space of the machine learning model, full or partial space search on lexical tokens, evolutionary optimization, and / or genetic optimization. Therefore, it is possible to refine the set of embedding parameters of the encoding using various optimization techniques. Gradient optimization within the embedding space of the machine learning model allows for iterative tuning of the embedding parameters. Exploring lexical tokens through full or partial space search enables the achievement of identifiers that efficiently represent the optimal embeddings for the system cues. Evolutionary and genetic optimization algorithms can mimic the process of natural selection to evolve a set of embedding parameters that best captures the semantic meaning of the provided system cues.
[0015] Further modifications to the set of embedded parameters could include: - Provide or receive at least one attack query, which is designed to extract provided system hints. - Use at least one attack query to query the machine learning model, which outputs at least one extracted system tip based on the query. - Analyze at least one extracted system prompt based on its similarity to the provided system prompts.
[0016] In possible additional steps, at least one extracted system cues may need to be identical to a specified cues different from the provided system cues. Providing at least one attack query can be executed in optimization or testing scenarios. Receiving at least one attack query can be executed in actual attack scenarios. Using these steps, potential weaknesses in the modified embedding parameters can be identified. This can be achieved by introducing at least one attack query designed to elicit information about the original system cues. By analyzing the extracted system cues against the provided system cues, similarity can reveal weaknesses in the encoded set of embedding parameters. This analysis can allow for the refinement of embedding parameters and the creation of more robust and secure system cues, thereby enhancing the security of machine learning models.
[0017] Alternatively, the analysis may include measuring the semantic and / or syntactic similarity between the extracted system cues and the provided system cues. This allows assessment of how closely the extracted system cues relate to the original system cues. This can be performed to evaluate the effectiveness of modifications to the embedding parameters. The similarity measurement can indicate whether the encryption process has successfully altered the meaning of the system cues while preserving their core functionality. Here, at least one metric can be used, such as edit distance, SSIM (structural similarity), Jaccard similarity, and / or Latent Semantic Analysis (LSA).
[0018] The method according to the invention can potentially be used in vehicles. The vehicle can be designed, for example, as a motor vehicle and / or a bus and / or at least a partially automated / autonomous vehicle. The vehicle can have vehicle equipment, for example, for providing autonomous driving functions and / or driver assistance systems. The vehicle equipment can be designed to at least partially automatically control and / or accelerate and / or brake and / or steer the vehicle.
[0019] In another aspect of the invention, a computer program, particularly a computer program product, can be provided, comprising instructions that, when executed by at least one computer, cause the computer to perform the method according to the invention. Therefore, the computer program according to the invention can have the same advantages as those already described in detail with reference to the method according to the invention.
[0020] In another aspect of the invention, an apparatus for data processing can be provided, configured to perform the method according to the invention. As such an apparatus, for example, at least one computer can be provided that executes a computer program according to the invention. The computer may include at least one processor capable of executing the computer program. Furthermore, a non-volatile data memory can be provided, in which the computer program can be stored, and from which the processor can read the computer program for execution.
[0021] According to another aspect of the invention, a computer-readable storage medium can be provided, comprising a computer program and / or instructions according to the invention, which, when executed by at least one computer, cause the computer to perform the steps of the method according to the invention. The storage medium can be formed as a data storage device, such as a hard disk and / or non-volatile memory and / or memory card and / or solid-state drive. For example, the storage medium can be integrated into a computer.
[0022] Furthermore, the method according to the invention can be implemented as a computer-based method. Alternatively or additionally, at least one of the disclosed method steps can be computer-based and / or automated.
[0023] Attached Figure Description Further advantages, features, and details of the invention will become clear from the following description, in which embodiments of the invention are described in detail with reference to the accompanying drawings. In this context, the features mentioned in the claims and the specification may be essential to the invention individually or in any combination. As shown: Figure 1 The present invention relates to methods, computer programs, storage media, and apparatus according to embodiments of the present invention; Figure 2 The method according to an embodiment of the present invention. Detailed Implementation
[0024] Figure 1 The invention includes a method 100, a technical system 1 (particularly a vehicle or production plant), a computer program 20, a storage medium 15, and an apparatus 10, according to embodiments of the invention.
[0025] Figure 1 Specifically, an embodiment of a method 10 for providing a safe system hint for a machine learning model is shown. In a first step 101, a system hint is provided, which is parameterized by an embedding parameter set to process the various tokens of the system hint. In a second step 102, a user query is generated based on the provided system hint. In a third step 103, the machine learning model is queried using the generated user query and the provided system hint, and the machine learning model generates a response based on the query. In a fourth step 104, the embedding parameter set is modified in consideration of the provided system hint, the generated user query, and the generated response to generate an encoded embedding parameter set. In a fifth step 105, a safe system hint is provided, which is semantically different from the provided system hint, by processing the various tokens of the system hint using the generated encoded embedding parameter set.
[0026] This invention specifically addresses the limitations of current system hint extraction defense methods without relying on knowledge of the attack query. According to embodiments of the invention, the provided system hint is encoded to derive similar performance from machine learning models, particularly large language models, while altering its interpretation from the attacker's perspective. Thus, the provided system hint can be completely replaced by a newly formulated secure system hint. This method ensures that even if an attacker successfully extracts the provided system hint, they will only obtain a secure system hint that appears remarkably like gibberish. As a result, the attacker may advantageously misinterpret it or mistakenly believe they failed to retrieve the correct system hint.
[0027] P This can represent a system prompt that guides a machine learning model when performing a specific task. System prompt P This can include using embedded parameters Processing n a mark ,in e This refers to the embedding dimension or the embedding size. (System prompt) P It can be divided into sentences to form a set. ,in m It refers to the number of sentences. Machine learning models, especially large language models, particularly through parameter embedding... Parameterized, it can be deployed to handle user queries Q Mapping to response R , written as Or written for the sake of brevity System prompt P The goal is specifically to guide machine learning models to generate relevant queries. response The response can follow a distribution. The predicted response is particularly , making .
[0028] An attacker's goal might be to extract system hints by issuing carefully crafted queries to a machine learning model. P Or system prompt P The semantically equivalent version. The extracted system prompts. G It can be defined as Attackers can submit k attack query A ,and This could be a function that guesses which response is the system suggestion. For simplicity, this can be written as... . G The sentence within can be represented as An attacker can perform actions without accessing: (i) system prompts. P or its embedded parameters (ii) Parameters of the machine learning model and (iii) related queries and target response .
[0029] The defender is specifically designed to counter cue extraction attacks. Following the defense, relevant queries... The security response can be represented as The defender can control the response of the machine learning model. R And has parameters and However, the defender: (i) cannot distinguish between malicious and benign queries, (ii) lacks prior knowledge of the attacker's strategy, and (iii) is unaware of the relevant queries. and response The defender's objective could be: (i) utility preservation: by maximizing the ratio To ensure reserve The functions, among which M (i) Task-specific metrics (e.g., accuracy, edit distance, Jaccard similarity, or Latent Semantic Analysis (LSA)); and (ii) Extracting differences: ensuring that the extracted cues... G Unlike P This reduces the attacker's success rate.
[0030] According to embodiments of the present invention, semantic matching (SM) metrics can be used to assess the success rate of an attacker. This can be indicated by measurement system prompts. P With the extracted system prompts G To achieve semantic consistency between them, capture within G Communication and P Cases where the same meaning is conveyed but may include different wording. For each sentence It can be calculated and Cosine similarity between sentences in the text, for example, using the cosine similarity between sentences in the text. A parameterized sentence embedding model. The most similar sentences. It can be selected as: in It can represent and The cosine similarity between sentence embeddings, and These can represent the parameters of the embedded model. Next, they can be evaluated. and Mutual implication (ME) between them, for example, using the... A parameterized pre-trained implication classification model. The SM metric is specifically defined as: , in Can indicate sentence pairs Whether they exhibit mutual implication, such as that which can be determined by the implication model. It has been determined.
[0031] The method according to embodiments of the present invention is particularly intended to generate security system prompts. As a replacement that performs similar functions but has different content and semantics, to protect secret system hintsP Not extracted. Although the original system indicated... P Preferably composed of natural language, but encrypted hints may not necessarily adhere to this constraint. Instead, they can take the form of character sequences that machine learning models can process but are meaningless to humans. This seemingly random series of characters could become a hint extracted by an attacker. G .
[0032] The prompting tuning according to the invention can be performed using various methods, such as gradient optimization in the embedding space of a machine learning model, full or partial space search on lexical tags, evolutionary and / or genetic optimization. Since the embedding space of a machine learning model is particularly expressive than the natural language space and inherently unreadable to humans, the embedding parameters of the system prompts according to the invention... It can be used Instead of hard prompts To encrypt. This can be referred to as encrypted soft hints. When a machine learning model... Decoded into security system prompt In this case, such encryption embedded in the space may make it more difficult for attackers to interpret.
[0033] Encrypted soft hints can be optimized to achieve two goals: minimize the impact on relevant queries. The original response and security response Distribution between The expected loss is calculated based on the extraction of the original hint, and the similarity loss between the extracted hint and the original hint is minimized. The joint optimization problem can be defined as follows: in In particular, cross-entropy loss, and The extracted prompts can be measured. G and original system prompts P The similarity between them. Because defenders may not be able to access relevant queries. Therefore, language models can be used to generate data that follows a distribution. User query Furthermore, defenders may lack attack queries. A and extraction function The knowledge, therefore, can be minimized. and Cosine similarity between them is used to approximate P and G The similarity loss between them. This can lead to the following modified optimization problem: .
[0034] Specifically, Oracle Language Model O can be used to generate potential user queries. These user queries are preferably related to system prompts. P Aligning specific focus, constraints, and / or preferences, it simulates natural interactions that guide users toward relevant information or recommendations, while preserving the intent and structure of the original system prompts. Once the proxy query is generated, the defender can then... Figure 2 Step 201 optimizes the encryption soft hints, for example by using gradient-based optimization, as defined in equation (3). Figure 2 An overview of the method according to this embodiment of the invention is described herein.
[0035] Because language models can only output natural language, they are vulnerable to attack when an attacker attempts to steal them. Will be decoded into . and P The key difference lies in their structural and semantic differences, effectively concealing the true nature of the original hint. Cryptographic hints can elicit semantically similar responses to hints from secret systems.
[0036] Thus, even if an attacker manages to extract data using attack query 'a'... They will also be unable to receive actual guidance. P Access, because Encoding the same operational instructions in different forms makes it difficult for attackers to understand or use the extracted prompts. Using encrypted prompts as system prompts, the prompts extracted by attackers deviate significantly from the original system prompts. Therefore, the method according to the invention provides robust defense against prompt extraction attacks by obfuscating the original system prompts while maintaining the performance of the machine learning model.
[0037] The concept of "semantic difference" used throughout this specification can be understood as follows. To quantify this difference, a semantic matching (SM) metric is used. This metric measures the original system prompt P and the security system prompt. The semantic similarity between them is calculated. As mentioned earlier, the SM metric is computed using a combination of cosine similarity of sentence embeddings and mutual entailment. A low SM value indicates a high degree of semantic dissimilarity. The objective is to minimize P and The invention employs a SM metric between semantic differences and functional equivalence while maintaining functional equivalence of the hints, ensuring that the model responds similarly to the same query regardless of which hint is used. This balance between semantic differences and functional equivalence is key to the invention's effectiveness in preventing hint extraction attacks.
[0038] The above explanation of the embodiments describes the invention in the context of examples. Of course, the various features of the embodiments can be freely combined with each other without departing from the scope of the invention—provided it is technically reasonable.
Claims
1. A method (100) for providing security system hints for a machine learning model, comprising the following steps: - Provide (101) system hints, which are parameterized by an embedded parameter set to process the individual tokens of the system hints. - Generate a (102) user query based on the provided system prompts. - Use the generated user query and the provided system prompts to query the (103) machine learning model, which generates a response based on the query. - Modify the (104) embedding parameter set based on the provided system prompts, generated user queries, and resulting responses to generate an encoded embedding parameter set. - Provide (105) a security system prompt that is semantically different from the provided system prompt by processing the individual tokens of the system prompt using the generated set of encoded embedding parameters.
2. The method (100) according to claim 1, characterized in that, Modifications to the set of embedding parameters are performed iteratively (104), wherein the encoded set of embedding parameters is optimized by using at least one optimization objective.
3. The method (100) according to claim 2, characterized in that, The at least one optimization objective includes minimizing the loss between the response generated for at least one corresponding generated user query and the corresponding security response, the corresponding security response being generated by querying the at least one corresponding generated user query while processing the various markers of the system prompt using an encoded set of embedding parameters.
4. The method (100) according to any one of claims 2 or 3, characterized in that, The at least one optimization objective includes minimizing the cosine similarity between the set of embedding parameters and the generated set of encoded embedding parameters.
5. The method (100) according to any one of claims 2 to 4, characterized in that, The set of encoding embedding parameters is optimized by using gradient optimization in the embedding space of a machine learning model, full or partial space search on lexical tags, evolutionary optimization, and / or genetic optimization.
6. The method (100) according to any one of the preceding claims, characterized in that, The modification (104) of the embedded parameter set includes: - Provide or receive at least one attack query, said attack query being designed to extract provided system hints. - Use the at least one attack query to query the machine learning model, and the machine learning model outputs at least one extracted system hint based on the query. - Analyze the at least one extracted system prompt based on its similarity to the provided system prompt.
7. The method (100) according to claim 6, characterized in that, The analysis includes measuring the semantic and / or syntactic similarity between the extracted system cues and the provided system cues.
8. The method (100) according to any one of the preceding claims, characterized in that, The machine learning model is implemented in a technical system (1), particularly in a vehicle or production plant.
9. A computer program (20) comprising instructions that, when the computer program (20) is executed by at least one computer (10), cause the computer (10) to perform the method (100) of any of the preceding claims.
10. A data processing apparatus (10) comprising components for carrying out the method (100) of any one of claims 1 to 8.
11. A computer-readable storage medium (15) comprising instructions that, when executed by at least one computer (10), cause the computer (10) to perform the steps of the method (100) of any one of claims 1 to 8.