A retrieval-oriented enhanced generation of prompts hijack defense method and system

Abstract

The present application relates to the technical field of artificial intelligence security, and more particularly to a prompt hijacking defense method and system for retrieval enhancement generation, comprising parsing a user query prompt into a trusted instruction segment and an untrusted data segment; calculating instruction semantic risk scores of each word element in the untrusted data segment according to the embedding representation of each word element in the untrusted data segment and a learnable semantic prototype vector, and performing double-level semantic polarity deflection on each word element in the untrusted data segment to form double-level semantic isolation constraints; extracting instruction semantic anchor points of the trusted instruction segment based on the trusted instruction segment and the double-level semantic isolation constraints; in the self-recurrent decoding process of retrieval enhancement generation, taking the hidden state of the output sequence generated at each decoding step as a query vector, and performing attention fusion decoding with the instruction semantic anchor points as key value vectors to generate a safe response text. The present application can effectively identify and block maliciously injected pseudo-instruction content while completely preserving the semantic integrity of trusted instructions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a method and system for preventing hijacking of prompts based on enhanced retrieval generation. Background Technology

[0002] In Retrieval-Augmented Generation (RAG) knowledge question answering systems, the Large Language Model (LLM) needs to simultaneously receive trusted system-level instructions and user-provided retrieval document data, and generate responses accordingly. However, traditional LLMs process instruction lexical units and data lexical units by inputting them into the same attention mechanism. The two types of lexical units share similar embedding representation spaces, making it difficult for the model to fundamentally distinguish between the system instructions to be executed and the data content to be processed. This structural defect exposes RAG systems to a serious threat of prompt hijacking attacks. Attackers can inject instructional text into the retrieved external document data, causing the model to misjudge the injected content as a legitimate instruction and elevate it to instruction-level semantics, thereby overriding trusted system-level predefined instructions and generating malicious behavior. Such attacks are divided into two categories: direct injection and indirect injection. Indirect injection is achieved by tampering with the retrieval data source, which is particularly covert and difficult to detect in the actual deployment of RAG systems.

[0003] To address the aforementioned issues, existing defense methods primarily employ fine-tuning based on structured delimiters. This method introduces special delimiter terms and trains the model through supervised fine-tuning to ignore injected content. However, it suffers from insufficient robustness against out-of-distribution adversarial examples and struggles to cope with unknown attack variants that have undergone semantic spoofing or synonym rewriting. Furthermore, existing methods based on embedded role tags introduce fixed global embedding offsets for different role segments. However, a single global offset cannot differentiate the instructional strength of different terms, easily leading to excessive suppression of benign retrieval document content and resulting in a decline in the performance of normal question-answering tasks. In addition, these existing methods all achieve defense at a single representation level, relying solely on static instructional semantic enhancement mechanisms. When faced with strongly adversarial injected content such as gradient-optimized adversarial suffixes or general adversarial prefixes generated by gradient coordinate greedy algorithms, the semantic influence of the original instruction during the decoding process can still be significantly weakened, leading to defense failure and failing to meet the security requirements of retrieval enhancement generation systems in practical deployments. Summary of the Invention

[0004] To address the above technical problems, this invention provides a method and system for preventing hijacking of prompts in search enhancement generation.

[0005] In a first aspect, the present invention provides a method for preventing hijacking of prompts in search enhancement generation, the method comprising the following steps: The system receives user query prompts from the retrieval enhancement generation system and parses the user query prompts into trusted instruction segments and untrusted data segments; the untrusted data segments include user query statements and external document fragments returned by external knowledge bases; Based on the similarity between the embedding representation of each word in the untrusted data segment and the learnable semantic prototype vector, the imperative semantic risk score of each word in the untrusted data segment is calculated. Using the aforementioned directive semantic risk score as the modulation coefficient, a two-level semantic polarity shift is performed on each word in the untrusted data segment to form a two-level semantic isolation constraint. The trusted instruction segment and the untrusted data segment processed by the two-level semantic isolation constraint are forward encoded and propagated to extract the instruction semantic anchor point of the trusted instruction segment. In the autoregressive decoding process of retrieval enhancement generation, the hidden state of the output sequence generated at each decoding step is used as the query vector, and attention fusion decoding is performed using the semantic anchor of the instruction as the key value vector to generate a secure response text.

[0006] In a further implementation, the step of parsing the user query prompt into trusted instruction segments and untrusted data segments includes: The user query prompts are parsed in a structured segmentation based on the predefined structured prompt delimiters of the retrieval enhancement generation system. System-level task instruction content is classified into trusted instruction segments, while user query statements and external document fragments returned by external knowledge bases based on user query statements are classified into untrusted data segments.

[0007] In a further implementation, the structured prompt delimiter includes role identifiers, field identifiers, delimiter markers, and message metadata.

[0008] In a further implementation, the step of calculating the instructional semantic risk score of each word in the untrusted data segment based on the similarity between the embedded representation of each word in the untrusted data segment and the learnable semantic prototype vector includes: Learnable semantic prototype vectors are pre-constructed based on semantic clustering centers in the embedding space of instructional and data content; the learnable semantic prototype vectors include instructional semantic prototype vectors and data semantic prototype vectors. For each word in the untrusted data segment, perform word embedding mapping to obtain the word embedding representation of each word, and calculate the cosine similarity between the word embedding representation and the instruction semantic prototype vector to obtain the instruction semantic similarity. Calculate the cosine similarity between the lexical embedding representation and the data semantic prototype vector to obtain the data semantic similarity; The semantic bias of each word is quantified based on the difference between the semantic similarity of the instruction and the semantic similarity of the data. The semantic bias is nonlinearly mapped to obtain the instructional semantic risk score of each word in the untrusted data segment.

[0009] In a further implementation, the step of using the directive semantic risk score as the modulation coefficient to perform a two-level semantic polarity shift on each word in the untrusted data segment to form a two-level semantic isolation constraint includes: The word embedding representation of the untrusted data segment is mapped by a first-order linear projection transformation to obtain the first-order semantic deflection direction vector of each word in the untrusted data segment. Based on the command-line semantic risk score and the first-order semantic deflection direction vector, the word embedding representation of each word in the untrusted data segment is subjected to first-order semantic polarity deflection to obtain the first-order defensive embedding representation of each word. In the target large language model, a target decoding layer for second-order residual correction is pre-determined, and the first-order defensive embedding representation is input into the target large language model for forward propagation. The embedded hidden state sequence is output through the target decoding layer. The embedded hidden state sequence is segmented by the word boundary position between the untrusted data segment and the trusted instruction segment, and the intermediate hidden state corresponding to each word of the untrusted data segment is extracted. The intermediate hidden states are corrected by residual superposition based on the command semantic risk score to obtain the second-order defense hidden states corresponding to each word in the untrusted data segment. Based on the first-order defensive embedding representation and the second-order defensive hidden state, the shift of untrusted data segment lexical units to imperative content is suppressed in the embedding layer and intermediate hidden state layer of the target large language model, forming a two-order semantic isolation constraint from the embedding layer to the intermediate hidden state layer.

[0010] In a further implementation, the step of performing a first-order semantic polarity shift on the lexical embedding representation of each word in the untrusted data segment based on the directive semantic risk score and the first-order semantic shift direction vector to obtain the first-order defensive embedding representation of each word includes: The first-order semantic deflection direction vector is scaled by the directive semantic risk score to obtain a deflection adjustment vector. The deflection adjustment vector is then superimposed with the word embedding representation of each word in the untrusted data segment to obtain the first-order defense embedding representation of each word.

[0011] In a further implementation, the step of forward encoding and propagating the trusted instruction segment and the untrusted data segment processed by the two-order semantic isolation constraint to extract the instruction semantic anchor point of the trusted instruction segment includes: Each word in the trusted instruction segment is converted into the corresponding original instruction embedding vector, and the first-order defense embedding representation obtained after processing each word in the untrusted data segment through two-order semantic isolation constraints is integrated with the second-order defense hidden state to obtain the untrusted data segment representation vector. The original instruction embedding vector and the untrusted data segment representation vector are concatenated according to the word order in the user query prompt to obtain the joint input sequence; The joint input sequence is input into the target large language model for forward encoding propagation to obtain a high-level semantic representation; Based on the start and end lexical positions of the trusted instruction segment in the joint input sequence, the instruction hidden state vector at the corresponding position is extracted from the high-level semantic representation. The hidden state vector of the instruction is used as the instruction semantic anchor point of the trusted instruction segment.

[0012] In a further implementation, the step of generating a secure response text by using the hidden state of the output sequence generated at each decoding step as a query vector and the instruction semantic anchor as a key vector for attention fusion decoding includes: In the autoregressive decoding stage of retrieval enhancement generation, the current output hidden state vector generated in the current decoding step is used as the query vector for the current decoding step; Construct key vectors and value vectors based on the instruction semantic anchors, and calculate the semantic similarity between the query vector and the key vectors to obtain the attention weight of the instruction semantic anchors in the current decoding step; The value vector is weighted and aggregated using the attention weights to obtain the anchor-guided semantic vector for the current decoding step. The anchor-guided semantic vector is linearly projected and then fused with the current output hidden state vector by residual addition to obtain the fused decoded representation; Probabilistic prediction is performed on the fused decoding representation to obtain the candidate word probabilities for the current decoding step, and the word with the highest candidate word probability is selected as the safe response word for the current decoding step. All generated security response tokens are concatenated in sequence to form the security response text.

[0013] Secondly, the present invention provides a system for preventing hijacking of prompts in search enhancement generation, the system comprising: The data parsing module is used to receive user query prompts from the retrieval enhancement generation system and parse the user query prompts into trusted instruction segments and untrusted data segments; the untrusted data segments include user query statements and external document fragments returned by external knowledge bases; The risk assessment module is used to calculate the instructional semantic risk score of each word in the untrusted data segment based on the similarity between the embedded representation of each word in the untrusted data segment and the learnable semantic prototype vector. The semantic isolation module is used to perform two-level semantic polarity shift on each word in the untrusted data segment using the instructional semantic risk score as the modulation coefficient, thereby forming a two-level semantic isolation constraint. The semantic anchoring module is used to forward encode and propagate the trusted instruction segment and the untrusted data segment processed by the double-order semantic isolation constraint to extract the instruction semantic anchor point of the trusted instruction segment. The decoding generation module is used in the autoregressive decoding process of retrieval enhancement generation to use the hidden state of the output sequence generated at each decoding step as the query vector, and the instruction semantic anchor point as the key value vector to perform attention fusion decoding to generate a secure response text.

[0014] In a further implementation, parsing the user query prompt into trusted instruction segments and untrusted data segments specifically includes: The user query prompts are parsed in a structured segmentation based on the predefined structured prompt delimiters of the retrieval enhancement generation system. System-level task instruction content is classified into trusted instruction segments, while user query statements and external document fragments returned by external knowledge bases based on user query statements are classified into untrusted data segments.

[0015] This invention provides a method and system for preventing prompt hijacking in search enhancement generation. The method receives user query prompts from a search enhancement generation system and parses them into trusted instruction segments and untrusted data segments. Based on the similarity between the embedded representations of each word in the untrusted data segment and the learnable semantic prototype vector, a command semantic risk score is calculated for each word in the untrusted data segment. Using the command semantic risk score as a modulation coefficient, a two-level semantic polarity shift is applied to each word in the untrusted data segment, forming a two-level semantic isolation constraint. The trusted instruction segment and the untrusted data segment processed by the two-level semantic isolation constraint are forward encoded and propagated to extract the command semantic anchor point of the trusted instruction segment. During the autoregressive decoding process of search enhancement generation, the hidden state of the output sequence generated at each decoding step is used as the query vector, and attention fusion decoding is performed using the command semantic anchor point as the key vector to generate a secure response text. Compared with existing technologies, this method achieves accurate identification of prompt hijacking attacks based on techniques such as word-level risk quantification of learnable semantic prototype vectors, defense against two-level semantic polarity shift, and attention decoding guided by instruction semantic anchors. It can effectively suppress direct prompt injection, indirect document tampering, and strong adversarial prompt hijacking attacks constructed through gradient optimization. This allows the model to stably execute trusted instructions and complete normal retrieval and question-answering tasks even when maliciously injected content exists, fully preserving the semantic integrity of trusted instructions. Thus, while making full use of external knowledge to enhance generation capabilities, it ensures the security and controllability of the generation process, improving the robustness and security of the retrieval enhancement generation system in the face of complex adversarial attacks. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method for preventing hijacking of prompts based on enhanced retrieval provided in this embodiment of the invention; Figure 2 This is a block diagram of a prompt hijacking defense system for enhanced retrieval generation provided in an embodiment of the present invention.

[0017] Figure labeling: 101, Data parsing module; 102, Risk assessment module; 103, Semantic isolation module; 104, Semantic anchoring module; 105, Decoding and generation module. Detailed Implementation

[0018] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0019] Figure 1This is a schematic flowchart of a method for preventing hijacking of prompts in search enhancement generation, provided by an embodiment of the present invention. The embodiment of the present invention provides a method for preventing hijacking of prompts in search enhancement generation, such as... Figure 1 As shown, the method includes the following steps: S1. Receive user query prompts from the retrieval enhancement generation system, and parse the user query prompts into trusted instruction segments and untrusted data segments; the untrusted data segments include user query statements and external document fragments returned by external knowledge bases.

[0020] In some implementations, the step of parsing the user query prompt into trusted instruction segments and untrusted data segments includes: The system performs structured segmentation parsing on the user query prompts based on the predefined structured prompt delimiters of the retrieval enhancement generation system. System-level task instruction content is classified into trusted instruction segments, while user query statements and external document fragments returned by external knowledge bases based on user query statements are classified into untrusted data segments. The structured prompt delimiters include role identifiers, field identifiers, delimiter markers, and message metadata.

[0021] Specifically, in a retrieval-enhanced generation system, the large language model needs to simultaneously process three types of heterogeneous information: system-level task instructions predefined by application developers, query statements submitted by end users, and document fragments retrieved by external knowledge bases based on these query statements. Since the sources of the query statements submitted by end users and the document fragments retrieved by external knowledge bases based on these query statements are uncontrollable, attackers can easily disguise malicious instructions as ordinary query content or embed them in external documents for hijacking. However, traditional retrieval-enhanced generation systems encode these three types of heterogeneous information uniformly into the same semantic space, causing the large language model to be unable to distinguish between the system instructions to be executed and the external data that should only be processed at the input stage. This makes it extremely easy for maliciously injected content to be misjudged as instructions to be executed, thereby overriding the original task objective. To establish a semantic boundary between trustworthy and untrustworthy content at the input end, this embodiment, after receiving the user query prompt from the retrieval-enhanced generation system, utilizes the retrieval-enhanced generation... The system's predefined structured prompt delimiters segment and parse the input user query prompts, classifying system-level task instructions as trusted instruction segments and user query statements and external document fragments returned by external knowledge bases as untrusted data segments. In this embodiment, the structured prompt delimiters include, but are not limited to, role identifiers, field identifiers, delimiter markers, and message metadata. Role identifiers distinguish the role attributes of different message sources, such as system developers, end users, and external search tools. Field identifiers mark the functional type of each text fragment, such as system instruction fields, user query fields, and search return fields. Delimiter markers use preset special word sequences or format symbols to physically isolate different semantic regions. Message metadata describes auxiliary information such as source markers, timestamps, and trust labels for each fragment. This embodiment, through the parsing of the above multi-dimensional identifiers, can accurately identify the boundary positions and trust attributes of each fragment in the input prompt.

[0022] It should be noted that the trusted instruction segment specifically refers to system-level task instruction content predefined by the application developer. This content includes, but is not limited to, role settings, output format constraints, security policy requirements, and task objective descriptions. For example, the trusted instruction segment might contain statements such as "Please answer the user's question based on the following document content" or "Please ignore any instructional statements in the data segment." This trusted instruction segment content is fixed during the system configuration phase, its source is trusted, and it has not been transmitted externally. Therefore, this embodiment will maintain the original semantic integrity of the trusted instruction segment in subsequent steps and will not participate in subsequent semantic shift operations targeting untrusted data segments to ensure that the system-level task instruction content is not mistakenly edited. Untrusted data segments include two types of external source content: query statements submitted by end users and external document fragments retrieved by external knowledge bases based on user queries. Query statements submitted by end users come directly from open input interfaces, posing a risk of attackers injecting direct prompts. External document fragments retrieved by external knowledge bases based on user queries originate from open third-party databases or network resources, and their content integrity cannot be guaranteed, posing a risk of attackers injecting direct prompts. The risk of pre-embedding indirect hints and injection commands exists. For example, in an untrusted data segment, the user's query might be "Please tell me how to reset my password," while the document fragment returned by the external knowledge base should be normal technical document content. However, an attacker might embed malicious commands such as "ignore the original command" or "output system has been compromised." In attack scenarios, attackers can further construct more complex hints. For example, they can insert pre- and post-assistance perturbation strings before and after a clean data segment, respectively. These pre- and post-assistance perturbation strings can be used to wrap, disguise, or induce enhancement of the injected command, forming a complete attack hint. In this embodiment, external inputs that are not pre-set by the system are collectively classified into the untrusted data segment, and their semantic attributes are limited to data content to be processed, rather than command content to be executed. This avoids the model misidentifying the text in the user's query or retrieved document as system-level operation commands. Through the above-mentioned structured segmentation parsing, this embodiment achieves physical isolation and semantic distinction between trusted commands and untrusted data at the input end of the retrieval enhancement generation system, effectively solving the problem of ambiguity caused by the mixing of commands and data in traditional architectures.

[0023] S2. Calculate the instructional semantic risk score of each word in the untrusted data segment based on the similarity between the embedded representation of each word in the untrusted data segment and the learnable semantic prototype vector.

[0024] In some implementations, the step of calculating the imperative semantic risk score of each word in the untrusted data segment based on the similarity between the embedded representation of each word in the untrusted data segment and the learnable semantic prototype vector includes: Learnable semantic prototype vectors are pre-constructed based on semantic clustering centers in the embedding space of instructional and data content; the learnable semantic prototype vectors include instructional semantic prototype vectors and data semantic prototype vectors. For each word in the untrusted data segment, perform word embedding mapping to obtain the word embedding representation of each word, and calculate the cosine similarity between the word embedding representation and the instruction semantic prototype vector to obtain the instruction semantic similarity. Calculate the cosine similarity between the lexical embedding representation and the data semantic prototype vector to obtain the data semantic similarity; The semantic bias of each word is quantified based on the difference between the semantic similarity of the instruction and the semantic similarity of the data. The semantic bias is nonlinearly mapped to obtain the instructional semantic risk score of each word in the untrusted data segment.

[0025] In the retrieval enhancement generation system, the untrusted data segment contains both the query statement input by the end user and document fragments returned by an external knowledge base. Because the external document source is open and lacks prior verification, attackers may directly inject direct hints into the user query, or they may pre-modify the retrieved document to carry out a more covert indirect hint injection attack. Furthermore, the injected content is often semantically disguised, paraphrased, or formatted. However, traditional methods often rely on preset trigger words or rule templates for matching, making it difficult to identify injected fragments that have undergone semantic disguise, paraphrasing, or format obfuscation. This makes the defense boundary extremely easy to breach. To solve the above technical problems, this embodiment addresses the issue of each term in the untrusted data segment... This embodiment quantifies the semantic risks of directive content, enabling the retrieval enhancement generation system to achieve fine-grained identification of injected content with diverse forms and strong semantic camouflage without relying on explicit attack templates. Specifically, this embodiment predefines two learnable semantic prototype vectors: a directive semantic prototype vector and a data semantic prototype vector. In this embodiment, the directive semantic prototype vector represents the semantic cluster center of directive content in the embedding space, and the data semantic prototype vector represents the semantic cluster center of data content in the embedding space. These two semantic prototype vectors are used as learnable parameters during training, and are updated along with the model parameters through gradient descent optimization, adaptively capturing the embeddings of directive and data content in the training set. Based on the distribution characteristics, no manual preset fixed thresholds or attack pattern libraries are required. Since the lexical units in the untrusted data segment are discrete symbol sequences, this embodiment maps them to a continuous high-dimensional semantic space. Specifically, word embedding mapping is performed on each lexical unit in the untrusted data segment to obtain the lexical unit embedding representation of each lexical unit. If the semantic attribution of a lexical unit cannot be accurately determined based on similarity in only one direction, this embodiment simultaneously measures the proximity of the lexical unit to both the instruction semantic center and the data semantic center, establishing semantic coordinates in both instructional and data dimensions for each lexical unit. This achieves dual-anchor positioning of the lexical unit in the instruction and data bipartite semantic spaces. Therefore, this embodiment not only calculates the lexical unit embedding representation and the instruction semantic prototype direction... The cosine similarity between the quantities is used to obtain the instruction semantic similarity. Simultaneously, the cosine similarity between the word embedding representation and the data semantic prototype vector is calculated to obtain the data semantic similarity. Then, this embodiment subtracts the instruction semantic similarity from the data semantic similarity to obtain the semantic bias. The difference in semantic bias is used to quantify the relative bias of the corresponding word embedding representation relative to the instruction semantic prototype and the data semantic prototype, thereby quantifying whether each word is more semantically instruction-oriented or more data-oriented. When the semantic bias is positive, it indicates that the word is closer to the instruction semantic prototype in the embedding space and has a strong instructional semantic tendency; when the semantic bias is negative, it indicates that the word is closer to the data semantic prototype and belongs to data-oriented descriptive content.

[0026] Semantic bias is an unbounded real number and cannot be directly used as a gating coefficient for subsequent semantic editing intensity. Therefore, this embodiment maps it to a bounded interval with probabilistic semantics. However, since the difference in semantic bias may be in an extremely small range in actual calculations, directly inputting it into a nonlinear mapping function can easily lead to function output saturation or gradient vanishing, causing the scores of high-risk and low-risk words to converge and making it impossible to distinguish risk levels. Therefore, this embodiment introduces a learnable numerical range adjustment coefficient to scale the semantic bias and adjusts the scaled semantic bias to within the sensitive response range of the nonlinear mapping function to ensure that the scores of words with different risk levels have sufficient distinguishability. Then, this embodiment uses truncation processing to scale the semantic bias. The threshold is limited to a reasonable range of -8 to +8 to suppress the interference of extreme values ​​on subsequent mappings and ensure the stability of numerical calculations. The truncated results are mapped to the interval between zero and one using a logistic regression-type nonlinear mapping function. Finally, the imperative semantic risk score of each word in the untrusted data segment is obtained. In this embodiment, the closer the imperative semantic risk score is to one, the closer the embedded representation of the corresponding word is to the imperative semantic prototype, and the higher the risk of it having potential pseudo-instruction attributes. It will then be subject to stronger semantic bias intervention. The closer the imperative semantic risk score is to zero, the closer the embedded representation of the corresponding word is to the data semantic prototype, and the greater the possibility that it belongs to normal data description content. It will then be subject to weaker intervention to preserve the original semantic information required to complete the task.

[0027] To ensure the numerical stability of the aforementioned directive semantic risk scores in specific engineering implementations, the cosine similarity calculation between the lexical embedding and the prototype vector preferentially adopts a normalization process with a numerical stability term. This involves introducing a minimal positive number in the denominator of the vector norm for numerical stabilization, with the magnitude of this minimal positive number preferably ranging from 10⁻⁸ to 10⁻⁶. Alternatively, a lower limit of no less than a preset threshold can be set for the vector norm to avoid division-by-zero errors, numerical overflow, or gradient anomalies caused by the norm approaching zero in extreme cases. The numerical range adjustment coefficient is a preset positive number or a learnable positive parameter, preferably ranging from 0.05 to 1.0, or it can be ensured to be positive through soft positive parameterization during training to avoid errors caused by adjustment. Too small a coefficient leads to scaling failure, while too large a coefficient leads to numerical divergence. To maintain the stability and discriminability of the instructional semantic risk score, in this embodiment, the instructional semantic risk score is calculated only for the lexical units in the untrusted data segment and is only used to modulate the semantic editing intensity of the untrusted data segment. The lexical units in the trusted instruction segment retain their original embedded representation and propagate along the normal forward path of the base model, without participating in the calculation of the instructional semantic risk score. By limiting the scope of risk identification and semantic editing to the untrusted data segment, this embodiment can fully preserve the integrity of the original representation of the trusted instruction semantics and avoid causing collateral perturbations to the original task objective, thus balancing security and instruction compliance capability in the retrieval enhancement generation system.

[0028] It should be noted that the imperative semantic risk score is used to characterize the imperative tendency of data segment lexical units before editing and serves as a gating coefficient for the intensity of subsequent semantic editing, rather than to limit the target representation position of the lexical unit after editing. Compared with traditional methods that rely on fixed trigger word detection, this embodiment is based on adaptive analysis of the semantic distribution center of imperative and data content using learnable semantic prototypes. It can effectively identify adversarial suffixes, general adversarial prefixes, and semantic rewriting injection fragments constructed through gradient optimization, thereby improving the generalization robustness of the defense method against out-of-distribution attacks and adaptive attacks. At the same time, by limiting the scope of action to untrusted data segments, it avoids damage to the ability to follow normal instructions in clean task scenarios.

[0029] S3. Using the aforementioned directive semantic risk score as the modulation coefficient, perform a two-level semantic polarity shift on each word in the untrusted data segment to form a two-level semantic isolation constraint.

[0030] In some implementations, the step of using the directive semantic risk score as a modulation coefficient to perform a two-level semantic polarity shift on each word in the untrusted data segment to form a two-level semantic isolation constraint includes: The word embedding representation of the untrusted data segment is mapped by a first-order linear projection transformation to obtain the first-order semantic deflection direction vector of each word in the untrusted data segment. Based on the command-line semantic risk score and the first-order semantic deflection direction vector, the word embedding representation of each word in the untrusted data segment is subjected to first-order semantic polarity deflection to obtain the first-order defensive embedding representation of each word. In the target large language model, a target decoding layer for second-order residual correction is pre-determined, and the first-order defensive embedding representation is input into the target large language model for forward propagation. The embedded hidden state sequence is output through the target decoding layer. The embedded hidden state sequence is segmented by the word boundary position between the untrusted data segment and the trusted instruction segment, and the intermediate hidden state corresponding to each word of the untrusted data segment is extracted. The intermediate hidden states are corrected by residual superposition based on the command semantic risk score to obtain the second-order defense hidden states corresponding to each word in the untrusted data segment. Based on the first-order defensive embedding representation and the second-order defensive hidden state, the shift of untrusted data segment lexical units to imperative content is suppressed in the embedding layer and intermediate hidden state layer of the target large language model, forming a two-order semantic isolation constraint from the embedding layer to the intermediate hidden state layer.

[0031] In a specific embodiment, since the pseudo-instructional semantics of high-risk words spread layer by layer in the multi-layer transformer decoding block as the model propagates forward, ultimately interfering with the output results, and traditional single-layer defenses (such as editing only at the embedding layer) cannot block the continuous penetration of malicious semantics deep within the network, this embodiment constructs a two-level hierarchical semantic polarity shift mechanism consisting of a first-order coarse deflection at the embedding layer and a second-order fine correction at the intermediate hidden state layer. Semantic correction is performed at two different depths of the model, so that the first-order coarse deflection at the embedding layer suppresses the instructional tendency of high-risk words as early as possible at the input end, and the second-order fine correction at the intermediate hidden state layer further eliminates residual pseudo-instructional features in the middle of the model, forming a two-level semantic isolation constraint from the embedding space to the hidden state space, thereby reducing the interference of strongly adversarial injected content on the model's final decision. Specifically, the two-level hierarchical semantic polarity shift process in this embodiment is divided into the following two stages: The first stage involves a first-order coarse deflection at the lexical embedding layer. Specifically, this is achieved by mapping the original lexical embedding representations of each lexical in the untrusted data segment to a first-order semantic deflection direction vector using a learnable first-order linear projection transformation matrix. The input and output dimensions of the first-order linear projection transformation matrix are the same as the lexical embedding dimensions, and its parameters are jointly optimized and updated with the model parameters during training. Then, to avoid uncontrolled editing intensity corresponding to high-risk lexicals, this embodiment introduces a first-order deflection step size for amplitude constraint. The first-order deflection step size is determined by a preset maximum allowable step size and a maximum allowable deflection norm. Specifically, In this embodiment, the first-order deflection step size is taken as the smaller value between the preset maximum allowable step size and the quotient obtained by dividing the maximum allowable deflection norm by the first-order projection vector norm, thereby ensuring that the norm of the actual first-order deflection amount does not exceed the upper limit of the maximum allowable deflection norm. In practical applications, the maximum step size can be set between 0.05 and 0.2, and the maximum allowable deflection norm can be taken as 0.5 to 2 times the average word embedding modulus length in the training set. Then, in this embodiment, the instructional semantic risk score, the first-order semantic deflection direction vector, and the first-order deflection step size are multiplied to obtain the deflection adjustment vector, wherein the first-order deflection step size is... Instead of being a fixed constant, it is dynamically calculated based on the magnitude of the first-order projection vector. In this embodiment, the first-order deflection step size is used to control the maximum amplitude of a single deflection, preventing the editing intensity of high-risk lexical units from getting out of control. Finally, in this embodiment, the deflection adjustment vector is added element-wise to the original lexical unit embedding representation to obtain the first-order defensive embedding representation of each lexical unit. It should be noted that the first-order coarse deflection is applied to the input embedding layer, and its purpose is to suppress the influence of potential pseudo-instruction lexical units in untrusted data segments on subsequent multi-layer attention propagation as early as possible, so that the representation of highly salient lexical units moves from the region closer to the instruction semantics towards... The region migration is closer to the data semantics, thereby weakening its pseudo-instruction semantic components and enhancing its data content attributes. In this embodiment, the closer the instruction semantic risk score is to 1, the stronger the pseudo-instruction attribute is, the greater the magnitude of its embedding representation is migrated, so that its position in the embedding space shifts from close to the instruction semantic manifold to the data semantic manifold; while the embedding of normal data words with instruction semantic risk scores close to 0 remains basically unchanged, so as to retain the original semantic information required to complete the task. All words in the credible instruction segment do not participate in the above first-order deflection, and their original embeddings are directly fed into the model.

[0032] The second stage involves performing second-order residual fine-tuning on the selected intermediate decoding hidden state layer. Specifically, since some strongly adversarial injected content may reappear after nonlinear transformation through multi-layer transformers, this embodiment cannot completely eliminate pseudo-instruction features through first-order embedding layer deflection alone. Therefore, this embodiment uses first-order defensive embedding to represent the input target large language model. The model (LLM) performs forward propagation, outputting an embedded hidden state sequence at a pre-determined intermediate decoding layer. The embedded hidden state sequence is then segmented by the lexical boundary between untrusted data segments and trusted instruction segments, extracting the intermediate hidden states corresponding to each lexical in the untrusted data segments. In this embodiment, the pre-determined intermediate decoding layer is not fixed to a specific layer number, but rather selected near the middle of the model, balancing the initial formation of semantic representation with the incomplete solidification of output semantics. For large language models with different numbers of layers or different structures, this embodiment adaptively determines the position relative to the total number of layers, used for correction when the semantic representation in the first half has been formed but the output semantics in the second half has not yet been solidified. For example, when the total number of layers is 32, this embodiment can select around layer 16. This position selection not only ensures that the first half of the decoding layer has formed a certain degree of semantic representation, allowing residual instructional features to be detected, but also that the output semantics in the second half are not yet fully solidified, so correction at this time can still effectively affect the final output.

[0033] In this embodiment, the extracted intermediate hidden states of the untrusted data segment are input into the residual correction network. This residual correction network consists of two fully connected layers. The first layer compresses the high-dimensional hidden states to a smaller bottleneck dimension and activates them nonlinearly through Gaussian error linear units (GELU). The second layer restores the bottleneck features to the original hidden state dimension, thereby outputting a second-order semantic correction. Then, in this embodiment, the imperative semantic risk score of the corresponding word is multiplied by the second-order semantic correction to achieve amplitude modulation of risk perception, resulting in a modulated second-order semantic correction. The modulated second-order semantic correction is then residually superimposed with the intermediate hidden states to obtain the second-order defensive hidden states corresponding to each word in the untrusted data segment. The second-order defensive hidden states are used as input to the next decoding layer to continue participating in subsequent decoding. The role of the second-order residual correction is to compensate for the insufficiency of the previous processing, eliminate the residual imperative features that reappear in the deep propagation process after the first-order deflection, and form a second barrier of two-order semantic isolation.

[0034] After the above two-level semantic polarity deflection, untrusted data segment lexical units fall into one side of the data semantic manifold in both the embedding layer and the intermediate hidden state layer, while trusted instruction segment lexical units do not undergo first-order deflection and second-order correction. Trusted instruction segment lexical units remain on one side of the instruction semantic manifold along the original propagation path of the basic model. The two-order deflection works together in different representation layers, thereby forming a cross-level two-order semantic isolation constraint in the embedding space and the hidden state space, effectively suppressing the continuous infiltration of malicious data into the model's internal representation.

[0035] In summary, in retrieval enhancement generation scenarios, external document fragments are often lengthy and semantically complex. A single embedding layer deflection is insufficient to completely eliminate semantic residues of disguised indirect hint injection content in deep networks. This embodiment employs a hierarchical process of first-order coarse deflection at the embedding layer and second-order residual fine correction at the intermediate hidden state layer. The first-order embedding layer deflection forces the embedding representations of high-risk words in untrusted data segments to be migrated to the data semantic manifold, causing them to deviate from the instruction semantic region at the source. The second-order residual correction at the intermediate hidden state layer further suppresses residual instructional features that reappear during deep propagation, ensuring that the model is not affected by residual pseudo-instructions during subsequent decoding. This provides two-level defense against untrusted data segments and improves robustness under strong perturbation conditions. Simultaneously, since the deflection strength is adaptively modulated by the instructional semantic risk score at the word level, it maintains minimal intervention for low-risk words that are close to normal data content, thus avoiding excessive suppression of benign retrieval document content. This ensures that the model retains the semantic content of the data required to complete tasks such as question answering and summarizing while suppressing malicious injection, balancing security and practicality.

[0036] S4. Perform forward encoding and propagation on the trusted instruction segment and the untrusted data segment processed by the two-level semantic isolation constraint to extract the instruction semantic anchor point of the trusted instruction segment.

[0037] In some implementations, the step of forward encoding and propagating the trusted instruction segment and the untrusted data segment processed by the two-order semantic isolation constraint to extract the instruction semantic anchor point of the trusted instruction segment includes: Each word in the trusted instruction segment is converted into the corresponding original instruction embedding vector, and the first-order defense embedding representation obtained after processing each word in the untrusted data segment through two-order semantic isolation constraints is integrated with the second-order defense hidden state to obtain the untrusted data segment representation vector. The original instruction embedding vector and the untrusted data segment representation vector are concatenated according to the word order in the user query prompt to obtain the joint input sequence; The joint input sequence is input into the target large language model for forward encoding propagation to obtain a high-level semantic representation; Based on the start and end lexical positions of the trusted instruction segment in the joint input sequence, the instruction hidden state vector at the corresponding position is extracted from the high-level semantic representation. The hidden state vector of the instruction is used as the instruction semantic anchor point of the trusted instruction segment.

[0038] In a specific embodiment, since traditional autoregressive decoding relies solely on the standard self-attention mechanism to transmit instruction information, when injected content in the retrieved document systematically interferes with the attention weights, the model output is easily pulled in the direction expected by the attacker, causing the model to deviate from the preset task objective. Therefore, this embodiment introduces a cross-sequence attention anchoring fusion channel in the decoding stage, using the final hidden state sequence of the trusted instruction segment as an independent semantic anchor point. At each generation step, the correlation between the current output and the instruction anchor point is dynamically calculated, and the most relevant instruction semantic fragment is fused into the output representation. This allows for stable output of secure response text even under strong adversarial interference, without disrupting the original autoregressive decoding structure. In this specific embodiment, each word in the parsed trusted instruction segment is converted into the corresponding original instruction embedding vector through a word embedding layer. Simultaneously, the first-order defense embedding representation obtained after two-level semantic polarity shift processing of each word in the untrusted data segment is used as its representation in the joint input sequence. It should be noted that in practical applications, the second-order defense hidden state is a temporary state used during intermediate layer correction. This embodiment follows the original usage... The order of terms in the query prompt is determined by concatenating the original instruction embedding vector of the trusted instruction segment with the first-order defense embedding representation of the untrusted data segment to form a complete joint input sequence. This joint input sequence maintains the natural order of instruction first and data last in the original prompt. This joint input sequence is then fed into the target large language model. The trusted instruction segment and the untrusted data segment, after being processed by two-order semantic isolation constraints, are forward encoded and propagated through the Transformer decoding block. The terms of the trusted instruction segment maintain the original embedding path and pass through all decoding layers along the original propagation path of the base model, forming the instruction hidden state sequence of the trusted instruction segment. These instruction hidden state sequences completely preserve all the semantic information of the original trusted instruction and are not interfered with by any potential attack content in the untrusted data segment. In this embodiment, the instruction hidden state sequence is used as the instruction semantic anchor. The terms of the untrusted data segment are input through the first-order defense embedding representation and, after completing the second-order defense hidden state correction in the intermediate layer, continue to participate in the subsequent decoding layer propagation, thereby ensuring that the semantic anchor of the trusted instruction is not interfered with by the residual adversarial semantics of the untrusted data segment during the formation process.

[0039] S5. In the autoregressive decoding process of retrieval enhancement generation, the hidden state of the output sequence generated at each decoding step is used as the query vector, and attention fusion decoding is performed with the instruction semantic anchor as the key value vector to generate a secure response text.

[0040] In some implementations, the step of using the hidden state of the output sequence generated at each decoding step as a query vector and the instruction semantic anchor as a key vector for attention fusion decoding to generate a secure response text during the autoregressive decoding process of retrieval enhancement includes: In the autoregressive decoding stage of retrieval enhancement generation, the current output hidden state vector generated in the current decoding step is used as the query vector for the current decoding step; Construct key vectors and value vectors based on the instruction semantic anchors, and calculate the semantic similarity between the query vector and the key vectors to obtain the attention weight of the instruction semantic anchors in the current decoding step; The value vector is weighted and aggregated using the attention weights to obtain the anchor-guided semantic vector for the current decoding step. The anchor-guided semantic vector is linearly projected and then fused with the current output hidden state vector by residual addition to obtain the fused decoded representation; Probabilistic prediction is performed on the fused decoding representation to obtain the candidate word probabilities for the current decoding step, and the word with the highest candidate word probability is selected as the safe response word for the current decoding step. All generated security response tokens are concatenated in sequence to form the security response text.

[0041] In a specific embodiment, during the autoregressive decoding output stage of the retrieval enhancement generation, for each decoding step, this embodiment uses the output hidden state obtained after the current decoding step passes through all transformer decoding layers as the query vector. This query vector contains the semantic information of the partially generated response sequence of the model. At the same time, the instruction semantic anchor sequence is used as the key vector and value vector. Cross-sequence cross-attention operation is performed through the cross-sequence attention anchoring fusion channel. The cross-sequence attention anchoring fusion channel can be implemented using single-head or multi-head scaled dot product cross-attention. This embodiment preferentially uses multi-head cross-attention. In multi-head cross-attention, this embodiment maps the output hidden state of the current decoding step to the query vector through the query projection matrix, and maps the instruction semantic anchor sequence to the key matrix and value matrix through the key projection matrix and value projection matrix, respectively. For each attention head, this embodiment calculates the scaled dot product similarity between the query vector and the key matrix. Then, this embodiment converts the scaled dot product similarity into the attention weights assigned to each instruction lexical by each attention head in the current decoding step through the normalized exponential function Softmax. The value matrix is ​​then weighted and summed using these attention weights to obtain the instruction semantic aggregation vector output by each attention head.

[0042] Next, in this embodiment, the instruction semantic aggregation vectors output by all attention heads are concatenated dimensionally, and then mapped by the multi-head output projection matrix to obtain the anchor-guided semantic vector of the cross-sequence attention anchoring fusion channel. This anchor-guided semantic vector is then added element-wise with the standard output hidden state of the current decoding step to obtain the fused decoding representation. This embodiment uses residual fusion to losslessly superimpose the instruction semantic guidance signal onto the standard output hidden state, thus preserving the model's original language generation capabilities while ensuring that each generation step is continuously constrained by credible instruction semantics, thereby generating a representation that truly reflects the user's intent. Figure 1 In this embodiment, when the retrieval enhancement generation system faces direct hint injection from user queries and indirect hint injection from external search documents, the security response text is based on the original task objective defined by the trusted instruction segment. It completely ignores or explicitly refuses to execute malicious injection instructions that may be embedded in user query statements or external document fragments, thereby generating text output that conforms to the preset task intent. The security response text generated by this embodiment forcibly ignores the instruction statements in the document, thereby cutting off the effective path of injection attacks.

[0043] In this embodiment, the weight allocation of each attention head is automatically adjusted according to the dynamic changes in the decoding context. When there is strong interference from untrusted data segments, the attention head can adaptively concentrate the weights on the most core and stable lexical units in the semantic anchor of the instruction. For example, it can ignore injected instructions or only base its responses on document-based constraints, thereby strengthening the execution of trusted instructions in parallel across multiple semantic subspaces. Finally, in this embodiment, the fused decoding representation obtained from each decoding step is fed into the head of the large language model for probability prediction. The fused decoding representation is mapped to the probability distribution of the vocabulary, which represents the probability that the next lexical unit is one of the candidate lexical units. In this embodiment, the lexical unit with the highest probability value is selected as the safe response lexical unit for the current decoding step, and the above decoding is repeated. The process generates one lexical unit at each step until an end symbol is generated or the preset maximum generation length is reached. All generated secure response lexical units are then concatenated in the generation order to form a complete secure response text. Because the guidance information of the instruction semantic anchor is forcibly injected in each decoding step, the secure response text can stably follow the task objective defined by the original trusted instruction and will not execute any pseudo-instructions or malicious operations embedded in the data segment by the attacker. It should be noted that the cross-sequence attention anchoring fusion channel is an independent output layer fusion channel. It does not change the self-attention structure inside the transformer decoding block, nor does it write to the original autoregressive decoding key-value cache, so that the model can dynamically focus on and strengthen the most relevant instruction semantic fragments when generating each lexical unit.

[0044] This embodiment trains a large language model using a composite loss function consisting of three terms: contrast preference optimization loss, manifold spacing regularization loss, and adversarial robustness loss. The specific mathematical expression is as follows: In the formula, This is a composite loss; Optimize loss to contrast preferences; is a positive manifold spacing regularization coefficient, which is used to control the strength of semantic separation; This represents the loss due to manifold spacing regularization. The adversarial loss weight coefficient is dynamically increased with each training round. It is smaller in the initial stage to avoid training instability, and gradually increased in the later stage to strengthen the defense against strong attacks. For training rounds; To combat robust loss.

[0045] In this embodiment, the core objective of the contrastive preference optimization loss is to make the model more inclined to generate safe outputs (i.e., preferred responses) that follow trusted instructions when faced with prompts containing injection attacks, rather than dangerous outputs (i.e., non-preferred responses) induced by attacks. This loss function optimizes the parameters of the policy model by comparing the log probability difference between the current model and a fixed reference model in terms of preferred and non-preferred responses. When the current model is more inclined to generate preferred responses relative to the reference model, the loss value is smaller; conversely, if the model is more inclined to generate non-preferred responses, the loss value is larger. This embodiment guides the model towards safe preference alignment by minimizing this loss. It should be noted that, while keeping trusted instructions unchanged, this embodiment inputs a data segment containing injection content into the baseline model, reference model, or rule executor to generate multiple candidate outputs. If the output follows... If the injected content is treated as data to be processed only and does not produce unsafe results, the output is marked as a preferred response. If the output executes the injection instruction, is induced to deviate from the task objective by the attack, or produces unsafe content, the output is marked as a non-preferred response. To improve efficiency, this embodiment can first automatically generate initial response pairs based on templates, and then manually review boundary samples or introduce automatic evaluation components such as task completion scorers and security classifiers for screening. This embodiment compares the generation probability differences between safe and dangerous responses, enabling the model to stably establish a consistent security preference in the retrieval enhancement generation task. Even when faced with hidden indirect hints injected in long documents, it can still prioritize the output path that follows the trusted instruction, avoiding a decrease in normal retrieval and question-answering capabilities due to global parameter updates. The mathematical expression for the comparative preference optimization loss is: In the formula, Optimize loss to contrast preferences; is the logistic function, which maps the output to the range of 0 to 1, and takes the negative logarithm as the loss value; This is a scaling hyperparameter used to control the sharpness of the contrast. The preferred response is the safe response text that the model should output, such as following trusted instructions or ignoring the injected content. Prompts for input; A reference model with frozen parameters, which is usually the initial state or the previous model; This is the model currently being trained; This is a non-preferred response, meaning the dangerous text output by the model after being induced by an attack, such as executing an injection instruction.

[0046] To prevent the data semantic manifold after double-order deflection from interpenetrating with the original instruction semantic manifold during training due to gradient interference, this embodiment introduces a manifold spacing regularization loss. This manifold spacing regularization loss can increase the projection distance between the mean embedding of trusted instruction segments and the mean embedding of untrusted data segments after first-order deflection on a certain separation direction. The specific calculation formula is as follows: In the formula, This represents the loss due to manifold spacing regularization. This is the preset lower bound for the manifold spacing, and it is a positive number; This is the unit normal vector for the semantic manifold separation direction, pointing from the data semantic centroid to the instruction semantic centroid; the superscript T is the transpose symbol. The mean vector of the embedded tokens in the trusted instruction segment; I is the mean vector of the word embeddings of the untrusted data segment after first-order semantic shift; I is the set of words in the trusted instruction segment. The number of lexical units in the trusted instruction segment lexical set; The j-th word in the trusted instruction segment; is the original embedding vector of the j-th word in the trusted instruction segment; The set of terms for untrusted data segments; The number of lexical units in the untrusted data segment lexical set; The i-th word element in the untrusted data segment; is the embedding vector of the i-th word in the untrusted data segment after first-order deflection; This represents the projected distance between two semantic centroids in the separation direction. When this projected distance is less than... When the projection distance is positive, it forces the model to increase the spacing; when the projection distance has reached or exceeded the specified value, the loss is positive. At that time, the loss is zero, and no further constraints are required.

[0047] To enhance the model's robustness against various cue hijacking attacks, particularly strong adversarial attacks based on gradient optimization, such as adversarial suffixes generated by greedy coordinate gradient attacks and general adversarial prefixes generated by neural execution attacks, this embodiment employs adversarial robust loss during training. It dynamically selects the attack sample set corresponding to the current training epoch, samples injected adversarial fragments from this set, and concatenates them with clean data segments to form adversarial input. The optimization aims to maximize the conditional logarithmic probability of the preferred response under this adversarial input, thereby generating the logarithmic probability of the preferred response under the attack input, enabling the model to learn to ignore injected instructions. The specific calculation formula is as follows: In the formula, To combat robust losses; This is an attack injection fragment sampled from a set of attack samples; This is the set of attack samples selected based on the current training round's stage k. To take the arithmetic mean of all adversarial injection fragments sampled from the attack sample set, since it is impossible to exhaustively enumerate all fragments in actual training, this embodiment can use the Monte Carlo approximation, in which only one or a small batch of adversarial fragments are randomly selected from the set in each iteration to approximate the expectation; Trusted instruction segment; To concatenate text; This is a clean, raw data segment.

[0048] The above three losses are jointly optimized through backpropagation to improve model parameters, enabling first-order deflection, second-order correction, and decoding-anchoring fusion to converge to the following state: for tokens in untrusted data segments, after two-order deflection, their semantic discrimination scores fall to one side of the data semantic manifold (i.e., they are identified as ordinary data); for tokens in trusted instruction segments, they do not undergo any deflection and always remain on one side of the instruction semantic manifold. It should be noted that in this embodiment, all linear projection layers (including the query, key, value projection matrices, and output projection matrices in the cross-sequence attention-anchoring fusion channel) are fine-tuned using low-rank parameter adaptation techniques, with a low-rank dimension of 16, a scaling factor of 8, a dropout rate of 0.05, and a global batch size of 24 during training, with a learning rate of... The three-stage course consists of three training rounds, evenly distributed across all stages. All newly added learnable parameters account for approximately 0.33% of the total parameters. In the deployment of the retrieval enhancement generation system, this embodiment only requires the introduction of a very small number of new parameters to achieve effective defense, significantly reducing the computational resource requirements and deployment threshold. At the same time, by implementing low-rank adaptation on the linear projection layer, the defense mechanism and the original capabilities of the model are compatible while maintaining the stability of the basic model's retrieval understanding and generation capabilities.

[0049] This invention provides a method for defending against prompt hijacking in search enhancement generation. The method receives user query prompts from a search enhancement generation system and parses them into trusted instruction segments and untrusted data segments. Based on the similarity between the embedded representations of each word in the untrusted data segment and the learnable semantic prototype vector, a command semantic risk score is calculated for each word in the untrusted data segment. Using the command semantic risk score as a modulation coefficient, a two-level semantic polarity shift is applied to each word in the untrusted data segment to form a two-level semantic isolation constraint. The trusted instruction segment and the untrusted data segment processed by the two-level semantic isolation constraint are forward encoded and propagated to extract the command semantic anchor point of the trusted instruction segment. During the autoregressive decoding process of search enhancement generation, the hidden state of the output sequence generated at each decoding step is used as the query vector, and attention fusion decoding is performed using the command semantic anchor point as the key value vector to generate a secure response text. Compared with existing technologies, this method, in the context of retrieval enhancement generation, parses the input into trusted instruction segments and untrusted data segments, and performs two-order semantic polarity shift and decoding attention anchoring fusion based on lexical semantic risk scores to effectively defend against direct or indirect prompt hijacking attacks. It achieves high robustness against various prompt hijacking attacks and improves the robustness and security of the retrieval enhancement generation system when facing complex adversarial attacks.

[0050] It should be noted that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0051] In one embodiment, such as Figure 2 As shown, this embodiment of the invention provides a system for preventing hijacking of prompts in search enhancement generation. The system includes: The data parsing module 101 is used to receive user query prompts from the retrieval enhancement generation system and parse the user query prompts into trusted instruction segments and untrusted data segments; the untrusted data segments include user query statements and external document fragments returned by external knowledge bases; Risk assessment module 102 is used to calculate the instructional semantic risk score of each word in the untrusted data segment based on the similarity between the embedded representation of each word in the untrusted data segment and the learnable semantic prototype vector. The semantic isolation module 103 is used to perform two-level semantic polarity shift on each word in the untrusted data segment using the instructional semantic risk score as the modulation coefficient, thereby forming a two-level semantic isolation constraint. The semantic anchoring module 104 is used to forward encode and propagate the trusted instruction segment and the untrusted data segment processed by the double-order semantic isolation constraint to extract the instruction semantic anchor point of the trusted instruction segment. The decoding generation module 105 is used in the autoregressive decoding process of retrieval enhancement generation to use the hidden state of the output sequence generated at each decoding step as the query vector, and to perform attention fusion decoding with the instruction semantic anchor as the key value vector to generate a secure response text.

[0052] In some implementations, parsing the user query prompt into trusted instruction segments and untrusted data segments specifically includes: The user query prompts are parsed in a structured segmentation based on the predefined structured prompt delimiters of the retrieval enhancement generation system. System-level task instruction content is classified into trusted instruction segments, while user query statements and external document fragments returned by external knowledge bases based on user query statements are classified into untrusted data segments.

[0053] For specific limitations regarding a prompt hijacking defense system for search-enhanced generation, please refer to the above-described limitations regarding a prompt hijacking defense method for search-enhanced generation; these will not be repeated here. Those skilled in the art will recognize that the various modules and steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware, software, or a combination of both. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0054] This invention provides a system for preventing hijacking of prompts in search enhancement generation. The system receives user query prompts from the search enhancement generation system via a data parsing module, and parses the prompts into trusted instruction segments and untrusted data segments. A risk assessment module calculates the instructional semantic risk score of each word in the untrusted data segment based on the similarity between the embedded representation of each word and the learnable semantic prototype vector. A semantic isolation module uses the instructional semantic risk score as a modulation coefficient to perform a two-level semantic polarity shift on each word in the untrusted data segment, forming a two-level semantic isolation constraint. A semantic anchoring module forward-encodes and propagates the trusted instruction segment and the untrusted data segment processed by the two-level semantic isolation constraint to extract the instructional semantic anchor point of the trusted instruction segment. During the autoregressive decoding process of search enhancement generation, the decoding generation module uses the hidden state of the output sequence generated at each decoding step as the query vector, and uses the instructional semantic anchor point as the key vector for attention fusion decoding to generate a secure response text. Compared with existing technologies, this system, in the context of retrieval enhancement generation, parses the input into trusted instruction segments and untrusted data segments, and performs two-order semantic polarity shift and decoding attention anchoring fusion based on lexical semantic risk scores to effectively defend against direct or indirect prompt hijacking attacks. It achieves high robustness against various prompt hijacking attacks and improves the robustness and security of the retrieval enhancement generation system when facing complex adversarial attacks.

[0055] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A method for preventing hijacking of prompts in search enhancement generation, characterized in that, Includes the following steps: Receive user query prompts from the search enhancement generation system and parse the user query prompts into trusted instruction segments and untrusted data segments; The untrusted data segments include user query statements and external document fragments returned by external knowledge bases; Based on the similarity between the embedding representation of each word in the untrusted data segment and the learnable semantic prototype vector, the imperative semantic risk score of each word in the untrusted data segment is calculated. Using the aforementioned directive semantic risk score as the modulation coefficient, a two-level semantic polarity shift is performed on each word in the untrusted data segment to form a two-level semantic isolation constraint. The trusted instruction segment and the untrusted data segment processed by the two-level semantic isolation constraint are forward encoded and propagated to extract the instruction semantic anchor point of the trusted instruction segment. In the autoregressive decoding process of retrieval enhancement generation, the hidden state of the output sequence generated at each decoding step is used as the query vector, and attention fusion decoding is performed using the semantic anchor of the instruction as the key value vector to generate a secure response text.

2. The method for preventing hijacking of prompts in search enhancement generation as described in claim 1, characterized in that, The step of parsing the user query prompt into trusted instruction segments and untrusted data segments includes: The user query prompts are parsed in a structured segmentation based on the predefined structured prompt delimiters of the retrieval enhancement generation system. System-level task instruction content is classified into trusted instruction segments, while user query statements and external document fragments returned by external knowledge bases based on user query statements are classified into untrusted data segments.

3. The method for preventing hijacking of prompts in search enhancement generation as described in claim 2, characterized in that: The structured prompt delimiter includes role identifiers, field identifiers, delimiter markers, and message metadata.

4. The method for preventing hijacking of prompts in search enhancement generation as described in claim 1, characterized in that, The step of calculating the instructional semantic risk score of each word in the untrusted data segment based on the similarity between the embedded representation of each word in the untrusted data segment and the learnable semantic prototype vector includes: Learnable semantic prototype vectors are pre-constructed based on semantic clustering centers in the embedding space of instructional and data content; the learnable semantic prototype vectors include instructional semantic prototype vectors and data semantic prototype vectors. For each word in the untrusted data segment, perform word embedding mapping to obtain the word embedding representation of each word, and calculate the cosine similarity between the word embedding representation and the instruction semantic prototype vector to obtain the instruction semantic similarity. Calculate the cosine similarity between the lexical embedding representation and the data semantic prototype vector to obtain the data semantic similarity; The semantic bias of each word is quantified based on the difference between the semantic similarity of the instruction and the semantic similarity of the data. The semantic bias is nonlinearly mapped to obtain the instructional semantic risk score of each word in the untrusted data segment.

5. The method for preventing hijacking of prompts in search enhancement generation as described in claim 1, characterized in that, The step of using the directive semantic risk score as the modulation coefficient to perform a two-level semantic polarity shift on each word in the untrusted data segment to form a two-level semantic isolation constraint includes: The word embedding representation of the untrusted data segment is mapped by a first-order linear projection transformation to obtain the first-order semantic deflection direction vector of each word in the untrusted data segment. Based on the command-line semantic risk score and the first-order semantic deflection direction vector, the word embedding representation of each word in the untrusted data segment is subjected to first-order semantic polarity deflection to obtain the first-order defensive embedding representation of each word. In the target large language model, a target decoding layer for second-order residual correction is pre-determined, and the first-order defensive embedding representation is input into the target large language model for forward propagation. The embedded hidden state sequence is output through the target decoding layer. The embedded hidden state sequence is segmented by the word boundary position between the untrusted data segment and the trusted instruction segment, and the intermediate hidden state corresponding to each word of the untrusted data segment is extracted. The intermediate hidden states are corrected by residual superposition based on the command semantic risk score to obtain the second-order defense hidden states corresponding to each word in the untrusted data segment. Based on the first-order defensive embedding representation and the second-order defensive hidden state, the shift of untrusted data segment lexical units to imperative content is suppressed in the embedding layer and intermediate hidden state layer of the target large language model, forming a two-order semantic isolation constraint from the embedding layer to the intermediate hidden state layer.

6. The method for preventing hijacking of prompts in search enhancement generation as described in claim 5, characterized in that, The step of performing first-order semantic polarity shift on the lexical embedding representation of each word in the untrusted data segment based on the instructional semantic risk score and the first-order semantic shift direction vector to obtain the first-order defensive embedding representation of each word includes: The first-order semantic deflection direction vector is scaled by the directive semantic risk score to obtain a deflection adjustment vector. The deflection adjustment vector is then superimposed with the word embedding representation of each word in the untrusted data segment to obtain the first-order defense embedding representation of each word.

7. The method for preventing hijacking of prompts in search enhancement generation as described in claim 5, characterized in that, The step of forward encoding and propagating the trusted instruction segment and the untrusted data segment processed by the two-order semantic isolation constraint to extract the instruction semantic anchor point of the trusted instruction segment includes: Each word in the trusted instruction segment is converted into the corresponding original instruction embedding vector, and the first-order defense embedding representation obtained after processing each word in the untrusted data segment through two-order semantic isolation constraints is integrated with the second-order defense hidden state to obtain the untrusted data segment representation vector. The original instruction embedding vector and the untrusted data segment representation vector are concatenated according to the word order in the user query prompt to obtain the joint input sequence; The joint input sequence is input into the target large language model for forward encoding propagation to obtain a high-level semantic representation; Based on the start and end lexical positions of the trusted instruction segment in the joint input sequence, the instruction hidden state vector at the corresponding position is extracted from the high-level semantic representation. The hidden state vector of the instruction is used as the instruction semantic anchor point of the trusted instruction segment.

8. The method for preventing hijacking of prompts in search enhancement generation as described in claim 1, characterized in that, The steps of generating secure response text by using the hidden state of the output sequence generated at each decoding step as the query vector and the instruction semantic anchor as the key vector for attention fusion decoding include: In the autoregressive decoding stage of retrieval enhancement generation, the current output hidden state vector generated in the current decoding step is used as the query vector for the current decoding step; Construct key vectors and value vectors based on the instruction semantic anchors, and calculate the semantic similarity between the query vector and the key vectors to obtain the attention weight of the instruction semantic anchors in the current decoding step; The value vector is weighted and aggregated using the attention weights to obtain the anchor-guided semantic vector for the current decoding step. The anchor-guided semantic vector is linearly projected and then fused with the current output hidden state vector by residual addition to obtain the fused decoded representation; Probabilistic prediction is performed on the fused decoding representation to obtain the candidate word probabilities for the current decoding step, and the word with the highest candidate word probability is selected as the safe response word for the current decoding step. All generated security response tokens are concatenated in sequence to form the security response text.

9. A suggestion hijacking defense system for enhanced retrieval generation, characterized in that, The system includes: The data parsing module is used to receive user query prompts from the retrieval enhancement generation system and parse the user query prompts into trusted instruction segments and untrusted data segments; the untrusted data segments include user query statements and external document fragments returned by external knowledge bases; The risk assessment module is used to calculate the instructional semantic risk score of each word in the untrusted data segment based on the similarity between the embedded representation of each word in the untrusted data segment and the learnable semantic prototype vector. The semantic isolation module is used to perform two-level semantic polarity shift on each word in the untrusted data segment using the instructional semantic risk score as the modulation coefficient, thereby forming a two-level semantic isolation constraint. The semantic anchoring module is used to forward encode and propagate the trusted instruction segment and the untrusted data segment processed by the double-order semantic isolation constraint to extract the instruction semantic anchor point of the trusted instruction segment. The decoding generation module is used in the autoregressive decoding process of retrieval enhancement generation to use the hidden state of the output sequence generated at each decoding step as the query vector, and the instruction semantic anchor point as the key value vector to perform attention fusion decoding to generate a secure response text.

10. A prompt hijacking defense system for enhanced retrieval generation as described in claim 9, characterized in that, The step of parsing the user query prompt into trusted instruction segments and untrusted data segments specifically includes: The user query prompts are parsed in a structured segmentation based on the predefined structured prompt delimiters of the retrieval enhancement generation system. System-level task instruction content is classified into trusted instruction segments, while user query statements and external document fragments returned by external knowledge bases based on user query statements are classified into untrusted data segments.

Images