A prompt injection defense method and system, electronic device and storage medium

Abstract

Embodiments of the present application disclose a prompt injection defense method, system, electronic device and storage medium. The method comprises: performing preliminary detection on input text to obtain a preliminary detection result; performing semantic analysis on the input text to generate a semantic understanding analysis report; obtaining current running parameters of a target large language model; obtaining behavior mode analysis results of the target large language model according to the current running parameters; and obtaining a current defense strategy of the target large language model according to the preliminary detection result, the semantic understanding analysis report and the behavior mode analysis results. The present application closely combines input text and running state of a large language model, can accurately detect prompt injection attacks on a large model and efficiently prevent malicious attacks on the large model, and can formulate corresponding defense strategies when facing complex attack scenarios, achieve more effective defense, and improve the robustness of the overall defense system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence; in particular, it relates to a prompt injection defense method and system, an electronic device and a storage medium. BACKGROUND

[0002] Large model technology has been widely applied in various industries and has promoted efficiency improvement. Through prompt engineering, input prompts are provided for large models to guide the large models to output high-quality output results. However, in the application process of prompt engineering, prompt injection attack is an attack technology in which an attacker manipulates the output of a large language model by carefully designing malicious instructions as part of the input prompts. The malicious instructions are connected with the main content so that the large language model cannot effectively distinguish the harmful information therein, which may induce the output of private information and cause data security risks.

[0003] Currently, the security detection of prompt injection is based on predefined rules. Specifically, the input data is checked according to the predefined rules to ensure that it meets the expected format and content requirements. For example, it is checked whether the input contains a specific character set, whether the length is within a reasonable range, whether it meets a specific syntax structure, and the like. However, this detection method requires manual setting of a large number of rules and is strongly dependent on the size of the rule library. In the face of dynamically changing attack methods, there is a problem of low detection accuracy.

[0004] It can be seen that how to improve the detection accuracy of large model prompt injection is a problem to be solved at present. SUMMARY

[0005] To solve the above technical problems, the embodiments of the present application provide a prompt injection defense method and system, an electronic device and a storage medium, which solve the problem of low detection accuracy of prompt injection in related technologies. The present application closely combines the input text and the running state of the large language model, which not only can accurately detect the prompt injection attack suffered by the large model and efficiently prevent the large model from being attacked maliciously, but also can formulate corresponding defense strategies when facing complex attack scenarios, realize more effective defense, and improve the robustness of the overall defense system.

[0006] According to an aspect of an embodiment of the present application, a prompt injection defense method is provided, the method comprising: performing preliminary detection on input text to obtain a preliminary detection result; the preliminary detection result comprising high risk, low risk and safe; when the preliminary detection result is low risk or safe, performing semantic analysis on the input text to generate a semantic understanding analysis report; establishing a connection with a target large language model, and obtaining current running parameters of the target large language model; according to the current running parameters, obtaining a behavior mode analysis result of the target large language model; and according to the preliminary detection result, the semantic understanding analysis report and the behavior mode analysis result, obtaining a current defense strategy of the target large language model.

[0007] Optionally, the preliminary detection on the input text to obtain a preliminary detection result comprises: screening the input text based on a pre-constructed sensitive word library to obtain a sensitive word screening result; performing syntax structure analysis on the input text to obtain a syntax structure analysis result; performing intent understanding on the input text to obtain a user's real purpose; and determining the input text as high risk, low risk or safe according to the sensitive word screening result, the syntax structure analysis result and the user's real purpose.

[0008] Optionally, the semantic analysis on the input text to generate a semantic understanding analysis report comprises: obtaining an intent type of the current input text based on a history record of multiple rounds of input; performing logical judgment on the current input text to obtain a logical rationality evaluation result; and performing risk assessment on the input text based on semantic understanding to obtain a risk assessment result; wherein the semantic understanding analysis report comprises the intent type, the logical rationality evaluation and the risk assessment result.

[0009] Optionally, obtaining an intent type of the current input text based on a history record of multiple rounds of input comprises: converting the current input text and previous multiple rounds of input text into text sequences based on word vectors to obtain a sentence sequence corresponding to each input text; performing semantic encoding on each sentence sequence to obtain a semantic representation corresponding to each input text; combining all semantic representations into a text sequence, and performing semantic encoding on the text sequence based on an attention mechanism to obtain a semantic representation of multiple rounds of input; and obtaining the intent type of the current input text according to the semantic representation corresponding to the current input text and the semantic representation of multiple rounds of input.

[0010] Optionally, semantic encoding is performed on each sentence sequence to obtain a semantic representation corresponding to each input text, including: obtaining a forward state representation and a backward state representation corresponding to each sentence sequence; obtaining a target state representation corresponding to each sentence sequence based on the forward state representation and the backward state representation; obtaining an intermediate state representation based on the target state representation and the parameter matrix; obtaining attention weights based on the similarity between the intermediate state representation and the background context vector; and obtaining a semantic representation corresponding to each input text based on the weighted sum of the target state table and the attention weights.

[0011] Optionally, the intent type of the current input text is obtained based on the semantic representation corresponding to the current input text and the semantic representation of the multi-turn input, including: calculating the correlation coefficient between the semantics of the current input text and the semantics of the context based on the semantic representation corresponding to the current input text and the semantic representation of the multi-turn input; calculating the probability distribution of the current input text and the context text in each intent category based on the correlation coefficient; and obtaining the intent type of the current input text based on the probability distribution.

[0012] Optionally, after obtaining the current defense strategy of the target large language model, the method further includes: injecting detection effect data according to the prompts after the implementation of the defense strategy, and establishing a reward preference dataset; establishing a reward model according to the reward preference dataset and the reward function; and aligning and optimizing the strategy-making agent according to the reward model.

[0013] According to one aspect of the embodiments of this application, a prompt injection defense system is provided. The system includes: an input detection agent, configured to perform preliminary detection on input text to obtain preliminary detection results; the preliminary detection results include high risk, low risk, and safe; a semantic parsing agent, configured to perform semantic analysis on the input text to generate a semantic understanding analysis report when the preliminary detection result is low risk or safe; a model protection agent, configured to establish a connection with a target large language model and obtain the current operating parameters of the target large language model; and further configured to obtain the behavior pattern analysis results of the target large language model based on the current operating parameters; and a strategy formulation agent, configured to obtain the current defense strategy of the target large language model based on the preliminary detection results, the semantic understanding analysis report, and the behavior pattern analysis results.

[0014] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the prompt injection defense method as described in the above technical solutions.

[0015] According to one aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to enable the electronic device to implement the prompt injection defense method as described in the above technical solutions.

[0016] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the prompt injection defense method as described in the above technical solutions.

[0017] The technical solution provided in this application includes at least the following beneficial effects:

[0018] This application performs preliminary detection on the input text, filtering out high-risk sensitive words and harmful instructions, and then conducts deeper semantic understanding and analysis on low-risk or safe input text. Furthermore, it evaluates the current operating parameters of the large language model to obtain behavioral pattern analysis results. Finally, based on the preliminary detection results, semantic understanding analysis package, and behavioral pattern analysis results, a corresponding defense strategy for the large language model is formulated. Therefore, by closely integrating the input text and the operating status of the large language model, this application can not only accurately detect hint injection attacks against the large model and effectively prevent malicious attacks, but also formulate corresponding defense strategies in the face of complex attack scenarios, achieving more effective defense and improving the robustness of the overall defense system. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0020] Figure 1 The diagram shown is a flowchart illustrating a method for preventing injection vulnerabilities provided in an embodiment of this application.

[0021] Figure 2 As shown Figure 1 An exemplary flowchart of step S20;

[0022] Figure 3 The diagram shown is a flowchart illustrating injection defense methods in related technologies.

[0023] Figure 4 The diagram shown is an application scenario illustration of a prompt injection defense system provided in an embodiment of this application;

[0024] Figure 5 The diagram shown is a schematic representation of a prompt injection defense system provided in an embodiment of this application.

[0025] Figure 6 The diagram shown is a schematic representation of the execution flow of a prompt injection defense system provided in an embodiment of this application.

[0026] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0030] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0031] Explanation of key terms:

[0032] An agent is an entity with the ability to perceive, make decisions, and act, capable of operating autonomously within a given environment and interacting with other agents or the environment to achieve specific goals. Agents possess the following key characteristics:

[0033] 1) Autonomy: The ability to make decisions and perform actions autonomously without direct human intervention;

[0034] 2) Perception ability: Able to acquire information about the environment through sensors or other means;

[0035] 3) Learning ability: Ability to learn from interactions with the environment and improve one's behavior and decision-making strategies;

[0036] 4) Adaptability: Ability to adapt to changes in the environment and adjust one's behavior accordingly;

[0037] 5) Goal-oriented: Having clear goals and striving to achieve them.

[0038] Current large model hinting injection attack defense methods mainly include:

[0039] (1) Rule-based approach: This method checks input data according to predefined rules to ensure it meets expected format and content requirements. For example, it checks whether the input contains a specific character set, whether the length is within a reasonable range, and whether it conforms to a specific grammatical structure. However, this approach requires manually setting a large number of rules and is heavily dependent on the size of the rule base. It is prone to failure when faced with dynamically changing attack methods.

[0040] (2) Detection based on deep learning or machine learning models: By collecting a large amount of cue injection data and building a cue injection detection model, the established model identifies unusual patterns that may indicate cue injection attacks and issues an alert when anomalies occur. This approach usually depends on the size of the collected cue injection dataset, and the established model is unlikely to perform well on new cue injections. Furthermore, a large amount of cue injection data is usually difficult to obtain.

[0041] Therefore, it can be seen that both rule-based security detection and deep learning-based model detection have limitations, such as dependence on the size of the rule base and the completeness of the dataset, and they are prone to failure when faced with dynamically changing attack methods, resulting in low detection accuracy.

[0042] To address the aforementioned issues, this application provides a method for preventing injection vulnerabilities, applicable to large models, specifically including the following embodiments:

[0043] Figure 1 The diagram shown is a flowchart illustrating a method for preventing injection vulnerabilities provided in an embodiment of this application; as follows: Figure 1 As shown, the method specifically includes the following steps:

[0044] Step S10: Perform preliminary detection on the input text to obtain preliminary detection results.

[0045] In one embodiment, preliminary detection of the input text is performed to obtain preliminary detection results, including: screening the input text based on a pre-built sensitive word database to obtain sensitive word screening results; performing grammatical structure analysis on the input text to obtain grammatical structure analysis results; performing intent understanding on the input text to obtain the user's true purpose; and determining whether the input text is high-risk, low-risk, or safe based on the sensitive word screening results, grammatical structure analysis results, and the user's true purpose; wherein, the preliminary detection results include high-risk, low-risk, and safe.

[0046] It should be noted that the preliminary detection and filtering of input text in this embodiment specifically includes: (1) text preprocessing: the user inputs prompt content, the large model receives the prompt content and performs text preprocessing on the content, such as text cleaning and normalization, syntax analysis and tagging, etc.; (2) rule-based preliminary screening: including sensitive word matching, syntax structure checking, etc., using a pre-built sensitive word library to check whether the input text contains known sensitive words, malicious command words or words that contradict specific security policies. Analyze whether the grammatical structure of the input text conforms to normal language usage habits and the system's expected interactive syntax. For example, in a specific task-oriented large model application, check whether the input follows the predetermined command format or question expression method. Abnormal grammatical structures may indicate potential injection attack attempts, such as chaotic sentence structures but containing key model operation keywords. Once grammatical abnormalities are found, they are also marked as suspicious and analyzed in depth; (3) Semantic analysis and intent inference: The preprocessed text is further input into the large model. The large model performs intent understanding and context association analysis on the text content to infer the true purpose and potential intent of the user input; (4) Comprehensive judgment analysis: The large model combines the content of the context and the detected user intent to comprehensively analyze whether there is a prompt injection attack. If the large model judges that the input may have risks such as malicious guidance of the model or bypassing security restrictions, the agent marks the input as high risk; if the large model believes that the input is basically normal but still has a certain degree of ambiguity or potential risk, it is marked as low risk; if the large model determines that the input belongs to normal, risk-free interaction, it is marked as safe; (5) Output detection results: Finally, the preliminary detection results (safe, low risk or high risk) are output to the subsequent agents so that they can take corresponding actions based on the detection results.

[0047] Step S20: When the preliminary detection result is low risk or safe, perform semantic analysis on the input text to generate a semantic understanding analysis report.

[0048] In this embodiment, semantic analysis of the input text to generate a semantic understanding analysis report includes: obtaining the intent type of the current input text based on the history of multi-round input; performing logical judgment on the current input text to obtain a logical rationality assessment result; and performing risk assessment on the input text based on semantic understanding to obtain a risk assessment result; wherein, the semantic understanding analysis report includes the intent type, the logical rationality assessment, and the risk assessment result.

[0049] It should be noted that performing deeper semantic understanding and analysis on the initially filtered input can further analyze the intent and context of low-risk or safe input text, determining whether there is a potential risk of tooltip injection; for example, analyzing whether the input is attempting to bypass normal interaction logic, or whether it is inconsistent with the model's intended use case; specifically including:

[0050] (1) Intent recognition and classification: It can identify and classify the intent of the input text, and determine whether the user is asking for information, expressing needs, giving instructions or engaging in other types of communication.

[0051] (2) Contextual understanding and association: Considering the contextual information of the current interaction, if there is a history of multiple rounds of input, the large model will integrate this contextual information to understand the complete semantics of the current input text.

[0052] (3) Logical rationality check: Analyze the logical rationality of the input text to determine whether there are semantic contradictions, unreasonable inferences, or situations that contradict normal interaction logic. Text that is seriously inconsistent with the normal interaction context may imply potential malicious injection or abnormal instructions.

[0053] (4) Injection Risk Detection: Based on semantic understanding, detect whether the input text contains hints of injection risk. Check for any veiled expressions or instructions that attempt to manipulate the behavior of large models, such as bypassing security restrictions through special semantic construction or inducing large models to output unauthorized information.

[0054] (5) Information Analysis and Transmission: Generate a semantic understanding analysis report on the input text, including intent classification results, contextual association information, logical rationality assessment, and potential risk assessment. Output the semantic analysis report to subsequent agents. If potential risks are found, a risk warning signal will be issued simultaneously so that subsequent modules can take corresponding measures, such as strengthening the monitoring of model operation, adjusting security strategies, or refusing to execute suspicious operations.

[0055] Step S30: Establish a connection with the target large language model and obtain the current running parameters of the target large language model.

[0056] In this embodiment, the target large language model is a large model that performs logical reasoning on user input text. By establishing a connection channel with the target large language model, the current operating parameters of the large model are continuously monitored. The current operating parameters include, but are not limited to, CPU utilization, response time, input data size, input data flow, input data source, output results, and the stability and consistency of the output data.

[0057] Step S40: Based on the current operating parameters, obtain the behavioral pattern analysis results of the target large language model.

[0058] It should be noted that a behavioral pattern model for the normal operation of a large model is established by accumulating long-term operational data. When a significant deviation is found between the current operating parameters and the normal behavioral data output by the behavioral pattern model, it is judged as an anomaly; for example, when the large model is processing a specific type of input, it suddenly exhibits a surge in resource consumption and unstable output results, which are different from the past. Next, a risk assessment is conducted based on the collected information from the large model and the results of behavioral pattern analysis to determine the severity of the anomaly and its potential impact, and the collected information is then transmitted to the policy-making agent.

[0059] Step S50: Based on the preliminary detection results, the semantic understanding analysis report, and the behavior pattern analysis results, obtain the current defense strategy of the target large language model.

[0060] It should be noted that preliminary detection results, semantic understanding analysis reports, and behavioral pattern analysis results from multiple data sources are collected and integrated. A situational analysis is then performed on the integrated data to determine the overall status of the current input text and the large model, identifying patterns, relationships, and trends hidden in the data and text. Based on the analysis results, a detailed report on the current operational status of the large model is generated, including an assessment of model performance, identification and classification of security risks, and predictions of future trends. Furthermore, based on the generated situational analysis report, corresponding response strategies are developed, such as information interception, access control, model performance optimization, and emergency response. Finally, after generating the current defense strategy, it is sent to the large model for execution.

[0061] Therefore, this application performs preliminary detection on the input text to filter out high-risk sensitive words and harmful instructions, and then performs deeper semantic understanding and analysis on low-risk or safe input text. In addition, by evaluating the current operating parameters of the large language model, the behavioral pattern analysis results of the large language model are obtained. Finally, based on the preliminary detection results, semantic understanding analysis package, and behavioral pattern analysis results, corresponding defense strategies for the large language model are formulated. Therefore, by closely combining the input text and the operating status of the large language model, this application can not only accurately detect hint injection attacks on the large model and effectively prevent malicious attacks on the large model, but also formulate corresponding defense strategies in the face of complex attack scenarios, achieving more effective defense and improving the robustness of the overall defense system.

[0062] In another embodiment of this application, such as Figure 2 As shown, based on the history of multi-turn input, the intent type of the current input text is obtained, which specifically includes the following steps:

[0063] Step S210: Convert the current input text and the previous multiple rounds of input text into a text sequence based on character vectors to obtain the sentence sequence corresponding to each input text.

[0064] It should be noted that in multi-turn dialogue scenarios, semantic encoding representation of the current turn's input text and the context input text is required. While it's possible to directly concatenate the multi-turn input text into a sequence and process it using a sentence-level semantic model, as the number of input turns increases and the concatenated sequence grows, a single sentence-level model cannot efficiently learn the long-range dependencies in such long sequences. This embodiment proposes a multi-level semantic encoder based on a hierarchical attention mechanism. Specifically, it encodes multi-turn input text at both the sentence and document levels, simultaneously obtaining semantic representations of the current turn and the context text. This includes sentence-level single-turn semantic encoding and document-level multi-turn semantic encoding.

[0065] In this embodiment, each input text is treated as a character-based text sequence. Each character in the text sequence is represented by a character vector, resulting in a vector sequence corresponding to each text sequence. In addition, the vector sequence is used as a sentence sequence.

[0066] Step S220: Semantically encode each sentence sequence to obtain the semantic representation corresponding to each input text.

[0067] In this embodiment, semantic encoding of each sentence sequence to obtain the semantic representation corresponding to each input text specifically includes: obtaining the forward state representation and the backward state representation corresponding to each sentence sequence; obtaining the target state representation corresponding to each sentence sequence based on the forward state representation and the backward state representation; obtaining the intermediate state representation based on the target state representation and the parameter matrix; obtaining the attention weight based on the similarity between the intermediate state representation and the background context vector; and obtaining the semantic representation corresponding to each input text based on the weighted sum of the target state table and the attention weight.

[0068] It should be noted that multi-turn dialogues consist of multiple single-turn utterances, and each single-turn utterance can be treated as a sentence-level text sequence. In the multi-level semantic encoder proposed in this embodiment, Bi-LSTM is used to encode all single-turn dialogues at the sentence level. The specific process is as follows:

[0069]

[0070] in, This indicates that the input at time t in the sentence sequence is used to obtain the forward state representation through a forward LSTM encoder. h represents the inverse state representation of the input at time t in the sentence sequence, obtained by passing it through an inverse LSTM encoder. t This represents the target state representation obtained by passing the input at time t in the sentence sequence through an LSTM.

[0071] Optionally, the attention mechanism is a structure proposed for neural machine translation tasks. When applied to text semantic representation tasks, it can learn the contribution weights of each part of the text sequence to the deep semantic representation of the text. Therefore, when processing single-turn utterances, the attention mechanism is used to learn the contribution weights of each input in the sequence to obtain a more accurate deep semantic representation. The process is as follows:

[0072] u kt =tanh(W w ×h kt +b w )

[0073]

[0074] Among them, h kt Let h represent the target state at time t of the k-th round of discourse. kt Through parameter matrix W w and b w Obtain the intermediate state representation u kt Then use this intermediate state u kt With background context vector u w The attention weight a is represented by the similarity. ktFinally, the encoded vector R of the text sequence in the kth round is obtained. sk , which represents the target state h kt Its attention weight a kt The weighted sum.

[0075] Step S230: Combine all semantic representations into a text sequence, and perform semantic encoding on the text sequence based on an attention mechanism to obtain semantic representations of multi-round inputs.

[0076] In this embodiment, the context text consists of multi-turn dialogue text. Unlike the approach of concatenating multi-turn utterances into a long sequence, this embodiment treats the multi-turn dialogue text as document-level text composed of multiple single-turn utterances for processing, thus obtaining a sentence-level single-turn semantic representation R. sk Then, the multi-turn discourse D is viewed as a sequence composed of multiple single-turn semantic representations:

[0077] D = [R] s1 ,R s2 ,…,R sk ,…,R sK ]

[0078] Among them, R sk The semantic representation of the k-th round of text sequence is used for encoding on a multi-round discourse sequence composed of multiple single-round semantic representations, employing an attention-based Bi-LSTM network consistent with single-round encoding.

[0079] h k =Bi-LSTM(R) sk )

[0080]

[0081] Among them, h k It is the target state representation of the k-th round of utterance in the sequence after Bi-LSTM encoding; R D It is a semantically weighted representation of each turn of discourse obtained by the attention mechanism, which is used as the semantic representation of the multi-turn input.

[0082] Step S240: Based on the semantic representation of the current input text and the semantic representation of the multi-turn input, obtain the intent type of the current input text.

[0083] In this embodiment, obtaining the intent type of the current input text based on the semantic representation corresponding to the current input text and the semantic representation of multiple rounds of input specifically includes the following steps: calculating the correlation coefficient between the semantics of the current input text and the semantics of the context based on the semantic representation corresponding to the current input text and the semantic representation of multiple rounds of input; calculating the probability distribution of the current input text and the context text in each intent category based on the correlation coefficient; and obtaining the intent type of the current input text based on the probability distribution.

[0084] In this embodiment, when the semantics of the dialogue change or the historical context differs from the intent of the current turn text, adding contextual information can actually become noise in the detection of the current discourse intent, negatively impacting intent detection. Therefore, it is necessary to control the degree of information sharing between contextual information and the current turn text. Thus, this embodiment uses a soft-sharing mode based on a gating mechanism, employing a bidirectional gating mechanism in the information sharing module instead of unidirectional information splicing to control the degree of information sharing between the primary and secondary tasks, and the correlation coefficient f between the current turn semantics and the context semantics. M The calculation method is as follows:

[0085] f M =σ(W f ×[R sc ×R D ]+b f )

[0086] Where σ represents the sigmoid function, W f With b f R represents the parameter matrix. sc R represents the semantic representation of the current input text. D The semantic representation of multi-turn input can be used to learn the correlation coefficient f between the semantics of the current turn and the semantics of the context through the gating mechanism described above, based on the semantic representation of the current input text and the semantic representations of multi-turn input. M This weight is then used as the shared weight in the information sharing module. After obtaining the shared weight, the shared information is represented as a weighted sum of the semantic representations in the two tasks and the shared weight:

[0087] R aux =f M ⊙R D

[0088] R main =f M ⊙R sc

[0089] Among them, R aux R represents the information shared by the auxiliary task with the main task. main This indicates information shared by the main task with the auxiliary task.

[0090] Under the gating mechanism, the scale of shared information increases or decreases depending on the semantic relevance between the current round of dialogue and the context. When the semantic relevance between the context and the current round is low, the sharing weight decreases to reduce the negative impact of noise on the result; when the semantic relevance between the context and the current round is high, the sharing weight increases to increase the scale of information sharing between the context and the current round text, thus achieving sufficient information sharing while controlling the impact of noise on the result. Finally, the semantic representations of the primary and secondary tasks are updated by concatenating them with the shared information to form a more comprehensive semantic representation R. ′ sc With R ′ D :

[0091] R ′ sc =[R sc ,R aux ]

[0092] R ′ D =[R D ,R main ]

[0093] The updated high-level semantic representation is used to obtain the probability distribution of the current round text and the context text belonging to each intent through a softmax layer:

[0094] P main =softmax(W sc ×R ′ sc +b sc )

[0095] P aux =softmax(W D ×R ′ D +b D )

[0096] Among them, P main With P aux W is the probability distribution of the current text and the context text across various intent categories. sc and W D b represents the weights of the softmax layer. sc and b D The softmax layer is biased; finally, the intent category with the highest probability distribution is taken as the intent type of the current input text.

[0097] In another embodiment of this application, after obtaining the current defense strategy of the target large language model, the method further includes: injecting detection effect data according to the prompts after the implementation of the defense strategy, and establishing a reward preference dataset; establishing a reward model according to the reward preference dataset and the reward function; and aligning and optimizing the strategy-making agent according to the reward model.

[0098] It should be noted that in this embodiment, the strategy-formulating agent obtains the current defense strategy of the target large language model based on the preliminary detection results, the semantic understanding analysis report, and the behavior pattern analysis results. To improve the accuracy of the defense strategy of the strategy-formulating agent and enhance the overall defense capability, feedback injection detection effect data after strategy implementation is collected to establish a reward preference dataset. For example, the preference dataset is {y1>y2}, indicating that for question x, strategy y1 is better than strategy y2, forming a preference alignment dataset D. A suitable reward function is defined to establish a reward model. Based on the obtained reward model, the DPO algorithm is used to align and optimize the strategy preference agent. The loss function is as follows:

[0099]

[0100] Where, π θ and π ref Let represent the language model and the reference model, respectively; σ represents the sigmoid function; and β is the weighted hyperparameter.

[0101] In one embodiment, this application provides a prompt injection defense system, specifically including the following embodiments:

[0102] Current methods for preventing injection vulnerabilities include: Figure 3 As shown, the main approach involves detecting malicious suggestion injection attacks by analyzing user input. Only safe user questions are allowed into the model for answering, while inputs detected as malicious suggestion injection attacks are blocked. However, this method heavily relies on the size of the rule base and suffers from low detection accuracy when facing dynamically changing attack methods. To address this issue, this embodiment proposes a suggestion injection defense system based on multi-agent collaboration, specifically including:

[0103] An input detection agent is used to perform preliminary detection on the input text and obtain preliminary detection results; the preliminary detection results include high risk, low risk, and safe.

[0104] A semantic parsing agent is used to perform semantic analysis on the input text and generate a semantic understanding analysis report when the preliminary detection result is low risk or safe.

[0105] The model protection agent is used to establish a connection with the target large language model and obtain the current operating parameters of the target large language model; it is also used to obtain the behavioral pattern analysis results of the target large language model based on the current operating parameters.

[0106] A strategy-formulating agent is used to obtain the current defense strategy of the target large language model based on the preliminary detection results, the semantic understanding analysis report, and the behavior pattern analysis results.

[0107] It should be noted that the application scenarios and system structure of the prompt injection defense system in this embodiment are as follows: Figure 4 and Figure 5 As shown, this system comprises four agents: an input detection agent, a semantic parsing agent, a model protection agent, and a policy formulation agent. Each agent is primarily composed of a large model and possesses capabilities such as communication, planning and reflection, memory, and tool usage. The input detection agent is responsible for initial detection and filtering of user input prompts. The semantic parsing agent is responsible for deeper semantic understanding and analysis of the initially filtered input. The model protection agent is closely integrated with the large model and is responsible for monitoring the model's operational status and output; it can detect whether the model has been affected by prompt injection. The policy formulation agent dynamically adjusts its defense strategy based on feedback from other agents. The execution flow is as follows: Figure 6 As shown, the functions of each agent will be explained in detail below:

[0108] 1. Input Detection Agent: Responsible for preliminary detection and filtering of user input prompts. It can use tools such as rule engines and blacklist mechanisms to identify and block input containing sensitive words, malicious commands, or non-compliant inputs, preventing this harmful information from directly entering the large model. For example, when user input contains dangerous words that could lead to prompt injection, such as "ignore all commands," the agent can promptly detect and block it. The specific steps of preliminary detection include:

[0109] (1) Text preprocessing: The user inputs the prompt content, the large model receives the prompt content and performs text preprocessing on the content, such as text cleaning and normalization, syntax analysis and tagging, etc.

[0110] (2) Rule-based preliminary screening: This includes sensitive word matching and grammatical structure checking. Using a pre-built sensitive word database, it checks whether the input text contains known sensitive words, malicious command words, or words that contradict specific security policies. It analyzes whether the grammatical structure of the input text conforms to normal language usage habits and the system's expected interaction syntax. For example, in a specific task-oriented large-scale model application, it checks whether the input follows a predetermined command format or question expression. Abnormal grammatical structures may indicate potential injection attack attempts, such as sentences with chaotic structures but containing key model operation keywords. Once grammatical anomalies are found, they are also marked as suspicious and analyzed in depth.

[0111] (3) Semantic analysis and intent inference: The preprocessed text is further input into the large model. The large model performs intent understanding and contextual analysis on the text content to infer the user's true purpose and potential intent.

[0112] (4) Comprehensive judgment analysis: The big model combines the context and the detected user intent to comprehensively analyze whether there is a prompt injection attack. If the big model judges that the input may have risks such as malicious guidance of the model or bypassing security restrictions, the agent marks the input as high risk. If the big model believes that the input is basically normal but still has some ambiguity or potential risk, it is marked as low risk. If the big model determines that the input is a normal and risk-free interaction, it is marked as safe.

[0113] (5) Output detection results: The agent outputs the final input detection results (safe, high risk or low risk, etc.) to subsequent agents so that they can take corresponding actions based on the detection results.

[0114] 2. Semantic Parsing Agent: This agent performs deeper semantic understanding and analysis on the initially filtered input. It can further analyze the intent and context of text deemed low-risk or safe by the input detection agent to determine if there are potential hint injection risks. For example, it analyzes whether the input is attempting to bypass normal interaction logic or whether it is inconsistent with the model's intended use case. Specific functions include:

[0115] (1) Intent recognition and classification: The large model recognizes and classifies the intent of the text. It can determine whether the user is asking for information, expressing needs, giving instructions, or engaging in other types of communication.

[0116] (2) Contextual understanding and association: Considering the contextual information of the current interaction, if there is a history of multiple rounds of dialogue, the large model will integrate this contextual information to understand the complete semantics of the current input text.

[0117] (3) Logical rationality check: Analyze the logical rationality of the input text to determine whether there are semantic contradictions, unreasonable inferences or situations that contradict the normal interaction logic. Text that is seriously inconsistent with the normal interaction context may contain potential malicious injection or abnormal instructions.

[0118] (4) Injection Risk Detection: Based on semantic understanding, detect whether the input text contains hints of injection risk. Check for any veiled expressions or instructions that attempt to manipulate the behavior of large models, such as bypassing security restrictions through special semantic construction or inducing large models to output unauthorized information.

[0119] (5) Information Analysis and Transmission: Generate a semantic understanding analysis report on the input text, including intent classification results, contextual association information, logical rationality assessment, and potential risk assessment. Output the semantic analysis report to subsequent agents. If potential risks are found, a risk warning signal will be issued simultaneously so that subsequent modules can take corresponding measures, such as strengthening the monitoring of model operation, adjusting security strategies, or refusing to execute suspicious operations.

[0120] 3. Model Protection Agent: The input detection agent and semantic parsing agent primarily inspect information from the text input side. The model protection agent is closely integrated with the target large language model (LLM) and is responsible for monitoring the model's running status and output. It can detect whether the model has been affected by cue injection, such as whether the model's output does not conform to expectations under normal circumstances, or whether it contains inappropriate or security-violating content. Once an anomaly is detected, appropriate measures are taken promptly, such as stopping the model's response and issuing alerts; specifically including:

[0121] (1) Model connection and initialization: Establish a connection channel with the target large language model to ensure that the running status information and input and output data of the large model can be obtained in real time.

[0122] (2) Operational status monitoring: Continuously monitor the operational parameters, input data flow, and output results of the large model. This includes CPU utilization, response time, input data size, source, type, and the stability and consistency of output data.

[0123] (3) Behavioral Pattern Analysis: By accumulating long-term operational data of large models, a behavioral pattern model of their normal operation is established and used as a tool for intelligent agents. When a significant deviation is found between the current operational data and the normal behavioral pattern, it is judged as a possible anomaly. For example, when the large model is processing a certain type of input, it may suddenly experience a surge in resource consumption and unstable output results, which are different from the past.

[0124] (4) Risk assessment and information transmission: Based on the collected large model information and the results of behavioral pattern analysis, risk assessment is carried out to determine the severity of the anomaly and the possible impact, and the collected information is transmitted to the strategy formulation agent.

[0125] (5) Response and Recovery: Receive decisions from the policy-making agent, such as pausing the large model's operation or initiating data backup. After the abnormal situation is handled, assist the large model in recovery operations. For example, after resolving the problem that caused the model's performance degradation, the protective agent resumes the operation of the large model and continues to monitor its restored operating status to ensure that the model can be stably put back into use.

[0126] 4. Strategy-Formulating Agent: This agent dynamically adjusts its defense strategy based on feedback from other agents. For example, upon discovering a new type of cue injection attack (environmental input), the agent can promptly update the filtering rules of the input detection agent, adjust the judgment methods of the semantic parsing agent, or optimize the monitoring mechanism of the model protection agent to enhance overall defense capabilities. Specifically, this includes:

[0127] (1) Information collection and integration: Collect information related to the large model's operating environment, performance, security status, and external threat intelligence from other intelligent agents and multiple data sources, and integrate the data.

[0128] (2) Situation analysis: The strategy formulation agent performs an overall situation analysis based on the obtained agent, determines the overall situation of the current input text and the large model, identifies the patterns, relationships and trends hidden in the data and text, and generates a detailed report on the current operating status of the large model based on the analysis results, including the evaluation of the model performance status, the identification and classification of security risks, and the prediction of future development trends.

[0129] (3) Strategy Generation and Optimization: Based on the generated situational analysis report, the strategy-formulating agent generates corresponding response strategies, such as information interception, access control, model performance optimization, and emergency response. Furthermore, after generating the initial strategy, the agent can simulate, evaluate, and optimize the strategy, and finally output the optimal strategy for the current state as feedback to the defense agent for execution.

[0130] (4) Agent Optimization: Collect feedback injection detection effect data after policy implementation to establish a reward preference dataset {y1>y2}, indicating that for problem x, policy y1 is better than policy y2, forming a preference alignment dataset D. Define a suitable reward function and establish a reward model. Based on the obtained reward model, use the DPO algorithm to align and optimize the policy preference agent. The loss function is as follows:

[0131]

[0132] Where, π θ and π ref Let represent the language model and the reference model, respectively; σ represents the sigmoid function; and β is the weighted hyperparameter.

[0133] In this embodiment, the environment refers to the external conditions and scenarios of the entire prompt injection defense system. It is primarily used to provide feedback on the system's behavior, interactions, and task completion status, prompting the system to adjust and optimize. Here, the environment can be human feedback, the social environment, laws and regulations, the technological environment, etc., and it possesses dynamic characteristics. The intelligent agent can obtain relevant information in real time through interaction and communication with the environment, promoting the agent's evolution.

[0134] This application proposes a dynamic defense method for large model hint injection based on multi-agent collaboration. This method not only accurately detects hint injection attacks on large models by leveraging the collaborative capabilities of multiple agents, effectively preventing malicious attacks, but also proposes a mechanism for multiple agents to collaborate and improve decision-making capabilities. This further enhances the accuracy of large models in identifying hint attacks and their response strategies to malicious attacks. In complex attack scenarios, multiple agents can jointly formulate defense strategies, leveraging their respective strengths to achieve more effective defense and improve the robustness of the overall defense system.

[0135] Currently, large-scale modeling technology is applied across various industries, bringing new opportunities and empowering businesses in different sectors. However, hint injection attacks pose new challenges to the security of various industry applications, such as fund security in the financial industry, patient information in healthcare, and data leakage in the cyber domain. Therefore, the method described in this application can be applied to all current large-scale modeling applications in the industry, providing effective protection for their security. Furthermore, the method described in this application can be used to defend against large-scale model hint injection attacks, thus its potential value is not limited to the following:

[0136] (1) Protecting data security: In many application scenarios, large language models may come into contact with sensitive information, so the method of this patent can effectively prevent data leakage.

[0137] (2) System security and stability: A successful hint injection attack may allow the attacker to gain control of the large language model system and then perform malicious operations. Therefore, the method of this patent can effectively maintain the overall security of the system.

[0138] (3) Protect corporate reputation and user rights: It should be noted that injection attacks may damage corporate reputation or obtain users' personal information and do something that infringes on users' rights. Effective defense can improve user experience and promote positive interaction between users and enterprises.

[0139] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0140] It should be noted that, Figure 7 The computer system 1000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0141] like Figure 7 As shown, the computer system 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1002 or programs loaded from storage portion 1008 into Random Access Memory (RAM) 1003, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1003. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. An Input / Output (I / O) interface 1005 is also connected to bus 1004.

[0142] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0143] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of this application.

[0144] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0147] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A method for preventing injection vulnerabilities, characterized in that, The method includes: The input text is subjected to preliminary detection to obtain preliminary detection results; the preliminary detection results include high risk, low risk, and safe. When the preliminary detection result is low risk or safe, semantic analysis is performed on the input text to generate a semantic understanding analysis report; Establish a connection with the target large language model and obtain the current running parameters of the target large language model; Based on the current operating parameters, obtain the behavioral pattern analysis results of the target large language model; Based on the preliminary detection results, the semantic understanding analysis report, and the behavioral pattern analysis results, the current defense strategy for the target large language model is obtained; The semantic analysis of the input text to generate a semantic understanding analysis report includes: converting the current input text and previous multi-round input texts into a text sequence based on character vectors to obtain a sentence sequence corresponding to each input text; performing semantic encoding on each sentence sequence to obtain a semantic representation corresponding to each input text; combining all semantic representations into a text sequence, and performing semantic encoding on the text sequence based on an attention mechanism to obtain a semantic representation of the multi-round input; obtaining the intent type of the current input text based on the semantic representation of the current input text and the semantic representation of the multi-round input; performing logical judgment on the current input text to obtain a logical rationality assessment result; and performing risk assessment on the input text based on semantic understanding to obtain a risk assessment result. The semantic understanding analysis report includes the intent type, the logical rationality assessment, and the risk assessment result.

2. The method according to claim 1, characterized in that, The input text is subjected to preliminary detection, and the preliminary detection results are obtained, including: The input text is screened based on a pre-built sensitive word database to obtain sensitive word screening results; Perform syntactic structure analysis on the input text to obtain the syntactic structure analysis results; The user's true purpose is determined by performing intent understanding on the input text. Based on the results of the sensitive word screening, the results of the grammatical structure analysis, and the user's true purpose, the input text is determined to be high-risk, low-risk, or safe.

3. The method according to claim 1, characterized in that, Semantic encoding is performed on each sentence sequence to obtain the semantic representation corresponding to each input text, including: Obtain the forward and backward state representations for each sentence sequence; Based on the forward state representation and the backward state representation, the target state representation corresponding to each sentence sequence is obtained; Based on the target state representation and parameter matrix, the intermediate state representation is obtained; Attention weights are obtained based on the similarity between the intermediate state representation and the background context vector; Based on the weighted sum of the target state table and the attention weights, the semantic representation corresponding to each input text is obtained.

4. The method according to claim 1, characterized in that, Based on the semantic representation of the current input text and the semantic representation of multiple rounds of input, the intent type of the current input text is obtained, including: Based on the semantic representation of the current input text and the semantic representation of multiple rounds of input, the correlation coefficient between the semantics of the current input text and the semantics of the context is calculated; Based on the correlation coefficient, the probability distribution of the current input text and the context text in each intent category is calculated; Based on the probability distribution, the intent type of the current input text is obtained.

5. The method according to claim 1, characterized in that, After obtaining the current defense strategy of the target large language model, the method further includes: Based on the prompts after the implementation of the defense strategy, inject detection effect data to establish a reward preference dataset; Based on the aforementioned reward preference dataset and reward function, a reward model is established; The policy-making agent is aligned and optimized based on the reward model.

6. A system for preventing injection of warning signals, characterized in that, The system includes: An input detection agent is used to perform preliminary detection on the input text and obtain preliminary detection results; the preliminary detection results include high risk, low risk, and safe. A semantic parsing agent is used to perform semantic analysis on the input text and generate a semantic understanding analysis report when the preliminary detection result is low risk or safe. The model protection agent is used to establish a connection with the target large language model and obtain the current operating parameters of the target large language model; it is also used to obtain the behavioral pattern analysis results of the target large language model based on the current operating parameters. A strategy-formulating agent is used to obtain the current defense strategy of the target large language model based on the preliminary detection results, the semantic understanding analysis report, and the behavior pattern analysis results. The semantic parsing agent is further configured to convert the current input text and previous multi-round input text into a text sequence based on character vectors, obtaining a sentence sequence corresponding to each input text; perform semantic encoding on each sentence sequence to obtain a semantic representation corresponding to each input text; combine all semantic representations into a text sequence, and perform semantic encoding on the text sequence based on an attention mechanism to obtain a semantic representation of the multi-round input; obtain the intent type of the current input text based on the semantic representation corresponding to the current input text and the semantic representation of the multi-round input; perform logical judgment on the current input text to obtain a logical rationality assessment result; and perform risk assessment on the input text based on semantic understanding to obtain a risk assessment result; wherein the semantic understanding analysis report includes the intent type, the logical rationality assessment, and the risk assessment result.

7. An electronic device, characterized in that, include: processor; as well as A memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to enable the electronic device to implement the prompt injection defense method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the prompt injection defense method as described in any one of claims 1 to 5.

Images