Prompt injection defense method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a prompt injection defense method and system, electronic equipment and a storage medium, and the method comprises the steps: carrying out the preliminary detection of an input text, and obtaining a preliminary detection result; performing semantic analysis on the input text to generate a semantic understanding analysis report; obtaining current operation parameters of the target large language model; according to the current operation parameters, obtaining a behavior pattern analysis result of the target large language model; obtaining a current defense strategy of the target large language model according to the preliminary detection result, the semantic comprehension analysis report and the behavior pattern analysis result; according to the method, the input text and the running state of the large language model are closely combined, so that the prompt injection attack suffered by the large model can be accurately detected, the large model can be efficiently prevented from being hostile attacked, and the corresponding defense strategy can be formulated in a complex attack scene; more effective defense is realized, and the robustness of the whole defense system is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology; specifically, to a prompt injection defense method, system, electronic device and storage medium. Background Art

[0002] Big model technology has been widely used in various industries and has promoted efficiency improvements. Prompt engineering provides input prompts for big models and guides them to output high-quality output results. However, in the application of prompt engineering, prompt injection attacks are an attack technology in which attackers use carefully designed malicious instructions as part of the input prompts to manipulate the output of large language models. Malicious instructions are connected to the main content, making it impossible for the big language model to effectively distinguish the harmful information, which may induce the output of privacy information and bring data security risks.

[0003] Currently, security detection of prompt injection is performed based on predefined rules. Specifically, the input data is checked according to the predefined rules to ensure that it meets the expected format and content requirements; for example, whether the input contains a specific character set, whether the length is within a reasonable range, whether it conforms to a specific grammatical structure, etc. However, this detection method requires manual setting of a large number of rules and is highly dependent on the size of the rule base. In the face of dynamically changing attack methods, there is a problem of low detection accuracy.

[0004] It can be seen that how to improve the detection accuracy of large model prompt injection is a problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present application provide a prompt injection defense method, system, electronic device and storage medium, which solves the problem of low detection accuracy of prompt injection in related technologies. The present application closely combines the two aspects of input text and the running status of a large language model. It can not only accurately detect prompt injection attacks on large models and efficiently prevent malicious attacks on large models, but also formulate corresponding defense strategies when facing complex attack scenarios, achieve more effective defense, and improve the robustness of the overall defense system.

[0006] According to one aspect of an embodiment of the present application, a prompt injection defense method is provided, the method comprising: performing a preliminary detection on an input text to obtain a preliminary detection result; the preliminary detection result includes high risk, low risk and safety; when the preliminary detection result is low risk or safe, performing a semantic analysis on the input text to generate a semantic understanding analysis report; establishing a connection with a target large language model to obtain the current operating parameters of the target large language model; according to the current operating parameters, obtaining a behavior pattern analysis result of the target large language model; according to the preliminary detection result, the semantic understanding analysis report and the behavior pattern analysis result, obtaining the current defense strategy of the target large language model.

[0007] Optionally, a preliminary test is performed on the input text to obtain a preliminary test result, including: screening the input text based on a pre-built sensitive word library to obtain a sensitive word screening result; performing a grammatical structure analysis on the input text to obtain a grammatical structure analysis result; performing intent understanding on the input text to obtain the user's true purpose; and determining whether the input text is high risk, low risk or safe based on the sensitive word screening result, the grammatical structure analysis result and the user's true purpose.

[0008] Optionally, a semantic analysis is performed on the input text to generate a semantic understanding analysis report, including: obtaining the intention type of the current input text based on historical records of multiple rounds of input; performing logical judgment on the current input text to obtain a logical rationality assessment result; performing a risk assessment on the input text based on semantic understanding to obtain a risk assessment result; wherein the semantic understanding analysis report includes the intention type, the logical rationality assessment and the risk assessment result.

[0009] Optionally, based on the historical records of multiple rounds of input, the intent type of the current input text is obtained, including: converting the current input text and the previous multiple rounds of input text into a text sequence based on word vectors to obtain a sentence sequence corresponding to each input text; semantically encoding each sentence sequence to obtain a semantic representation corresponding to each input text; combining all semantic representations into a text sequence, semantically encoding the text sequence based on an attention mechanism to obtain a semantic representation of multiple rounds of input; and obtaining the intent type of the current input text based on the semantic representation corresponding to the current input text and the semantic representations of multiple rounds of input.

[0010] Optionally, each sentence sequence is semantically encoded to obtain a semantic representation corresponding to each input text, including: obtaining a forward state representation and a reverse state representation corresponding to each sentence sequence; obtaining a target state representation corresponding to each sentence sequence based on the forward state representation and the reverse state representation; obtaining an intermediate state representation based on the target state representation and a parameter matrix; obtaining an attention weight based on the similarity between the intermediate state representation and the background context vector; and obtaining a semantic representation corresponding to each input text based on a weighted sum of the target state table and the attention weight.

[0011] Optionally, the intention type of the current input text is obtained based on the semantic representation corresponding to the current input text and the semantic representations of multiple rounds of input, including: calculating the correlation coefficient between the semantics of the current input text and the semantics of the context based on the semantic representation corresponding to the current input text and the semantic representations of multiple rounds of input; based on the correlation coefficient, calculating the probability distribution of the current input text and the context in each intention category; and obtaining the intention type of the current input text based on the probability distribution.

[0012] Optionally, after obtaining the current defense strategy of the target large language model, the method further includes: establishing a reward preference data set based on the prompt injection detection effect data after the implementation of the defense strategy; establishing a reward model based on the reward preference data set and the reward function; and aligning and optimizing the strategy formulation agent based on the reward model.

[0013] According to one aspect of an embodiment of the present application, a prompt injection defense system is provided, the system comprising: an input detection agent, used to perform preliminary detection on an input text to obtain preliminary detection results; the preliminary detection results include high risk, low risk and safety; a semantic parsing agent, used to perform semantic analysis on the input text to generate a semantic understanding analysis report when the preliminary detection result is low risk or safe; a model protection agent, used to establish a connection with a target large language model and obtain the current operating parameters of the target large language model; and also used to obtain the behavior pattern analysis results of the target large language model according to the current operating parameters; a strategy formulation agent, used to obtain the current defense strategy of the target large language model according to the preliminary detection results, the semantic understanding analysis report and the behavior pattern analysis results.

[0014] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the prompt injection defense method in the above technical solution is implemented.

[0015] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions so that the electronic device implements the prompt injection defense method as in the above technical solution.

[0016] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the prompt injection defense method in the above technical solution is implemented.

[0017] The technical solution provided by this application includes at least the following beneficial effects:

[0018] This application performs preliminary detection on the input text, filters high-risk sensitive words and harmful instructions, and then performs a deeper semantic understanding and analysis on low-risk or safe input text; in addition, by evaluating the current operating parameters of the large language model, the behavior pattern analysis results of the large prediction model are obtained, and finally a corresponding large language model defense strategy is formulated according to the preliminary detection results, the semantic understanding analysis package and the behavior pattern analysis results; therefore, this application closely combines the two aspects of input text and the operating status of the large language model, which can not only accurately detect the prompt injection attack on the large model and efficiently prevent the large model from malicious attacks, but also formulate corresponding defense strategies when facing complex attack scenarios, achieve more effective defense, and improve the robustness of the overall defense system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 The figure is a flow chart of a prompt injection defense method provided by an embodiment of the present application;

[0021] Figure 2 Shown Figure 1 An exemplary flow chart of step S20 in FIG.

[0022] Figure 3 Shown is a flowchart of a prompt injection defense method in the related art;

[0023] Figure 4 The figure shows a schematic diagram of an application scenario of a prompt injection defense system provided by an embodiment of the present application;

[0024] Figure 5 The figure is a schematic diagram of the structure of a prompt injection defense system provided by an embodiment of the present application;

[0025] Figure 6 The figure is a schematic diagram of an execution flow of a prompt injection defense system provided by an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0028] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0030] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0031] Key terms explained:

[0032] Agent: An entity that has the ability to perceive, make decisions, and act, can operate autonomously in a certain environment, and interact with other agents or environments to achieve specific goals. Agents have the following key characteristics:

[0033] 1) Autonomy: The ability to make decisions and perform actions autonomously without direct human intervention;

[0034] 2) Perception: Ability to obtain information about the environment through sensors or other means;

[0035] 3) Learning ability: Ability to learn from interactions with the environment and improve one’s behavior and decision-making strategies;

[0036] 4) Adaptability: Ability to adapt to changes in the environment and adjust one's behavior accordingly;

[0037] 5) Goal-oriented: Having clear goals and working hard to achieve them.

[0038] The current large model prompt injection attack defense methods mainly include:

[0039] (1) Rule-based: Check input data according to predefined rules to ensure that it meets the expected format and content requirements. For example, check whether the input contains a specific character set, whether the length is within a reasonable range, whether it conforms to a specific grammatical structure, etc. However, this method requires manual setting of a large number of rules and is highly dependent on the size of the rule base. In the face of dynamically changing attack methods, this method is easily ineffective.

[0040] (2) Detection based on deep learning or machine learning models: By collecting a large amount of prompt injection data and establishing a prompt injection detection model, the established model identifies unusual patterns that may indicate prompt injection attacks and issues an alarm when an anomaly occurs. This method usually relies on the size of the collected prompt injection data set. The established model is difficult to show good results on new prompt injections, and a large amount of prompt injection data is usually difficult to obtain.

[0041] From this, we can see that both rule-based security detection and deep learning-based model detection have limitations, such as relying on the size of the rule base and the integrity of the data set, and are prone to failure when facing dynamically changing attack methods. Both have the problem of low detection accuracy.

[0042] In order to solve the above problems, the present application provides a prompt injection defense method, which is applied to large models and specifically includes the following embodiments:

[0043] Figure 1 FIG. 1 is a flow chart of a prompt injection defense method provided by an embodiment of the present application; Figure 1 As shown, the method specifically comprises the following steps:

[0044] Step S10: Perform a preliminary detection on the input text to obtain a preliminary detection result.

[0045] In one embodiment, a preliminary test is performed on the input text to obtain a preliminary test result, including: screening the input text based on a pre-built sensitive word library to obtain a sensitive word screening result; performing a grammatical structure analysis on the input text to obtain a grammatical structure analysis result; understanding the intent of the input text to obtain the user's true purpose; determining whether the input text is high risk, low risk or safe based on the sensitive word screening result, the grammatical structure analysis result and the user's true purpose; wherein the preliminary test result includes high risk, low risk and safe.

[0046] It should be noted that the preliminary detection and filtering of input text in this embodiment specifically includes: (1) Text preprocessing: the user inputs prompt content, the big model receives the prompt content and performs text preprocessing on the content, such as text cleaning and normalization, grammatical analysis and tagging, etc.; (2) Rule-based preliminary screening: including sensitive word matching, grammatical structure checking, etc., using a pre-built sensitive word library to check whether the input text contains known sensitive words, malicious instruction words, or words that are contrary to specific security policies. Analyze whether the grammatical structure of the input text conforms to normal language usage habits and the interactive grammar expected by the system. For example, in a specific task-oriented big model application, check whether the input follows the predetermined command format or question expression method. Abnormal grammatical structure may indicate potential injection attack attempts, such as situations where the sentence structure is chaotic but contains key model operation keywords. Once a grammatical abnormality is found, it is also marked as suspicious and analyzed in depth; (3) Semantic analysis and intent inference: The text after data preprocessing is further input into the large model. The large model performs intent understanding and context association analysis on the text content to infer the real purpose and potential intention of the user input; (4) Comprehensive judgment analysis: The large model combines the content of the context and the detected user intent to comprehensively analyze whether it contains prompts for injection attacks. If the large model determines that the input may have risks such as maliciously guiding the model or bypassing security restrictions, the intelligent agent will mark the input as high risk; if the large model believes that the input is basically normal but still has certain ambiguity or potential risks, it will be marked as low risk; if the large model determines that the input belongs to a normal, risk-free interaction, it will be marked as safe; (5) Output detection results: Finally, the preliminary detection results (safe, low risk or high risk) are output to subsequent intelligent agents so that they can take corresponding actions based on the detection results.

[0047] Step S20: When the preliminary detection result is low risk or safe, semantic analysis is performed on the input text to generate a semantic understanding analysis report.

[0048] In this embodiment, a semantic analysis is performed on the input text to generate a semantic understanding analysis report, including: obtaining the intention type of the current input text based on historical records of multiple rounds of input; performing logical judgment on the current input text to obtain a logical rationality assessment result; performing a risk assessment on the input text based on semantic understanding to obtain a risk assessment result; wherein the semantic understanding analysis report includes the intention type, the logical rationality assessment and the risk assessment result.

[0049] It should be noted that a deeper semantic understanding and analysis of the input after preliminary filtering can further parse the input intent, context and other information of low-risk or safe input text to determine whether there is a potential risk of prompt injection; for example, analyze whether the input is trying to bypass the normal interaction logic, or whether it is inconsistent with the expected usage scenario of the model; specifically:

[0050] (1) Intent recognition and classification: Identify and classify the intent of the input text. It can determine whether the user is asking for information, expressing a need, giving instructions, or engaging in other types of communication.

[0051] (2) Contextual understanding and association: Considering the contextual information of the current interaction, if there is a historical record of multiple rounds of input, the large model will integrate this contextual information to understand the complete semantics of the current input text.

[0052] (3) Logical rationality check: Analyze the logical rationality of the input text to determine whether there are semantic contradictions, unreasonable inferences, or situations that are contrary to normal interaction logic. Text that is seriously inconsistent with the normal interaction context may contain potential malicious injections or abnormal instructions.

[0053] (4) Injection risk detection: Based on semantic understanding, detect whether the input text indicates injection risk. Check whether there are any obscure statements or instructions that attempt to manipulate the behavior of the big model, such as bypassing security restrictions through special semantic constructions, inducing the big model to output unauthorized information, etc.

[0054] (5) Information analysis and information transmission: Generate a semantic understanding analysis report on the input text, including intent classification results, contextual information, logical rationality assessment, and potential risk assessment. The semantic analysis report is output to the subsequent intelligent agent. If potential risks are found, a risk warning signal will be issued at the same time so that the subsequent modules can take corresponding measures, such as strengthening the monitoring of model operation, adjusting security policies, or refusing to perform suspicious operations.

[0055] Step S30: Establish a connection with the target large language model and obtain the current operating parameters of the target large language model.

[0056] In this embodiment, the target large language model is a large model that performs logical reasoning on user input text. By establishing a connection channel with the target large language model, the current operating parameters of the large model are continuously monitored. The current operating parameters include but are not limited to CPU usage, response time, input data scale, input data flow, input data source, output results, and stability and consistency of output data.

[0057] Step S40: Obtain a behavior pattern analysis result of the target large language model according to the current operating parameters.

[0058] It should be noted that by accumulating the long-term operation data of the large model, a behavior pattern model of its normal operation is established. When it is found that the current operation parameters are significantly deviated from the normal behavior data output by the behavior pattern model, it is determined to be an abnormal situation; for example, when the large model processes a certain type of specific input, there is a sudden increase in resource consumption and unstable output results that are different from the past; next, based on the collected large model information and the results of the behavior pattern analysis, a risk assessment is conducted to determine the severity of the abnormality and the possible impact, and the collected information is passed to the strategy formulation agent.

[0059] Step S50: Obtain the current defense strategy of the target large language model according to the preliminary detection result, the semantic understanding analysis report and the behavior pattern analysis result.

[0060] It should be noted that the relevant information such as preliminary detection results, semantic understanding analysis reports, and behavioral pattern analysis results of large models collected from multiple data sources are integrated; and the integrated data is analyzed for situation, the overall situation of the current input text and the large model is determined, and the patterns, relationships and trends hidden in the data and text are identified. Based on the results of the analysis, a detailed report on the current operation situation of the large model is generated, including an assessment of the model performance status, identification and classification of security risks, and predictions of future development trends. Further, based on the generated situation analysis report, corresponding response strategies are generated, such as information interception, access control, model performance optimization, and emergency response; further, after the current defense strategy is generated, it is sent to the large model for execution.

[0061] It can be seen that the present application performs preliminary detection on the input text, filters high-risk sensitive words and harmful instructions, and then performs a deeper semantic understanding and analysis on the low-risk or safe input text; in addition, by evaluating the current operating parameters of the large language model, the behavior pattern analysis results of the large prediction model are obtained, and finally, the corresponding large language model defense strategy is formulated according to the preliminary detection results, the semantic understanding analysis package and the behavior pattern analysis results; therefore, the present application closely combines the two aspects of the input text and the operating status of the large language model, which can not only accurately detect the prompt injection attack on the large model and efficiently prevent the large model from malicious attacks, but also formulate corresponding defense strategies when facing complex attack scenarios, so as to achieve more effective defense and improve the robustness of the overall defense system.

[0062] In another embodiment of the present application, Figure 2 As shown, based on the historical records of multiple rounds of input, the intent type of the current input text is obtained, which specifically includes the following steps:

[0063] Step S210: convert the current input text and the previous multiple rounds of input text into a text sequence based on word vectors to obtain a sentence sequence corresponding to each input text.

[0064] It should be noted that in the scenario of multi-round dialogue, it is necessary to perform semantic encoding representation on the current round input text and the context input text. The multi-round input text can be directly concatenated into a sequence, and a sentence-level semantic model can be used to process it. However, as the number of input rounds increases, the concatenated sequence grows, and a single sentence-level model cannot efficiently learn long-range dependencies in ultra-long sequences. This embodiment proposes a multi-level semantic encoder based on a hierarchical attention mechanism. Specifically, it encodes multi-round input texts at both the sentence level and the document level, and can simultaneously obtain the semantic representation of the current round and the context text, specifically including single-round semantic encoding at the sentence level and multi-round semantic encoding at the document level.

[0065] In this embodiment, each input text is regarded as a word-based text sequence, each word in the text sequence is represented by a word vector, and a vector sequence corresponding to each text sequence is obtained, and the vector sequence is regarded as a sentence sequence.

[0066] Step S220: semantically encode each sentence sequence to obtain a semantic representation corresponding to each input text.

[0067] In this embodiment, semantic encoding is performed on each sentence sequence to obtain the semantic representation corresponding to each input text, which specifically includes: obtaining the forward state representation and the reverse state representation corresponding to each sentence sequence; obtaining the target state representation corresponding to each sentence sequence according to the forward state representation and the reverse state representation; obtaining the intermediate state representation according to the target state representation and the parameter matrix; obtaining the attention weight according to the similarity between the intermediate state representation and the background context vector; and obtaining the semantic representation corresponding to each input text based on the weighted sum of the target state table and the attention weight.

[0068] It should be noted that a multi-round dialogue consists of multiple single-round utterances, and each single-round utterance can be treated as a text sequence at the sentence level. In the multi-level semantic encoder proposed in this embodiment, Bi-LSTM is used to encode all single-round dialogues at the sentence level. The specific process is as follows:

[0069]

[0070] in, The input at time t in the sentence sequence is represented by the forward state through the forward LSTM encoder. represents the reverse state representation obtained by the reverse LSTM encoder at time t in the sentence sequence, h t Represents the target state representation obtained by LSTM at time t in the sentence sequence.

[0071] Optionally, the attention mechanism is a structure proposed under the neural machine translation task. When it is applied to the text semantic representation task, it can learn the contribution weight of each part in the text sequence to the deep semantic representation of the text. Therefore, when processing a single round of speech, the attention mechanism is used to learn the contribution weight of each input in the sequence to obtain a more accurate deep semantic representation. The process is as follows:

[0072] u kt =tanh(W w ×h kt +b w )

[0073]

[0074] Among them, h kt represents the target state at time t of the kth round of speech, h kt Through the parameter matrix W w and b w Get the intermediate state representation u kt ; Then use this intermediate state u kt With the background context vector u w The similarity of ktFinally, we get the encoding vector R of the k-th round text sequence sk , which represents the target state h kt Its attention weight a kt The weighted sum of .

[0075] Step S230: combine all semantic representations into a text sequence, perform semantic encoding on the text sequence based on an attention mechanism, and obtain semantic representations of multiple rounds of input.

[0076] In this embodiment, the context text is composed of multi-round dialogue texts. Different from the practice of splicing multi-round utterances into a long sequence, this embodiment treats the multi-round dialogue texts as document-level texts composed of multiple single-round utterances for processing. sk Finally, the multi-turn discourse D is regarded as a sequence composed of multiple single-turn semantic representations:

[0077] D=[R s1 ,R s2 ,…,R sk ,…,R sK ]

[0078] Among them, R sk Represents the semantic representation of the k-th round text sequence. On the multi-round discourse sequence composed of multiple single-round semantic representations, a Bi-LSTM network based on the attention mechanism consistent with the single-round encoding is used for encoding:

[0079] h k =Bi-LSTM(R sk )

[0080]

[0081] Among them, h k is the target state representation of the kth round of speech in the sequence after Bi-LSTM encoding; R D It is the semantic weighted representation of each round of discourse obtained by the attention mechanism, which is used as the semantic representation of multiple rounds of input.

[0082] Step S240: Obtain the intention type of the current input text according to the semantic representation corresponding to the current input text and the semantic representations of multiple rounds of input.

[0083] In this embodiment, obtaining the intention type of the current input text according to the semantic representation corresponding to the current input text and the semantic representation of multiple rounds of input specifically includes the following steps: calculating the correlation coefficient between the semantics of the current input text and the semantics of the context according to the semantic representation corresponding to the current input text and the semantic representation of multiple rounds of input; calculating the probability distribution of the current input text and the context in each intention category based on the correlation coefficient; and obtaining the intention type of the current input text according to the probability distribution.

[0084] In this embodiment, when the semantics of the conversation changes and the historical context is different from the intention of the current turn text, the addition of context information will become noise for the current speech intention detection, which will have a negative impact on the intention detection; therefore, it is necessary to control the degree of information sharing between the context information and the current turn text. Therefore, this embodiment uses a two-way gating mechanism in the information sharing module instead of a one-way information splicing to control the degree of information sharing between the main and auxiliary tasks. The correlation coefficient f between the current turn semantics and the context semantics is M The calculation method is:

[0085] f M =σ(W f ×[R sc ×R D ]+b f )

[0086] Among them, σ represents the sigmoid function, W f With b f represents the parameter matrix, R sc Represents the semantic representation of the current input text, R D Represents the semantic representation of multiple rounds of input. Through the above formula, the gating mechanism can learn the correlation coefficient f between the current round semantics and the context semantics through the semantic representation of the current input text and the semantic representation of multiple rounds of input. M , and use it as the shared weight in the information sharing module. After obtaining the shared weight, the shared information is represented as the weighted sum of the semantic representation in the two tasks and the shared weight:

[0087] R aux =f M ⊙R D

[0088] R main =f M ⊙R sc

[0089] Among them, R aux Represents the information shared by the auxiliary task to the main task, R main Indicates the information shared by the main task to the auxiliary task.

[0090] Under the gating mechanism, the scale of shared information will increase or decrease with the semantic relevance between the current round of dialogue and the context. When the relevance between the context and the current round of semantics is low, the sharing weight will be reduced to reduce the negative impact of noise information on the results; when the relevance between the context and the current round of semantics is high, the sharing weight will be increased to increase the scale of information sharing between the context information and the current round of text, thereby achieving full information sharing while controlling the impact of noise information on the results. Finally, the semantic representations of each task in the main and auxiliary tasks are updated by splicing with the shared information to form a more comprehensive semantic representation R. ′ sc With R ′ D :

[0091] R ′ sc =[R sc ,R aux ]

[0092] R ′ D =[R D ,R main ]

[0093] The updated high-level semantic representation is passed through the softmax layer to obtain the probability distribution of the current turn text and the context text belonging to each intent:

[0094] P main =softmax(W sc ×R ′ sc +b sc )

[0095] P aux =softmax(W D ×R ′ D +b D )

[0096] Among them, P main With P aux is the probability distribution of the current round text and the context text in each intent category, W sc and W D is the weight of the softmax layer, b sc and b D is the bias of the softmax layer; finally, the highest intent category in the probability distribution is taken as the intent type of the current input text.

[0097] In another embodiment of the present application, after obtaining the current defense strategy of the target large language model, the method further includes: establishing a reward preference data set based on the prompt injection detection effect data after the implementation of the defense strategy; establishing a reward model based on the reward preference data set and the reward function; and aligning and optimizing the strategy formulation agent based on the reward model.

[0098] It should be noted that in this embodiment, the current defense strategy of the target large language model is obtained by the strategy-making agent according to the preliminary detection results, the semantic understanding analysis report and the behavior pattern analysis results; in order to improve the accuracy of the defense strategy of the strategy-making agent and enhance the overall defense capability, the prompt injection detection effect data after the implementation of the strategy is collected, and a reward preference data set is established. For example, the preference data set is {y1>y2}, indicating that for problem x, the response strategy y1 is better than the strategy y2, forming a preference alignment data set D, and defining a suitable reward function to establish a reward model. Based on the obtained reward model, the DPO algorithm is used to align and optimize the strategy preference agent, and the loss function is as follows:

[0099]

[0100] Among them, π θ and π ref They represent the language model and the reference model respectively, σ represents the sigmoid function, and β is the weighted hyperparameter.

[0101] In one embodiment, the present application provides a prompt injection defense system, which specifically includes the following embodiments:

[0102] Current hint injection defense methods include Figure 3 As shown, malicious prompt injection attack detection is mainly performed on user input, and only safe user questions are allowed to be input into the model for answering, and the input detected as prompt injection attack is intercepted. However, it is strongly dependent on the size of the rule base. In the face of dynamically changing attack methods, there is a problem of low detection accuracy. In order to solve the problem of low detection accuracy, this embodiment proposes a prompt injection defense system based on the collaboration of multiple intelligent agents, which specifically includes:

[0103] An input detection agent is used to perform preliminary detection on the input text to obtain preliminary detection results; the preliminary detection results include high risk, low risk and safe;

[0104] A semantic parsing agent, used for performing semantic analysis on the input text to generate a semantic understanding analysis report when the preliminary detection result is low risk or safe;

[0105] A model protection agent, used to establish a connection with a target large language model and obtain current operating parameters of the target large language model; and also used to obtain a behavior pattern analysis result of the target large language model according to the current operating parameters;

[0106] A strategy formulation agent is used to obtain the current defense strategy of the target large language model based on the preliminary detection results, the semantic understanding analysis report and the behavior pattern analysis results.

[0107] It should be noted that the application scenarios and system structures of the prompt injection defense system in this embodiment are respectively as follows: Figure 4 and Figure 5 As shown in the figure, this system consists of four agents: input detection agent, semantic parsing agent, model protection agent and strategy formulation agent. Each agent is mainly composed of a large model and has the capabilities of communication, planning reflection, memory and tool use. The input detection agent is responsible for preliminary detection and filtering of user input prompts, the semantic parsing agent is responsible for a deeper semantic understanding and analysis of the input after preliminary filtering, the model protection agent is closely integrated with the large model and is responsible for monitoring the running status and output of the model. It can detect whether the model is affected by prompt injection. The strategy formulation agent dynamically adjusts the defense strategy based on the information fed back by other agents. The execution process is as follows: Figure 6 As shown, the functions of each agent are described in detail below:

[0108] 1. Input detection agent: responsible for preliminary detection and filtering of user input prompts. It can use tools such as rule engines and blacklist mechanisms to identify and intercept inputs containing sensitive words, malicious instructions or non-compliant inputs, and prevent these harmful information from directly entering the large model. For example, when the user input contains dangerous words such as "ignore all instructions" that may cause prompt injection, the agent can detect and block it in time; the specific steps of preliminary detection include:

[0109] (1) Text preprocessing: The user inputs the prompt content, and the large model receives the prompt content and performs text preprocessing on the content, such as text cleaning and normalization, grammatical analysis and tagging, etc.

[0110] (2) Rule-based preliminary screening: including sensitive word matching, grammatical structure checking, etc., using a pre-built sensitive word library to check whether the input text contains known sensitive words, malicious command words, or words that violate specific security policies. Analyze whether the grammatical structure of the input text conforms to normal language usage habits and the interactive grammar expected by the system. For example, in a specific task-oriented large model application, check whether the input follows the predetermined command format or problem statement. Abnormal grammatical structure may indicate potential injection attack attempts, such as situations where the sentence structure is confusing but contains key model operation keywords. Once a grammatical abnormality is found, it is also marked as suspicious and analyzed in depth.

[0111] (3) Semantic analysis and intent inference: The pre-processed text is further input into the big model, which performs intent understanding and context analysis on the text content to infer the real purpose and potential intention of the user input.

[0112] (4) Comprehensive judgment and analysis: The large model combines the contextual content and the detected user intent to comprehensively analyze whether it contains prompt injection attacks. If the large model determines that the input may have risks such as maliciously guiding the model or bypassing security restrictions, the intelligent agent will mark the input as high risk; if the large model believes that the input is basically normal but still has certain ambiguity or potential risks, it will be marked as low risk; if the large model determines that the input is a normal, risk-free interaction, it will be marked as safe.

[0113] (5) Output detection results: The agent outputs the final input detection results (safe, high risk or low risk, etc.) to subsequent agents so that they can take corresponding actions based on the detection results.

[0114] 2. Semantic parsing agent: It can conduct a deeper semantic understanding and analysis of the input after preliminary filtering. It can further parse the input intention, context and other information of the text determined by the input detection agent as low-risk or safe, and determine whether there is a potential risk of prompt injection. For example, it can analyze whether the input is trying to bypass the normal interaction logic, or whether it is inconsistent with the expected usage scenario of the model; specific functions include:

[0115] (1) Intent recognition and classification: The large model recognizes and classifies the intent of the text. It can determine whether the user is asking for information, expressing a need, issuing instructions, or engaging in other types of communication.

[0116] (2) Contextual understanding and association: Considering the contextual information of the current interaction, if there is a history of multiple rounds of dialogue, the large model will integrate this contextual information to understand the complete semantics of the current input text.

[0117] (3) Logical rationality check: Analyze the logical rationality of the input text to determine whether there are semantic contradictions, unreasonable inferences, or situations that are contrary to normal interaction logic. Text that is seriously inconsistent with the normal interaction context may contain potential malicious injections or abnormal instructions.

[0118] (4) Injection risk detection: Based on semantic understanding, detect whether the input text indicates injection risk. Check whether there are any obscure statements or instructions that attempt to manipulate the behavior of the big model, such as bypassing security restrictions through special semantic constructions, inducing the big model to output unauthorized information, etc.

[0119] (5) Information analysis and information transmission: Generate a semantic understanding analysis report on the input text, including intent classification results, contextual information, logical rationality assessment, and potential risk assessment. The semantic analysis report is output to the subsequent intelligent agent. If potential risks are found, a risk warning signal will be issued at the same time so that the subsequent modules can take corresponding measures, such as strengthening the monitoring of model operation, adjusting security policies, or refusing to perform suspicious operations.

[0120] 3. Model protection agent: The input detection agent and semantic parsing agent mainly detect information from the text input side. The model protection agent is closely integrated with the target large language model LLM and is responsible for monitoring the running status and output of the model. It can detect whether the model is affected by prompt injection, such as whether the output of the model is inconsistent with expectations under normal circumstances, and whether it contains inappropriate content or violates security policies. Once an abnormality is found, take appropriate measures in a timely manner, such as stopping the model's response and issuing an alarm, etc. Specifically, it includes:

[0121] (1) Model connection and initialization: Establish a connection channel with the target large language model to ensure that the operating status information and input and output data of the large model can be obtained in real time.

[0122] (2) Operation status detection: Continuously monitor the operation parameters, input data flow, output result information, etc. of the large model, such as CPU usage, response time, input data size, source, type, output data stability and consistency, etc.

[0123] (3) Behavior pattern analysis: By accumulating the long-term operation data of the large model, a behavior pattern model of its normal operation is established and used as a tool for the intelligent agent. When it is found that the current operation data deviates significantly from the normal behavior pattern, it is determined to be a possible abnormal situation. For example, when the large model processes a certain type of specific input, the resource consumption suddenly increases and the output results are unstable, which is different from the past.

[0124] (4) Risk assessment and information transmission: Risk assessment is performed based on the collected large model information and the results of behavior pattern analysis to determine the severity of the anomaly and the possible impact, and the collected information is transmitted to the strategy-making agent.

[0125] (5) Response and recovery: Receive decisions from the policy-making agent, such as pausing the operation of the large model, starting data backup, etc. After the abnormal situation is handled, assist the large model in recovery operations. For example, after solving the problem that caused the model performance to deteriorate, the protection agent resumes the operation of the large model and continues to monitor whether its restored operating status is normal to ensure that the model can be put back into use stably.

[0126] 4. Strategy formulation agent: dynamically adjusts the defense strategy based on the information fed back by other agents. For example, when a new hint injection attack method (input from the environment) is discovered, the agent can promptly update the filtering rules of the input detection agent, adjust the judgment method of the semantic parsing agent, or optimize the monitoring mechanism of the model protection agent to enhance the overall defense capability; specifically, it includes:

[0127] (1) Information collection and integration: Collect information related to the large model’s operating environment, performance, security status, and external threat intelligence from other intelligent agents and multiple data sources, and integrate the data.

[0128] (2) Situation Analysis: The strategy formulation agent conducts an overall situation analysis based on the obtained agents, determines the overall situation of the current input text and the large model, identifies the patterns, relationships and trends hidden in the data and text, and generates a detailed report on the current operation status of the large model based on the analysis results, including the evaluation of the model performance status, the identification and classification of security risks, and the prediction of future development trends.

[0129] (3) Strategy generation and optimization: Based on the generated situation analysis report, the strategy formulation agent generates corresponding response strategies, such as information interception, access control, model performance optimization, and emergency response. Furthermore, after generating the preliminary strategy, the agent can simulate, evaluate, and optimize the strategy, and finally output the optimal strategy in the current state and feed it back to the defense agent for execution.

[0130] (4) Agent optimization: Collect the prompt injection detection effect data after the implementation of the strategy, establish the reward preference data set {y1>y2}, which means that for problem x, the response strategy y1 is better than strategy y2, and form the preference alignment data set D. Define a suitable reward function and establish a reward model. Based on the obtained reward model, the DPO algorithm is used to align and optimize the strategy preference agent. The loss function is as follows:

[0131]

[0132] Among them, π θ and π ref They represent the language model and the reference model respectively, σ represents the sigmoid function, and β is the weighted hyperparameter.

[0133] The environment in this embodiment is the external conditions and scenarios of the entire prompt injection defense system, which is mainly used to provide feedback on the system's behavior, interaction, and task completion to prompt the system to adjust and optimize. Here, the environment can be human feedback, social environment, laws and regulations, technical environment, etc., which has the characteristics of dynamic change. The intelligent agent can obtain relevant information in real time by interacting with the environment, which promotes the evolution of the intelligent agent.

[0134] This application proposes a dynamic defense method for large model prompt injection based on multi-agent collaboration, which can not only use the collaborative ability of multiple agents to accurately detect prompt injection attacks on large models and effectively prevent large models from being attacked maliciously; this application also proposes a mechanism for multiple agents to collaborate to improve decision-making capabilities, further improving the accuracy of large models in identifying prompt attacks and coping strategies when facing malicious attacks. When facing complex attack scenarios, multiple agents can jointly formulate defense strategies, give full play to their respective advantages, achieve more effective defense, and improve the robustness of the overall defense system.

[0135] Currently, big model technology is applied in various fields of the industry, bringing new opportunities for business empowerment in different fields; however, prompt injection attacks have brought new challenges to the security issues of various industrial applications, such as financial security in the financial industry, patient information in healthcare, data leakage in the network field, etc. Therefore, the method described in this application can be applied to all big model applications in the current industry to effectively protect the security of big model applications; in addition, the method described in this application can be used to defend against big model prompt injection attacks, so its possible value is not limited to the following:

[0136] (1) Protecting data security: In many application scenarios, large language models may be exposed to sensitive information, so this patented method can effectively prevent data leakage.

[0137] (2) System security and stability: A successful prompt injection attack may allow the attacker to gain control over the large language model system and then perform malicious operations. Therefore, the patented method can effectively maintain the overall security of the system.

[0138] (3) Protecting corporate reputation and user rights: Warning injection attacks may damage corporate reputation or obtain user personal information and infringe upon user rights. Effective defense can improve user experience and promote positive interaction between users and companies.

[0139] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0140] It should be noted that Figure 7 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 7 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003, such as executing the method described in the above embodiment. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through the bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0142] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0143] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0144] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0145] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0147] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A prompt injection defense method, characterized in that: The method comprises: Performing a preliminary test on the input text to obtain a preliminary test result; the preliminary test result includes high risk, low risk and safe; When the preliminary detection result is low risk or safe, performing semantic analysis on the input text to generate a semantic understanding analysis report; Establishing a connection with a target large language model and obtaining current operating parameters of the target large language model; According to the current operating parameters, obtaining a behavior pattern analysis result of the target large language model; According to the preliminary detection result, the semantic understanding analysis report and the behavior pattern analysis result, the current defense strategy of the target large language model is obtained.

2. The method according to claim 1, characterized in that Perform a preliminary test on the input text and obtain preliminary test results, including: Screening the input text based on a pre-built sensitive word library to obtain a sensitive word screening result; Performing grammatical structure analysis on the input text to obtain a grammatical structure analysis result; Understand the intention of the input text to obtain the user's real purpose; According to the sensitive word screening result, the grammatical structure analysis result and the user's actual purpose, the input text is determined to be high risk, low risk or safe.

3. The method according to claim 1, characterized in that Performing semantic analysis on the input text to generate a semantic understanding analysis report, including: Based on the history of multiple rounds of input, get the intent type of the current input text; Perform logical judgment on the current input text to obtain a logical rationality evaluation result; Perform risk assessment on the input text based on semantic understanding to obtain a risk assessment result; wherein the semantic understanding analysis report includes the intent type, the logical rationality assessment and the risk assessment result.

4. The method according to claim 3, characterized in that Based on the history of multiple rounds of input, get the intent type of the current input text, including: Convert the current input text and the previous multiple rounds of input text into a text sequence based on word vectors to obtain a sentence sequence corresponding to each input text; Semantically encode each sentence sequence to obtain the semantic representation corresponding to each input text; Combining all semantic representations into a text sequence, semantically encoding the text sequence based on an attention mechanism, and obtaining semantic representations of multiple rounds of input; According to the semantic representation corresponding to the current input text and the semantic representation of multiple rounds of input, the intent type of the current input text is obtained.

5. The method according to claim 4, characterized in that Semantically encode each sentence sequence to obtain the semantic representation corresponding to each input text, including: Get the forward state representation and reverse state representation corresponding to each sentence sequence; Obtaining a target state representation corresponding to each sentence sequence according to the forward state representation and the reverse state representation; Obtaining an intermediate state representation according to the target state representation and the parameter matrix; Obtaining an attention weight according to the similarity between the intermediate state representation and the background context vector; Based on the weighted sum of the target state table and the attention weight, a semantic representation corresponding to each input text is obtained.

6. The method according to claim 4, characterized in that According to the semantic representation of the current input text and the semantic representation of multiple rounds of input, the intent type of the current input text is obtained, including: According to the semantic representation corresponding to the current input text and the semantic representation of multiple rounds of input, the correlation coefficient between the semantics of the current input text and the semantics of the context is calculated; Based on the correlation coefficient, the probability distribution of the current input text and the context in each intent category is calculated; According to the probability distribution, the intent type of the current input text is obtained.

7. The method according to claim 1, characterized in that After obtaining the current defense strategy of the target large language model, the method further includes: A reward preference dataset was established based on the prompt injection detection effect data after the implementation of the defense strategy; Establishing a reward model according to the reward preference data set and the reward function; Alignment optimization is performed on the policy making agent according to the reward model.

8. A prompt injection defense system, characterized in that: The system comprises: An input detection agent is used to perform preliminary detection on the input text to obtain preliminary detection results; the preliminary detection results include high risk, low risk and safe; A semantic parsing agent, used for performing semantic analysis on the input text to generate a semantic understanding analysis report when the preliminary detection result is low risk or safe; A model protection agent, used to establish a connection with a target large language model and obtain current operating parameters of the target large language model; and also used to obtain a behavior pattern analysis result of the target large language model according to the current operating parameters; A strategy formulation agent is used to obtain the current defense strategy of the target large language model based on the preliminary detection results, the semantic understanding analysis report and the behavior pattern analysis results.

9. An electronic device, characterized in that: include: processor; as well as A memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to enable the electronic device to implement the prompt injection defense method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the prompt injection defense method according to any one of claims 1 to 7.

Images