Lie cycle attack detection method and electronic device

Abstract

The application discloses a lie cycle attack detection method and electronic equipment, and relates to the technical field of network security. In related technologies, manual identification may cause mis-approval of dangerous operations, and may cause a high risk of misjudging malicious operations as safe operations. The application obtains response information generated by a pre-trained language model based on a to-be-processed task. The response information includes an explanation text and a to-be-executed instruction. The explanation text is an operation description text generated by the pre-trained language model and used for showing a user in a human-in-the-loop confirmation stage. The to-be-executed instruction is a to-be-executed instruction generated by the pre-trained language model. An original input context on which the response information is based is obtained. Whether the to-be-executed instruction has a lie cycle attack is determined according to the explanation text, the original input context and the to-be-executed instruction. The lie cycle attack can be automatically detected, mis-approval of dangerous operations can be effectively avoided, the accuracy of detection is improved, and the risk of misjudging malicious operations as safe operations is reduced.

Description

Technical Field

[0001] This application relates to the field of cybersecurity technology, and in particular to a method and electronic device for detecting a lie loop attack. Background Technology

[0002] With the widespread deployment of pre-trained language models (LLMs) across various fields, the Human-in-the-Loop (HITL) security measures relied upon for performing sensitive operations (such as running system commands, modifying files, and accessing network resources) have been listed by the Open Worldwide Application Security Project (OWASP) as a recommended mitigation measure against vulnerabilities such as hint injection. However, current Lies-in-the-Loop (LITL) attacks demonstrate that attackers can indirectly inject hints by polluting external data sources of the LLM, inducing the model to disguise malicious payloads as operation summaries with lengthy and harmless descriptions. Existing HITL measures lack the ability to verify the semantic consistency between the operation summary and the original context, relying entirely on the user's visual identification, which can easily lead to users unknowingly approving dangerous operations. Therefore, how to detect LITL attacks and intercept fraudulent operation requests at the HITL interaction stage has become a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0003] This application provides a method and electronic device for detecting lie loop attacks, which at least solves the problem in related technologies that rely on users to identify lie loop attacks, leading to users unknowingly approving dangerous operations and misjudging malicious operations as high-risk safe operations.

[0004] This application provides a method for detecting a lie loop attack, including: Obtain response information generated by the pre-trained language model based on the task to be processed; wherein, the response information includes explanatory text and proposed execution instructions, the explanatory text is the operation description text generated by the pre-trained language model and used to be displayed to the user during the human-loop confirmation stage, and the proposed execution instructions are the instructions to be executed generated by the pre-trained language model; The original input context on which the response information is based; Based on the explanatory text, the original input context, and the intended execution instruction, determine whether the intended execution instruction is subject to a lie loop attack.

[0005] This application also provides a device for detecting a lie loop attack, including: The first acquisition module is used to acquire response information generated by the pre-trained language model based on the task to be processed; wherein, the response information includes explanatory text and proposed execution instructions, the explanatory text is an operation description text generated by the pre-trained language model and used to be displayed to the user during the human-loop confirmation stage, and the proposed execution instructions are the instructions to be executed generated by the pre-trained language model; The second acquisition module is used to acquire the original input context on which the response information is based; The determination module is used to determine whether the instruction to be executed is subject to a lie loop attack based on the interpretation text, the original input context, and the instruction to be executed.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described lie loop attack detection methods when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described methods for detecting loop attack lies.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described methods for detecting loop attack lies.

[0009] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: This application provides a method for detecting a lie loop attack, comprising: acquiring response information generated by a pre-trained language model based on a task to be processed; wherein the response information includes explanatory text and a proposed execution instruction, the explanatory text being an operation description text generated by the pre-trained language model for displaying to the user during the human-loop confirmation stage, and the proposed execution instruction being a proposed execution instruction generated by the pre-trained language model; acquiring the original input context on which the response information is based; and determining whether the proposed execution instruction contains a lie loop attack based on the explanatory text, the original input context, and the proposed execution instruction.

[0010] Therefore, this application achieves automatic detection of lie loop attacks by acquiring response information generated by a pre-trained language model, which includes explanatory text and the instruction to be executed, as well as the original input context on which the response information is based. Based on the explanatory text, the original input context, and the instruction to be executed, it determines whether the instruction to be executed is a lie loop attack. This application eliminates the need for users to manually compare the consistency between the explanatory text and the original context, thus effectively preventing users from unknowingly approving dangerous operations. This intercepts fraudulent operation requests in the HITL interaction stage, improves the accuracy of detection, and reduces the risk of misjudging malicious operations as safe operations.

[0011] In addition, the present invention also provides an electronic device with corresponding advantages. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a type of lie loop attack in related technologies; Figure 2 A flowchart illustrating a method for detecting a lie loop attack, as provided in an embodiment of this application; Figure 3 A flowchart for determining whether a lie loop attack exists in an instruction to be executed, as provided in this application embodiment; Figure 4 Another flowchart for determining whether a lie loop attack exists in an instruction to be executed, provided in an embodiment of this application; Figure 5 This is a structural diagram of a lie loop attack detection device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] It's worth noting that the application of pre-trained language model agents across various industries is rapidly expanding, with an increasing number of organizations deploying such agents in their business workflows. Currently, AI (Artificial Intelligence) agents widely use HITL as a last line of defense. However, the Lie Loop LITL attack utilizes indirect prompt injection techniques to contaminate remote resources (such as GitHub Issues and web page content), inducing the LLM to disguise malicious payloads (such as reverse shells and data theft commands) as harmless operations (such as code repair or dependency installation) when generating operation requests. Unlike previous penetration methods targeting the model itself, LITL compromises the human-loop interaction confirmation stage. Based on its constructed prompt injection, it forges and tampers with dialogue content, turning the original security gate into a breach point for Remote Code Execution (RCE). This attack method has a high success rate and has already penetrated multiple mainstream intelligent agent platforms. A single unintentional confirmation operation by the victim is enough to trigger a catastrophic chain reaction, including server compromise, sensitive information leakage, and the implantation of hidden backdoors.

[0018] Attackers can inject malicious prompts into the runtime environment of an AI agent by exploiting the context in which the user receives the prompts, through external channels such as code repositories and web pages. Based on these tampered instructions, the AI ​​agent generates a seemingly harmless human-machine collaborative verification dialog box. For example, when using Claudecode to analyze code / github-issue commands, it might reveal issues with GitHub entries set by untrusted users.

[0019] When a program encounters a problem, such as code containing security vulnerabilities, Claude Code reports a permission request to write a safe code snippet to be inserted into the application to resolve the issue. This is a form of human-assisted defense. Attackers embed seemingly harmless text content into malicious payloads, hiding dangerous commands outside the visible area of ​​the terminal window. When users browse these seemingly safe instructions, they unknowingly approve the execution of malicious code on their devices. If the user selects "yes" in the confirmation process, the attacker-injected malicious commands are successfully executed, similar to user deception attacks (such as phishing). Attackers can both obfuscate the malicious behavior itself and ensure it is masked by more harmless descriptions, thus deceiving developers into executing malicious payloads. The process of a lie loop attack can be similarly described. Figure 1In other words, when a user starts a conversation, the LLM collects contextual information. Attackers can interact with remote resources to maliciously pollute them, inserting malicious commands and long texts. The LLM decides to perform an operation and requests permission. At this point, the malicious payload is embedded in a seemingly harmless long text, effectively hiding its malicious intent. The user chooses to approve or deny. If the user approves, the malicious commands injected by the attacker will be successfully executed.

[0020] The core of HITL is to combine the automated processing capabilities of artificial intelligence with human expertise, ethical judgment, and creative thinking. It does not replace human input, but rather ensures the quality of critical decisions in highly complex, ethically demanding, and high-risk scenarios through human participation. This model supports the deployment of AI in sensitive areas and achieves continuous system optimization through feedback loops.

[0021] The core components of HITL deployment include human supervision, intervention and correction, feedback-driven learning, decision enhancement, human-machine collaboration, and upgrade strategies. Human supervision involves monitoring the agent's performance and output through log review and real-time dashboards to ensure compliance and prevent undesirable results. Intervention and correction involve the agent requesting human intervention to correct errors and supplement data when encountering errors or ambiguous scenarios, while also providing a basis for subsequent improvements. Feedback-driven learning involves collecting human feedback (such as preferences and suggestions) to optimize the AI ​​model (e.g., human feedback reinforcement learning), guiding the agent's learning trajectory. Decision enhancement involves the agent providing analysis and suggestions, with humans making the final decision; AI insights improve human decision-making efficiency rather than complete automation. Human-machine collaboration involves both humans and the agent leveraging their respective strengths: the agent handles routine, repetitive data analysis tasks, while humans handle creative problems and complex negotiations. Upgrade strategies involve the agent escalating tasks beyond its capabilities to human operators according to predetermined protocols when encountering tasks beyond its capabilities, preventing errors from occurring at the source.

[0022] To address the technical problems existing in related technologies, embodiments of this application provide a method for detecting a lie loop attack. Please refer to... Figure 2 The method for detecting the lie loop attack includes the following steps S110 to S130.

[0023] S110: Obtain response information generated by the pre-trained language model based on the task to be processed; wherein, the response information includes explanatory text and intended execution instructions, the explanatory text is the operation description text generated by the pre-trained language model and used to be displayed to the user during the human-loop confirmation stage, and the intended execution instructions are the instructions to be executed generated by the pre-trained language model.

[0024] It should be noted that when a user submits a task to be processed to the LLM, the LLM generates response information for that task. This response information includes at least explanatory text and a proposed execution instruction. In this embodiment, the explanatory text (i.e., the operation summary described in the background section) is an operation description text generated by the LLM and displayed to the user during the HITL confirmation phase. Its purpose is to inform the user what operation will be performed and the reason for performing it, so that the user can make a decision to approve or reject it. The proposed execution instruction is a pending execution instruction generated by the LLM, which may take the form of system commands, code snippets (e.g., calling an Application Programming Interface (API), executing a function), etc. This proposed execution instruction is the operational payload that the LLM intends to actually execute at the underlying level. Under normal, unattacked scenarios, the explanatory text and the proposed execution instruction should be semantically consistent, that is, the explanatory text is an accurate description of the proposed execution instruction. However, in LITL attack scenarios, attackers inject malicious prompts into the LLM by contaminating external data sources. This causes a significant semantic deviation between the interpreted text generated by the LLM and the intended execution command. In other words, the interpreted text disguises malicious payloads (such as reverse shells, data theft commands, or file deletion commands) in the intended execution command as harmless operations (such as "fixing code vulnerabilities," installing dependency packages, or writing secure code snippets). Therefore, obtaining the interpreted text and intended execution command from the LLM's response information is the data foundation for detecting LITL attacks.

[0025] In practical applications, the security detection module in the system can obtain the interpreted text and the intended execution instructions from the LLM's output response after the LLM generates the response information but before displaying the interpreted text in the response information to the user, and before sending the intended execution instructions in the response information to the underlying execution engine. The specific method of acquisition can be achieved by intercepting the LLM's output interface calls or by reading the contents of the LLM's output buffer. The specific method used can be determined according to actual needs, and this application embodiment does not impose specific limitations on this.

[0026] S120: The original input context on which the response information is obtained.

[0027] It is understood that the original input context in this application embodiment refers to the input information on which the LLM generates the above-mentioned response information. This information may include the original request text submitted by the user to the LLM, as well as the context information read by the LLM from external data sources during the execution of the task to be processed. Since the implementation of a LITL attack involves the attacker injecting malicious prompts by polluting external data sources, the original input context may contain malicious instructions and deceptive descriptive text embedded by the attacker. The significance of obtaining the original input context is that subsequent steps can compare the semantic consistency between the explanation text and the original input context. If the explanation text deviates significantly from the original input context semantically, it indicates that the explanation text generated by the LLM may have been manipulated by the attacker, and there is suspicion of a LITL attack. In practical application scenarios, the security detection module can obtain the original input context through the LLM's context management interface, or it can obtain the original input context by recording the content of all external data sources read by the LLM during the processing of the task to be processed. The specific method used for obtaining the original input context can be determined according to actual needs, and this application embodiment does not specifically limit this.

[0028] S130: Based on the interpreted text, the original input context, and the instruction to be executed, determine whether the instruction to be executed is subject to a lie loop attack.

[0029] It should be noted that, in this embodiment, the presence of a LITL attack in a proposed instruction can be determined by comprehensively considering information from three aspects: the interpreted text, the original input context, and the instruction to be executed. In practical applications, on the one hand, the semantic consistency between the interpreted text and the original input context can be compared to determine whether the LLM is "falsely reporting" to the user. If there is a semantic deviation between the interpreted text and the original input context, it indicates that the LLM may have been manipulated by an attacker. On the other hand, the maliciousness of the proposed instruction itself can be analyzed to determine whether it is a malicious payload. In this embodiment, the results of both analyses can be combined for a comprehensive judgment, which can effectively identify LITL attacks and avoid false positives or missed detections caused by relying on only a single dimension. Since the detection process is automatically executed by the system, there is no need for the user to manually compare the consistency between the interpreted text and the original input context. Therefore, it can effectively prevent users from unknowingly approving dangerous operations, thereby intercepting fraudulent operation requests at the HITL interaction stage.

[0030] In this embodiment, by acquiring response information generated by a pre-trained language model, which includes explanatory text and the intended instruction to be executed, as well as the original input context on which the response information is based, and by comprehensively determining whether the intended instruction to be executed is a lie loop attack based on the explanatory text, the original input context, and the intended instruction to be executed, automatic detection of lie loop attacks is achieved. This eliminates the need for users to manually compare the consistency between the explanatory text and the original input context, thus effectively preventing users from unknowingly approving dangerous operations. This intercepts fraudulent operation requests in advance during the HITL interaction process, improving the accuracy of detection and reducing the risk of misjudging malicious operations as safe operations.

[0031] In one implementation, please refer to Figure 3 The process of determining whether the intended instruction to be executed is a lie loop attack, based on the original input context and the intended instruction in S130 above, may include: S210: Determine the semantic deviation based on the explanatory text and the original input context; the semantic deviation is used to characterize the degree of semantic inconsistency between the explanatory text and the original input context.

[0032] It should be noted that the semantic deviation degree in this application embodiment is a quantitative indicator that can be used to measure the degree of semantic inconsistency between the explanatory text and the original input context. In a LITL attack, the attacker induces the LLM to fabricate a seemingly reasonable operational reason (i.e., the explanatory text) to cover up the real malicious operation. Therefore, there will be a semantic deviation between the explanatory text and the original input context. In this application embodiment, the semantic deviation degree is used to represent the degree of deviation, which can objectively determine whether the explanatory text has been manipulated by the attacker. The higher the value of the semantic deviation degree, the greater the semantic difference between the explanatory text and the original input context, and the greater the suspicion of a LITL attack.

[0033] S220: Determine the malicious confidence level based on the instruction to be executed; the malicious confidence level is used to characterize the probability that the instruction to be executed is a malicious instruction.

[0034] It should be noted that the malicious confidence level in this application embodiment can be used to assess the probability that the instruction to be executed is malicious. In LITL attacks, the malicious payload injected by the attacker (such as reverse shell, data theft commands, file deletion commands, etc.) usually has obvious malicious characteristics. By performing maliciousness analysis on the instruction to be executed, it can be determined whether it is a malicious payload. The higher the malicious confidence level, the greater the probability that the instruction to be executed is malicious.

[0035] S230: Determine the information hiding density based on the instruction to be executed and the interpretation text; the information hiding density is used to characterize the degree to which the instruction to be executed is masked by the interpretation text.

[0036] It should be noted that the information hiding density in this application embodiment can be used to characterize the degree to which the intended execution instruction is obscured by the interpreted text. In LITL attacks, attackers typically embed a large amount of seemingly harmless, lengthy text content (i.e., interpreted text) into the malicious attack payload, thereby hiding the dangerous command outside the visible area. Understandably, the attacker induces the LLM to generate a very lengthy interpreted text, while the actual intended execution instruction is very short, making it difficult for the user to notice the hidden malicious instruction when faced with a large amount of text. Therefore, the higher the information hiding density value, the longer the interpreted text and the shorter the intended execution instruction, indicating a greater risk that the intended execution instruction is obscured by the lengthy interpreted text, and a stronger suspicion of a LITL attack.

[0037] S240: Determine the comprehensive risk score based on semantic deviation, malicious confidence, and information hiding density.

[0038] It should be noted that in this embodiment, semantic deviation, malicious confidence, and information hiding density reflect the risk of LITL attacks from three different dimensions. Semantic deviation reflects the degree of semantic inconsistency between the interpreted text and the original context, malicious confidence reflects whether the instruction to be executed is malicious, and information hiding density reflects whether the instruction to be executed is obscured by the lengthy interpreted text. In this embodiment, the indicators of these three dimensions are comprehensively judged, which can more comprehensively and accurately assess the overall risk level of LITL attacks.

[0039] S250: Based on the comprehensive risk score, determine whether the instruction to be executed is subject to a lie loop attack.

[0040] Understandably, after obtaining the comprehensive risk score, comparing it with a preset risk threshold can determine whether the intended execution instruction is susceptible to a LITL attack. In practical applications, a higher comprehensive risk score indicates a greater risk of a LITL attack. For example, the comprehensive risk score can be judged using a threshold range to determine whether the intended execution instruction is susceptible to a lie loop attack. The lower limit of the threshold range is the first threshold, and the upper limit is the second threshold. If the comprehensive risk score is less than the first threshold, the instruction is considered safe; if the comprehensive risk score is greater than or equal to the first threshold but less than the second threshold, it is considered suspicious; and if the comprehensive risk score is greater than or equal to the second threshold, it is considered malicious and a LITL attack is confirmed.

[0041] In one implementation, the process of determining the semantic deviation degree based on the interpreted text and the original input context in S210 above may include the following steps: (1) Map the explanatory text to the first vector and the original input context to the second vector.

[0042] It should be noted that, in order to calculate the semantic similarity between the explanatory text and the original input context, the two text segments can first be converted into mathematical vectors, i.e., numerical representations in high-dimensional space, a type of continuous vector that can be used for computer computation. In this embodiment, a pre-trained text embedding model can be used to map the explanatory text into a first vector V. req Mapping the original input context to a second vector V ctx .

[0043] In practical applications, the explanatory text R can be used. gen The input is fed into a pre-trained text embedding model, and the model's output vector is extracted as the first vector V. req ; Change the original input context C tx The input is fed into the same pre-trained text embedding model, and the output vector of that model is extracted as the second vector V. ctx Vector V req and V ctx All are continuous vectors in the real number field, and their number of dimensions depends on the output dimension of the text embedding model used. It should be noted that the specific type and output dimension of the text embedding model used can be selected and configured according to the actual application needs, and this application embodiment does not impose specific limitations on this.

[0044] (2) Calculate the cosine similarity between the first vector and the second vector.

[0045] It should be noted that after obtaining the first vector V req Second vector V ctx Then, the cosine similarity between the two vectors can be further calculated. Cosine similarity measures the degree of similarity in direction by calculating the cosine of the angle between the two vectors. Its value range is [0,1] (when all vector components are non-negative). The larger the value, the more consistent the directions of the two vectors are, that is, the more semantically similar they are. The formula for calculating cosine similarity in this embodiment is: ; Among them, V req V ctx Represents vector V req With vector V ctx dot product, Represents vector V req The length of the mold, Represents vector V ctx The modulus (Euclidean norm) of cosine similarity. In practical applications, the closer the cosine similarity value is to 1, the more semantically consistent the explanatory text is with the original input context; the closer the value is to 0, the greater the semantic difference between the two.

[0046] (3) Determine the semantic deviation based on cosine similarity. The semantic deviation is negatively correlated with cosine similarity.

[0047] It should be noted that the semantic deviation degree D is used in the embodiments of this application. sem The semantic deviation D represents the degree of semantic inconsistency between the explanatory text and the original input context. sem There is a negative correlation between cosine similarity and semantic deviation. The higher the cosine similarity (that is, the more semantically consistent the explanatory text is with the original input context), the lower the semantic deviation. The lower the cosine similarity (that is, the greater the semantic difference between the explanatory text and the original input context), the higher the semantic deviation.

[0048] The semantic deviation D in the embodiments of this application sem The calculation formula can be: .

[0049] Among them, D sem The range of D is [0,1]. sem =0 indicates that the interpreted text is semantically identical to the original input context (without deviation), and when D... sem A value of 1 indicates that the two are completely semantically inconsistent (maximum deviation). This is due to the cosine similarity... The range of D is [0,1], therefore D sem The range of D is also [0,1]. sem The larger the value, the greater the semantic deviation between the interpreted text and the original input context, and the greater the suspicion of a LITL attack.

[0050] This application embodiment maps the explanatory text and the original input context to a first vector and a second vector respectively, and calculates the cosine similarity between the two to determine the semantic deviation. This achieves automatic evaluation of the semantic consistency between the explanatory text and the original input context, providing an objective basis for the automatic detection of LITL attacks.

[0051] In one implementation, the process of determining the degree of malice based on the instruction to be executed in S220 above may include: (1) Input the instruction to be executed into the classifier that was trained based on the malicious sample, and obtain the first probability value of the instruction to be executed output by the classifier as a malicious instruction.

[0052] It should be noted that in this embodiment, a classifier trained on malicious samples can be used to predict the malice of the intended execution command. This classifier is a machine learning model whose input is the representation of the intended execution command (e.g., a command string, token sequence, etc.), and whose output is the probability value that the intended execution command belongs to the malicious category. During the training phase, the classifier needs to perform supervised learning using a large number of labeled malicious samples (including but not limited to known reverse shell commands, data theft commands, file deletion commands, etc.) and normal samples. The classifier can be implemented using various machine learning algorithms, such as random forest classification models, support vector machine classification models, deep neural network-based classification models, gradient boosting decision tree-based classification models, etc. This embodiment does not specifically limit the choice of machine learning algorithm in practical applications. During the training process, the classifier can be trained with malicious or non-malicious labels as outputs, and after training, the model parameters of the classifier are fixed for real-time prediction during the online inference phase.

[0053] In practical applications, when the instruction P to be executed is... exec After being input into the trained classifier, the classifier outputs the first probability value Pclf(P) of the intended execution instruction being a malicious instruction. exec The first probability value has a range of [0,1]. The closer the value is to 1, the greater the likelihood that the classifier considers the instruction to be malicious. It should be noted that the classifier in this embodiment can be implemented using various machine learning algorithms known in the art. The specific algorithm type and training method of the classifier can be selected and configured by those skilled in the art according to actual needs. This embodiment does not specifically limit this.

[0054] (2) Match the instruction to be executed with the preset blacklist database, and determine the second probability value based on the matching result.

[0055] It should be noted that a blacklist database can be pre-established in this application embodiment. This pre-established blacklist database can store known malicious instruction patterns, such as known malicious system commands (e.g., rm -rf / , chmod 777, etc.), known malicious code snippets (e.g., typical code patterns of reverse shells), and known malicious API call patterns. The instruction to be executed, P... exec The data is matched against entries in the blacklist database to obtain matching results, and a second probability value I(P) is determined based on the matching results. exec If the instruction to be executed matches any malicious instruction pattern in the blacklist, the second probability value is 1; if the instruction to be executed does not match any malicious instruction pattern in the blacklist, the second probability value is 0.

[0056] (3) Determine the degree of malice based on the first probability value and the second probability value.

[0057] It should be noted that the malicious confidence level C mal It is to set the first probability value Pclf(P) exec ) and the second probability value I(P exec The result obtained by weighted fusion of ∈BlackList has a value range of [0,1]. The larger the value, the higher the probability that the instruction to be executed is malicious. The formula for calculating the malicious confidence is: ,in, C mal represents the malicious confidence level, used to comprehensively assess the probability that the instruction to be executed is malicious. Its value range is [0,1], with a higher value indicating a higher likelihood of malicious intent. α represents a preset weighting coefficient, a real number greater than 0 and less than 1, used to balance the weighting of classifier predictions and blacklist matching results in the malicious confidence level calculation. Pclf(P exec ) indicates that instruction P will be executed. exec After being input into a classifier trained on malicious samples, the classifier outputs the first probability value of the malicious instruction to be executed, with a value range of [0,1]; I(P exec (∈BlackList) is an indicator function, which is executed when instruction P is to be executed. exec The value is 1 when the blacklist is matched, and 0 when the blacklist is not matched.

[0058] In this embodiment, by combining the generalization detection capability of the classifier with the precise matching capability of the blacklist, the malicious confidence level can both identify known malicious command patterns (through blacklist matching) and generalize to detect unknown malicious variants (through classifier prediction), thereby improving the coverage and accuracy of malicious command detection.

[0059] It should be noted that the value of the weight coefficient α in the embodiments of this application can be configured according to the actual application scenario. For example, when the system relies more on the generalization detection capability of the classifier, a higher α value can be set, and when the system relies more on the precise matching of the blacklist, a lower α value can be set. In one implementation, α can be 0.7, that is, the classifier prediction result accounts for 70% of the weight, and the blacklist matching result accounts for 30% of the weight. Those skilled in the art can adjust the value of α according to actual needs, and the embodiments of this application do not specifically limit this.

[0060] This application embodiment obtains a first probability value by inputting the instruction to be executed into a classifier pre-trained based on malicious samples, and obtains a second probability value by matching the instruction to be executed with a preset blacklist database. Then, the malicious confidence is determined by the weighted sum of the first and second probability values. This achieves dual verification of machine learning classifier generalization detection and blacklist precise matching, thereby improving the coverage and accuracy of malicious instruction identification.

[0061] In one implementation, the process of determining the information hiding density based on the instruction to be executed and the interpretation text in S230 above may include: (1) Obtain the first length of the instruction to be executed and the second length of the interpretation text.

[0062] It should be noted that the first length Len(P) of the instruction to be executed, Pexec, is obtained. exec ) and explanatory text R gen The second length Len(R) gen The length can be measured in characters or tokens, but in the same embodiment, both should use the same unit of measurement to ensure consistency in subsequent ratio calculations. In practical applications, to facilitate implementation and avoid the uncertainty introduced by the word segmenter, it is preferable to use the number of characters as the unit of length measurement. The length is obtained by directly counting the number of characters or tokens in the corresponding text string. The specific method can be determined according to actual needs, and will not be elaborated here.

[0063] (2) Calculate the ratio of the second length to the first length.

[0064] It should be noted that, in the embodiments of this application, the second length Len(R) of the explanatory text can be further calculated. gen ) and the first length of the instruction to be executed, Len(P exec The ratio of Len(R) gen ) / Len(P exec In normal operating scenarios, the length of the interpreted text is usually directly proportional to the length of the intended command; that is, the more complex and longer the intended command, the longer its corresponding interpreted text. However, in LITL attack scenarios, attackers induce the LLM to generate extremely long and seemingly reasonable interpreted text to mask a very short malicious command (for example, a reverse shell command may have only one line of code, but its interpreted text can be hundreds of words long), resulting in an abnormally large ratio. By calculating this ratio, the unique obfuscation pattern of LITL attacks—based on lengthy text masking short malicious commands—can be detected.

[0065] (3) Calculate the normalized information entropy of the explanatory text.

[0066] It should be noted that, in order to further measure the information richness of the explanatory text, the embodiments of this application calculate the normalized information entropy of the explanatory text. Information entropy is an indicator in information theory that measures the amount of information or uncertainty. The lower the entropy value, the higher the text repetition, the lower the information content, and the more hollow the content. In LITL attacks, the verbose explanatory text generated by the attacker usually contains seemingly reasonable and information-rich text, which is used to disguise normal operation descriptions to cover up the actual malicious instructions, and its information entropy value is high.

[0067] The normalized information entropy in the embodiments of this application can be expressed as information entropy Entropynorm(R). gen ) indicates that, among which, the normalized information entropy Entropynorm(R) gen The range of ) is [0,1]. The higher the value, the more reasonable and information-rich the explanatory text seems, and the greater the possibility that the information is hidden.

[0068] (4) Determine the information hiding density based on the ratio and normalized information entropy.

[0069] It should be noted that the information hiding density H den It is based on the ratio of the second length to the first length. With normalized information entropy Entropynorm(R) gen The formula for calculating the information hiding density, as determined comprehensively, is as follows: ; where H den Len(P) represents the information hiding density; a higher value indicates a greater risk that the intended instruction will be obscured by a lengthy interpreted text. exec Len(R) represents the first length of the instruction to be executed, measured in characters or tokens; gen ) indicates the second length of the interpreted text, measured in characters or tokens; Entropynorm(R gen The information hidden density H represents the normalized information entropy of the explanatory text, obtained by dividing the information entropy of the explanatory text by the maximum possible information entropy of the explanatory text. den and ratio There is a positive correlation, meaning that the larger the ratio (i.e., the longer the interpreted text is than the intended instruction), the higher the information hiding density; the information hiding density H den With normalized information entropy Entropynorm(R) gen This is also positively correlated with H, meaning that the higher the normalized information entropy (i.e., the more reasonable and information-rich the explanatory text appears), the higher the information hiding density. Therefore, H den The higher the value of Len(R), the greater the risk that the intended instruction will be obscured by a verbose interpreted text, and the greater the suspicion of a LITL attack. In practical applications, when Len(R) = 1, the value of Len(R) = 1. gen ) is much larger than Len(P)exec Furthermore, when the normalized information entropy of the explanatory text is high, H den A value approaching a large value indicates a high risk of information hiding.

[0070] In this embodiment, by obtaining the first length of the instruction to be executed and the second length of the explanatory text, calculating the ratio of the second length to the first length, and calculating the normalized information entropy of the explanatory text, and then determining the information hiding density based on the ratio and the normalized information entropy, it is beneficial to effectively capture the behavior pattern of attackers using lengthy explanatory text to cover up malicious instructions.

[0071] In one implementation, the process of determining the comprehensive risk score based on semantic deviation, malicious confidence, and information hiding density in S240 above may include: (1) Using semantic deviation as input, calculate the first risk component corresponding to semantic deviation through the S-type activation function.

[0072] It should be noted that, in order to more sensitively capture the fraud signals reflected by subtle changes in semantic deviation in the embodiments of this application, the semantic deviation D is... sem As input, it is processed by the sigmoid activation function to obtain the first risk component σ(D). sem The sigmoid activation function is a non-linear function widely used in machine learning and mathematics. Its characteristic is that when the input value exceeds a certain critical point, the output value changes drastically, thus non-linearly amplifying the change in the input value. The expression for the sigmoid activation function used in this embodiment can be: ; Where x represents the input value (here, semantic deviation D) sem Let e ​​represent the base of the natural logarithm, and σ(x) represent the output value of the sigmoid activation function. The range of this sigmoid activation function is (0,1). When the input value x is near 0.5, the function value transitions sharply from near 0 to near 1. Understandably, when x < 0.5, σ(x) approaches 0; when x > 0.5, σ(x) approaches 1; and when x = 0.5, σ(x) = 0.5. This is achieved by using semantic deviation D... sem Substituting x into the above sigmoid activation function, the first risk component σ(D) is calculated. sem When semantic deviation D sem When the critical point exceeds 0.5, σ(D) sem The signal intensity increases exponentially, enabling the keen detection of minute semantic deviations and effectively avoiding missed detections.

[0073] (2) Malicious confidence is used as the second risk component.

[0074] It should be noted that the second risk component in this application embodiment is the malicious confidence level C. mal The inherent malice confidence level C mal The value range is [0,1], and the larger the value, the higher the probability that the instruction to be executed is a malicious instruction. In this embodiment, the malicious confidence level is directly used as the second risk component in the subsequent weighted summation, without the need for additional conversion processing.

[0075] (3) Information hiding density is used as the third risk component.

[0076] It should be noted that the third risk component in this embodiment is H. den That is, the information hiding density, because the information hiding density H den A higher H value indicates a greater risk that the intended instructions will be obscured by a verbose interpreted text. den A higher value indicates a greater risk of information hiding; that is, the risk component is positively correlated with the degree of risk. Furthermore, the third risk component has a value range of [0,1], with a larger value indicating a higher risk of information hiding.

[0077] (4) Using the preset weights corresponding to the first risk component, the second risk component and the third risk component respectively, calculate the weighted sum of the first risk component, the second risk component and the third risk component to obtain the comprehensive risk score.

[0078] It should be noted that the comprehensive risk score S in the embodiments of this application... risk It is the first risk component σ(D) sem ), second risk component C mal and the third risk component H den The weighted sum. The calculation formula for the comprehensive risk score in this embodiment is: ; Among them, S risk This represents the overall risk score; a higher value indicates a higher risk of a LITL attack on the instruction to be executed. This represents the first weighting coefficient corresponding to the first risk component. This represents the second weighting coefficient corresponding to the second risk component. This represents the third weighting coefficient corresponding to the third risk component. , , All three are real numbers greater than 0 and less than 1, and their sum equals 1 (i.e., ...). + + =1), σ(Dsem) represents the first risk component calculated using the sigmoid activation function. Weighting coefficients , , The value reflects the relative importance of the three dimensions—semantic deviation, malicious confidence, and information hiding density—in the comprehensive risk score.

[0079] In one embodiment of this application, considering that semantic deviation is a core feature of LITL attacks, it can be... Setting it to a higher value (e.g., 0.5) will Set it to 0.3, The value is set to 0.2. In practical applications, the weighting coefficient can be configured and adjusted by those skilled in the art according to actual needs. This application does not limit the specific value of the weighting coefficient.

[0080] It is understood that in this embodiment of the application, the semantic deviation is nonlinearly amplified by the S-shaped activation function and used as the first risk component, the malicious confidence is used as the second risk component, and the information hiding density is used as the third risk component. The weighted sum of the three components is calculated using the corresponding preset weights to obtain the comprehensive risk score. This achieves a nonlinear combination of the three-dimensional indicators, which makes the risk score rise sharply when the semantic deviation exceeds the critical point, thereby improving the detection sensitivity of small fraud signals.

[0081] In one implementation, the process of determining whether the instruction to be executed is subject to a lie loop attack based on a comprehensive risk score in S250 may include: When the overall risk score is less than the first threshold, the instruction to be executed is determined to be a safe instruction; When the comprehensive risk score is greater than or equal to the first threshold and less than the second threshold, the instruction to be executed is determined to be a suspicious instruction. When the comprehensive risk score is greater than or equal to the second threshold, the instruction to be executed is determined to be a malicious instruction, and it is determined that the instruction to be executed contains a lie loop attack. The first threshold is less than the second threshold.

[0082] It should be noted that, in this embodiment of the application, a first threshold θ can be preset. low Second threshold θ high , where θ low <θ high , where the first threshold θ low Second threshold θ high All are real numbers between 0 and 1. When the comprehensive risk score S... risk Less than the first threshold θ lowWhen the overall risk score is low, it indicates that the semantic deviation is low, the malicious confidence level is low, and the information hiding density is high (i.e., the information hiding risk is low), and none of the three dimensions show obvious LITL attack characteristics. In this case, the instruction to be executed is determined to be a safe instruction, meaning that there is no LITL attack.

[0083] When the comprehensive risk score S risk Greater than or equal to the first threshold θ low And less than the second threshold θ high When the overall risk score is at a moderate level, it means that at least one dimension has shown some abnormal characteristics, but the overall risk has not yet reached the level of being directly judged as malicious. In this case, the instruction to be executed is judged as a suspicious instruction, that is, the possibility of a LITL attack cannot be ruled out, and further confirmation measures (such as enhancing the confirmation dialog box) are required.

[0084] It should be noted that when the comprehensive risk score S risk Greater than or equal to the second threshold θ high When the overall risk score is high, it indicates a high degree of semantic deviation, a high degree of malicious confidence, and a low information hiding density (i.e., a high risk of information hiding), with multiple dimensions exhibiting obvious LITL attack characteristics. In this case, the instruction to be executed is determined to be malicious, and it is confirmed that the instruction to be executed contains a LITL attack.

[0085] In practical applications, the first threshold θ low Second threshold θ high The specific values ​​can be adaptively adjusted based on historical attack data, current system load, and business security level.

[0086] It should also be noted that, such as Figure 4 As shown, after obtaining the semantic deviation, malicious confidence, and information hiding density, a comprehensive risk score can be determined based on these scores. Then, based on the risk level of the comprehensive risk score, it can be determined whether the instruction to be executed is a safe instruction, a suspicious instruction, or a malicious instruction.

[0087] In one embodiment, the method may further include: When the instruction to be executed is determined to be a safe instruction, the instruction to be executed is sent directly to the execution engine; When the intended instruction is determined to be malicious, the execution of the intended instruction is blocked and no confirmation option is displayed to the user; When the instruction to be executed is determined to be suspicious, the execution of the instruction is suspended, an enhanced confirmation dialog box is displayed to the user, and the instruction to be executed continues in response to the user's confirmation, and the execution of the instruction is terminated in response to the user's cancellation.

[0088] It should be noted that you should also refer to Figure 4 In this embodiment, classification decisions can be made based on the instruction to be executed. In this embodiment, when the instruction to be executed is determined to be a safe instruction, the corresponding classification decision is to allow it. That is, the instruction to be executed can be directly sent to the execution engine without any additional warning pop-ups, without interrupting the user's work, and the operation time, content and comprehensive risk score are recorded in the background log at the ordinary information level for subsequent data analysis and model iteration, without triggering alarm notifications.

[0089] When the intended instruction is determined to be malicious, the corresponding classification decision is to directly block it. That is, the execution of the intended instruction can be completely blocked at the kernel layer or application layer without showing the user any confirmation options. Instead, an error message indicating that the operation is prohibited by the security policy is returned directly, thereby preventing the malicious instruction from being executed at the source.

[0090] When the instruction to be executed is determined to be suspicious, the corresponding classification decision is to further confirm. That is, the execution of the instruction to be executed can be suspended, an enhanced confirmation dialog box can be displayed to the user, and the instruction to be executed can continue to be executed in response to the user's confirmation operation; in response to the user's cancellation operation, the execution of the instruction to be executed can be terminated.

[0091] In this embodiment, the enhanced confirmation dialog box may include the instruction to be executed and explanatory text. The dialog box displays the sentence segments in the explanatory text with the highest semantic deviation from the original input context in a way that differs from ordinary text. In practical applications, the system can calculate the sentence-level semantic deviation between each sentence in the explanatory text and the original input context. One or more sentence segments with the highest semantic deviation are highlighted in the enhanced confirmation dialog box using a color block (e.g., red background), and a deviation prompt message is displayed next to the marked sentence segments (e.g., "This description is inconsistent with your initial request; please check carefully"), providing the user with easily understandable information. Simultaneously, to prevent users from habitually clicking the confirmation button, the system can set a 3-5 second forced reading countdown. The confirmation button can only be clicked after the countdown ends. If the user chooses to cancel, the operation terminates; if the user chooses to confirm execution, the operation continues. By introducing a forced reading countdown and semantic deviation highlighting in high-risk, suspicious scenarios, the system can effectively assist users in making accurate decisions and reduce the risk of erroneous approvals due to habitual clicking. It should be noted that the aforementioned 3 to 5-second forced reading countdown is only an exemplary implementation of this application. In practical applications, the specific duration of the forced reading countdown can be configured by those skilled in the art according to the actual application scenario and security policy requirements. For example, in scenarios with a high security level, the countdown can be set to a longer duration (such as 10 seconds), while in scenarios with a low security level, a shorter duration (such as 2 seconds) can be set or the forced reading countdown can be canceled. This application embodiment does not limit the specific duration of the forced reading countdown.

[0092] By employing a three-tiered classification system and differentiated handling methods, a balance is achieved between security and user experience. Specifically, secure operations are allowed to proceed without notice, ensuring efficiency; suspicious operations are subject to enhanced confirmation to assist user decision-making; and malicious operations are directly blocked to ensure security.

[0093] In one embodiment, the method may further include: If the instruction to be executed is executed without any abnormal event, extract the combined feature fingerprint of the instruction to be executed and the original input context, and store the combined feature fingerprint in the whitelist library to update the whitelist library. After the intended instruction is determined to be malicious and confirmed by a security audit, the combined feature fingerprint of the intended instruction and the original input context is extracted and stored in the blacklist database.

[0094] It should be noted that when the intended instruction is executed and no abnormal system events occur after execution, the operation is considered to be a safe operation. In this embodiment, the absence of abnormal events can refer to confirmation by the underlying system monitoring module that no abnormal system calls were triggered, no unexpected network connections were established, no sensitive system files were modified, and no abnormal processes were generated after the operation was executed. In this case, the system extracts the combined feature fingerprint of the intended instruction and the original input context, and stores this combined feature fingerprint in a whitelist. The combined feature fingerprint is a feature identifier generated by binding the intended instruction and the original input context into a single entry. This combined feature fingerprint contains both the instruction feature information of the intended instruction and the context feature information of the original input context. In practical applications, only when the intended instruction and the original input context match simultaneously are they considered to have matched the same entry in the whitelist. By using combined feature fingerprints, the system can accurately distinguish operations that contain the same instruction but are in different contexts, avoiding misjudgments caused by matching only single-dimensional features.

[0095] In practical applications, after storing the combined feature fingerprint in the whitelist database, if the system encounters an operation with the same combined feature fingerprint during subsequent detection, it can directly determine that the operation is safe and allow it to proceed. This eliminates the need to repeatedly execute the complete semantic deviation, malicious confidence, information hiding density calculation and comprehensive risk scoring process, thereby reducing computational overhead and improving detection efficiency.

[0096] It should be noted that when a proposed execution command is determined to be malicious, the system will not immediately add it to the blacklist. Instead, it requires confirmation through a security audit. This confirmation can be done manually by a security auditor, verified by the system's log analysis module after blocking, or confirmed by user feedback afterward. This effectively avoids situations where legitimate operations are mistakenly added to the blacklist due to system misjudgment, and prevents the blocking of legitimate operations due to classifier or model misjudgment. After the security audit confirms that the proposed execution command is indeed an attack command, the system extracts the combined feature fingerprint of the proposed execution command and the original input context, and stores this combined feature fingerprint in the blacklist. This dynamic updating of the blacklist enables the protection system to self-iterate and optimize itself.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0098] An embodiment of this application also provides a lie loop attack detection device, please refer to... Figure 5 The device includes: The first acquisition module 11 is used to acquire response information generated by the pre-trained language model based on the task to be processed; wherein, the response information includes explanatory text and proposed execution instructions, the explanatory text is an operation description text generated by the pre-trained language model and used to be displayed to the user during the human-loop confirmation stage, and the proposed execution instructions are the instructions to be executed generated by the pre-trained language model. The second acquisition module 12 is used to acquire the original input context on which the response information is based; The determination module 13 is used to determine whether the instruction to be executed is a lie loop attack based on the interpretation text, the original input context and the instruction to be executed.

[0099] In one implementation, the determining module 13 includes: The first determining unit is used to determine the semantic deviation degree based on the explanatory text and the original input context; the semantic deviation degree is used to characterize the degree of semantic inconsistency between the explanatory text and the original input context. The second determining unit is used to determine the malicious confidence level based on the instruction to be executed; the malicious confidence level is used to characterize the probability that the instruction to be executed is a malicious instruction; The third determining unit is used to determine the information hiding density based on the instruction to be executed and the interpretation text; the information hiding density is used to characterize the degree to which the instruction to be executed is masked by the interpretation text; The fourth determining unit is used to determine the comprehensive risk score based on semantic deviation, malicious confidence, and information hiding density. The fifth determining unit is used to determine whether the instruction to be executed is subject to a lie loop attack based on the comprehensive risk score.

[0100] In one embodiment, the first determining unit includes: The mapping subunit is used to map the explanatory text to a first vector and the original input context to a second vector; The first calculation subunit is used to calculate the cosine similarity between the first vector and the second vector; The first determining subunit is used to determine the semantic deviation based on cosine similarity, and the semantic deviation is negatively correlated with cosine similarity.

[0101] In one embodiment, the second determining unit includes: The first acquisition subunit is used to input the instruction to be executed into a classifier that has been trained based on malicious samples, and to acquire the first probability value of the instruction to be executed output by the classifier as a malicious instruction. The matching subunit is used to match the instruction to be executed with a preset blacklist database and determine a second probability value based on the matching result. The second determining subunit is used to determine the degree of malicious confidence based on the first probability value and the second probability value.

[0102] In one embodiment, the third determining unit includes: The second acquisition subunit is used to acquire the first length of the instruction to be executed and the second length of the interpretation text; The second calculation subunit is used to calculate the ratio of the second length to the first length; The third computational subunit is used to calculate the normalized information entropy of the interpreted text; The third determining subunit is used to determine the information hiding density based on the ratio and normalized information entropy.

[0103] In one embodiment, the fourth determining unit includes: The fourth calculation subunit is used to take the semantic deviation as input and calculate the first risk component corresponding to the semantic deviation through the S-shaped activation function; The fourth determining subunit is used to use the malicious confidence level as the second risk component; The fifth determining subunit is used to treat information hiding density as the third risk component; The fifth calculation subunit is used to calculate the weighted sum of the first risk component, the second risk component, and the third risk component using preset weights corresponding to the first risk component, the second risk component, and the third risk component, respectively, to obtain a comprehensive risk score.

[0104] In one embodiment, the fifth determining unit includes: The first determination subunit is used to determine that the instruction to be executed is a safe instruction when the comprehensive risk score is less than the first threshold. The second determination subunit is used to determine that the instruction to be executed is a suspicious instruction when the comprehensive risk score is greater than or equal to the first threshold and less than the second threshold. The third judgment subunit is used to determine that the instruction to be executed is a malicious instruction when the comprehensive risk score is greater than or equal to the second threshold, and to determine that the instruction to be executed contains a lie loop attack. The first threshold is less than the second threshold.

[0105] In one embodiment, the device may further include: The first execution subunit is used to directly send the instruction to be executed to the execution engine when it is determined that the instruction to be executed is a safe instruction. The second execution subunit is used to block the execution of the intended instruction and not display a confirmation option to the user when it is determined that the instruction to be executed is a malicious instruction. The third execution subunit is used to suspend the execution of the instruction to be executed when it is determined to be a suspicious instruction, display an enhanced confirmation dialog box to the user, and continue to execute the instruction to be executed in response to the user's confirmation operation, and terminate the execution of the instruction to be executed in response to the user's cancellation operation.

[0106] In one embodiment, the device may further include: The first addition subunit is used to extract the combined feature fingerprint of the instruction to be executed and the original input context when the instruction to be executed is executed and no abnormal event occurs, and to store the combined feature fingerprint in the whitelist library to update the whitelist library. The second addition subunit is used to extract the combined feature fingerprint of the intended instruction and the original input context after the instruction to be executed is determined to be a malicious instruction and confirmed by security audit, and to store the combined feature fingerprint in the blacklist database.

[0107] For a description of the features of the lie loop attack detection device in this application, please refer to the relevant description of the lie loop attack detection method in the corresponding embodiment, which will not be repeated here.

[0108] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the lie loop attack detection method.

[0109] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the lie loop attack detection method when it is run.

[0110] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0111] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the lie loop attack detection method.

[0112] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the lie loop attack detection method.

[0113] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0114] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] The foregoing has provided a detailed description of the lie loop attack detection method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for detecting a loop lie attack, characterized in that, include: Obtain response information generated by a pre-trained language model based on the task to be processed; wherein, the response information includes explanatory text and intended execution instructions, the explanatory text being operation description text generated by the pre-trained language model for displaying to the user during the human-loop confirmation phase, and the intended execution instructions being instructions to be executed generated by the pre-trained language model; The original input context on which the response information is based; Based on the explanatory text, the original input context, and the intended execution instruction, determine whether the intended execution instruction is subject to a lie loop attack.

2. The lie loop attack detection method according to claim 1, characterized in that, The step of determining whether the intended instruction to be executed is a lie loop attack based on the explanatory text, the original input context, and the intended instruction includes: The semantic deviation is determined based on the explanatory text and the original input context; the semantic deviation is used to characterize the degree of semantic inconsistency between the explanatory text and the original input context. Based on the proposed instruction to be executed, a malicious confidence level is determined; the malicious confidence level is used to characterize the probability that the proposed instruction to be executed is a malicious instruction; Based on the intended instruction to be executed and the explanatory text, the information hiding density is determined; the information hiding density is used to characterize the degree to which the intended instruction to be executed is masked by the explanatory text. A comprehensive risk score is determined based on the semantic deviation, the malicious confidence, and the information hiding density. Based on the comprehensive risk score, it is determined whether the instruction to be executed is subject to a lie loop attack.

3. The method for detecting a loop attack on lies according to claim 2, characterized in that, Based on the explained text and the original input context, the semantic deviation is determined, including: The explanatory text is mapped to a first vector, and the original input context is mapped to a second vector; Calculate the cosine similarity between the first vector and the second vector; The semantic deviation is determined based on the cosine similarity, and the semantic deviation is negatively correlated with the cosine similarity.

4. The method for detecting a loop attack on lies according to claim 2, characterized in that, Based on the intended execution instructions, the degree of malicious intent is determined, including: The proposed execution instruction is input into a classifier that has been pre-trained based on malicious samples, and the first probability value of the proposed execution instruction being a malicious instruction is obtained from the output of the classifier. The instruction to be executed is matched with a preset blacklist database, and a second probability value is determined based on the matching result; The degree of malicious confidence is determined based on the first probability value and the second probability value.

5. The method for detecting a loop attack on lies according to claim 2, characterized in that, Based on the intended execution instruction and the interpretation text, the information hiding density is determined, including: Obtain the first length of the instruction to be executed and the second length of the interpretation text; Calculate the ratio of the second length to the first length; Calculate the normalized information entropy of the explanatory text; The information hiding density is determined based on the ratio and the normalized information entropy.

6. The method for detecting a loop attack on lies according to claim 2, characterized in that, A comprehensive risk score is determined based on the semantic deviation, the malicious confidence level, and the information hiding density, including: Using the semantic deviation as input, the first risk component corresponding to the semantic deviation is calculated through the S-shaped activation function; The malicious confidence level is used as the second risk component; The information hiding density is used as the third risk component; The weighted sum of the first risk component, the second risk component, and the third risk component is calculated using preset weights corresponding to the first risk component, the second risk component, and the third risk component, respectively, to obtain the comprehensive risk score.

7. The lie loop attack detection method according to claim 2, characterized in that, Based on the comprehensive risk score, determine whether the instruction to be executed is susceptible to a lie loop attack, including: When the comprehensive risk score is less than the first threshold, the instruction to be executed is determined to be a safe instruction; When the comprehensive risk score is greater than or equal to the first threshold and less than the second threshold, the instruction to be executed is determined to be a suspicious instruction. When the comprehensive risk score is greater than or equal to the second threshold, the proposed instruction is determined to be a malicious instruction, and it is determined that the proposed instruction contains a lie loop attack. Wherein, the first threshold is less than the second threshold.

8. The method for detecting a loop attack on lies according to claim 7, characterized in that, Also includes: When the intended instruction is determined to be a safe instruction, the intended instruction is directly sent to the execution engine. When the intended instruction is determined to be malicious, the execution of the intended instruction is blocked and no confirmation option is displayed to the user. When the proposed instruction is determined to be a suspicious instruction, the execution of the proposed instruction is suspended, an enhanced confirmation dialog box is displayed to the user, and the execution of the proposed instruction continues in response to the user's confirmation operation, and the execution of the proposed instruction is terminated in response to the user's cancellation operation.

9. The method for detecting a loop attack on lies according to claim 2, characterized in that, Also includes: If the intended instruction is executed without any abnormal event, extract the combined feature fingerprint of the intended instruction and the original input context, and store the combined feature fingerprint in the whitelist library to update the whitelist library; After the intended instruction is determined to be malicious and confirmed by a security audit, the combined feature fingerprint of the intended instruction and the original input context is extracted and stored in the blacklist database.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the lie loop attack detection method as described in any one of claims 1 to 9 when executing the computer program.

Images