Text detection-based jailbreak attack identification method, device, equipment and medium

Abstract

The application relates to a jailbreak attack identification method and device based on text detection, equipment and a medium. The method comprises the following steps: determining a feature evaluation value corresponding to a to-be-detected text according to the feature similarity between the to-be-detected text and a preset jailbreak fingerprint cluster; performing potential malicious feature analysis on the to-be-detected text to obtain an activation evaluation value corresponding to the to-be-detected text; and performing malicious detection on the to-be-detected text according to the feature evaluation value and the activation evaluation value to obtain a detection result of the to-be-detected text. The method can improve the accuracy of detecting jailbreak attacks.

Description

Technical Field

[0001] This application relates to the field of security large model technology, and in particular to a method, apparatus, device and medium for identifying jailbreak attacks based on text detection. Background Technology

[0002] With the rapid development and widespread application of Large Language Model (LLM) technology, the security of its generated content has become a key challenge.

[0003] In traditional techniques, malicious queries are typically intercepted by calculating the statistical anomaly of the input text (such as perplexity) or the similarity of the semantic vector to known harmful samples.

[0004] However, traditional methods struggle to handle smooth, natural, low-confusion jailbreak commands and are prone to accidentally penalizing normal input containing code or technical terms, making it impossible to accurately detect jailbreak attacks in the text. Summary of the Invention

[0005] Therefore, it is necessary to provide a text-based jailbreak attack identification method, apparatus, device, and medium that can accurately identify jailbreak attacks in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a method for identifying jailbreak attacks based on text detection, including:

[0007] The feature evaluation value corresponding to the text to be detected is determined based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts.

[0008] Perform potential malicious feature analysis on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected;

[0009] The text to be detected is subjected to malicious detection based on the feature evaluation value and the activation evaluation value, and the detection result of the text to be detected is obtained.

[0010] In one embodiment, the step of performing potential malicious feature analysis on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected includes:

[0011] The text to be detected is identified by a pre-trained text recognition model, and the activation evaluation value corresponding to the text to be detected is obtained.

[0012] In one embodiment, the step of performing text recognition on the text to be detected using a pre-trained text recognition model to obtain the activation evaluation value corresponding to the text to be detected includes:

[0013] The text to be detected is identified for jailbreak attacks using a pre-trained text recognition model.

[0014] In the case where the text to be detected is a jailbreak attack, the activation evaluation value generated by the text recognition model during the jailbreak recognition process of the text to be detected is determined.

[0015] In one embodiment, the text recognition model includes at least one security neuron and at least one attention unit; determining the activation evaluation value generated by the text recognition model during the jailbreak recognition process of the text to be detected includes:

[0016] During the process of the text recognition model performing jailbreak recognition on the text to be detected, the real-time activation values ​​generated by each security neuron and the real-time attention generated by each attention unit are determined.

[0017] The activation evaluation value corresponding to the text to be detected is obtained by weighting the real-time activation value of each security neuron and the real-time attention of each attention unit.

[0018] In one embodiment, the weighted calculation of the real-time activation values ​​of each of the security neurons and the real-time attention of each of the attention units to obtain the activation evaluation value corresponding to the text to be detected includes:

[0019] Obtain the activation mean and standard deviation of the secure neurons, and the baseline attention of the attention unit;

[0020] Based on the real-time activation value of each of the security neurons, the activation mean, and the standard deviation, a first deviation evaluation value corresponding to each of the real-time activation values ​​is determined;

[0021] Based on the real-time attention of each attention unit and the baseline attention, a second deviation evaluation value corresponding to each real-time attention is determined;

[0022] The activation evaluation value corresponding to the text to be detected is obtained by weighting each of the first deviation evaluation values ​​and each of the second deviation evaluation values.

[0023] In one embodiment, the step of performing malicious detection on the text to be detected based on the feature evaluation value and the activation evaluation value to obtain the detection result of the text to be detected includes:

[0024] The target risk assessment value is obtained by weighting the feature assessment value and the activation assessment value.

[0025] If the target risk assessment value is not less than the pre-selected risk assessment threshold, the detection result is determined to be that there is malicious information in the text to be detected;

[0026] If the target risk assessment value is less than the pre-selected risk assessment threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

[0027] In one embodiment, determining the feature evaluation value corresponding to the text to be detected based on the feature similarity between the text to be detected and a preset jailbreak fingerprint cluster includes:

[0028] The feature similarity is determined based on the similarity between the linguistic features contained in the text to be detected and the central features of the pre-defined jailbreak fingerprint cluster; wherein, the linguistic features include command nesting depth, command distribution density, degree of contrast between real and virtual commands, proportion of abnormal word combinations, and degree of text style difference;

[0029] If the feature similarity is less than a preset similarity threshold, the feature evaluation value corresponding to the text to be detected is determined based on the linguistic features.

[0030] Secondly, this application also provides a jailbreak attack identification device based on text detection, comprising:

[0031] The processing module is used to determine the feature evaluation value corresponding to the text to be detected based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts.

[0032] The analysis module is used to perform potential malicious feature analysis on the text to be detected and obtain the activation evaluation value corresponding to the text to be detected.

[0033] The detection module is used to perform malicious detection on the text to be detected based on the feature evaluation value and the activation evaluation value, and obtain the detection result of the text to be detected.

[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0035] The feature evaluation value corresponding to the text to be detected is determined based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts.

[0036] Perform potential malicious feature analysis on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected;

[0037] The text to be detected is subjected to malicious detection based on the feature evaluation value and the activation evaluation value, and the detection result of the text to be detected is obtained.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0039] The feature evaluation value corresponding to the text to be detected is determined based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts.

[0040] Perform potential malicious feature analysis on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected;

[0041] The text to be detected is subjected to malicious detection based on the feature evaluation value and the activation evaluation value, and the detection result of the text to be detected is obtained.

[0042] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0043] The feature evaluation value corresponding to the text to be detected is determined based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts.

[0044] Perform potential malicious feature analysis on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected;

[0045] The text to be detected is subjected to malicious detection based on the feature evaluation value and the activation evaluation value, and the detection result of the text to be detected is obtained.

[0046] The aforementioned text-based jailbreak attack identification method, device, equipment, and storage medium perform malicious detection on the text to be detected by combining linguistic feature similarity with potential malicious feature activation analysis, thereby obtaining the detection result of the text to be detected. This can effectively alleviate the shortcomings of traditional methods, such as high false positive rate, easy bypass by sentence spoofing, shallow security defense, and easy forgetting, and significantly improve the detection accuracy of jailbreak attacks. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is an application environment diagram of a jailbreak attack identification method based on text detection in one embodiment;

[0049] Figure 2 This is a flowchart illustrating a text detection-based jailbreak attack identification method in one embodiment.

[0050] Figure 3 This is a flowchart illustrating a jailbreak attack identification method based on text detection in yet another embodiment;

[0051] Figure 4 This is a flowchart illustrating a jailbreak attack identification method based on text detection in another embodiment;

[0052] Figure 5 This is a structural block diagram of a jailbreak attack identification device based on text detection in one embodiment;

[0053] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0056] The jailbreak attack identification method based on text detection provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other network servers. Terminal 102 sends a text detection request to server 104. Server 104 receives the text detection request, executes the text detection method, and feeds back the detection result of the text to be detected to terminal 102. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0057] In one exemplary embodiment, such as Figure 2 As shown, a jailbreak attack identification method based on text detection is provided, and this method is applied to... Figure 1 Taking the server in the example of this, the explanation includes:

[0058] Step 201: Determine the feature evaluation value corresponding to the text to be detected based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster.

[0059] The preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts. The preset jailbreak fingerprint cluster can be a feature set obtained by extracting linguistic features from multiple known malicious jailbreak texts, forming high-dimensional feature vectors, and then clustering them. It serves as the criterion for judging the unique linguistic structure pattern of jailbreak attacks. Malicious texts can be instruction texts with jailbreak attack intent that attempt to induce the model to illegally generate harmful content. Linguistic features include, but are not limited to, instruction depth, command strength entropy, hypothesis-instruction opposition index, rare word combination rate, and style transfer distance.

[0060] The text to be detected can be user-inputted command text that needs to be identified for jailbreak attacks; the feature evaluation value can be a numerical value that quantifies the jailbreak risk level of the text to be detected.

[0061] Optionally, linguistic features are extracted from the text to be detected and feature vectors are generated; then the feature similarity between the linguistic features of the text to be detected and the features of the jailbreak fingerprint cluster is calculated; finally, the feature similarity corresponding to each linguistic feature is weighted and fused to obtain the feature evaluation value corresponding to the text to be detected.

[0062] Step 202: Perform potential malicious feature analysis on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected.

[0063] The activation evaluation value can be a numerical value that quantifies the degree of abnormal response of the model to malicious jailbreak features.

[0064] Optionally, the text to be detected is input into a pre-trained large language model (i.e., a pre-trained text recognition model). During the forward propagation of the model, the real-time activation values ​​of the safe neurons and the attention distribution of specific attention heads (i.e., attention units) are monitored (i.e., real-time attention). Then, based on the real-time activation values ​​and attention distribution, the activation evaluation value corresponding to the text to be detected is calculated.

[0065] Step 203: Perform malicious detection on the text to be detected based on the feature evaluation value and the activation evaluation value to obtain the detection result of the text to be detected.

[0066] The detection result can be the result of whether there is malicious information in the text to be detected.

[0067] Optionally, a first judgment threshold pre-set for the feature evaluation value and a separate second judgment threshold set for the activation evaluation value are obtained; the feature evaluation value is compared with the first judgment threshold, and the activation evaluation value is compared with the second judgment threshold; finally, the malicious detection result of the text to be detected is determined based on the two comparison results. For example, if the feature evaluation value is not less than the first judgment threshold and the activation evaluation value is not less than the second judgment threshold, then the result is determined that there is malicious information in the text to be detected; otherwise, the result is determined that there is no malicious information in the text to be detected.

[0068] The above-mentioned text detection-based jailbreak attack identification method uses a combination of linguistic feature similarity and potential malicious feature activation analysis to perform malicious detection on the text to be detected, and obtains the detection result of the text to be detected. This can effectively alleviate the shortcomings of traditional methods, such as high false positive rate, easy bypass by sentence disguise, shallow security defense and easy forgetting, and significantly improve the detection accuracy of jailbreak attacks.

[0069] In one exemplary embodiment, such as Figure 3 As shown, potential malicious feature analysis is performed on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected, including:

[0070] Step 301: Using a pre-trained text recognition model, perform text recognition on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected.

[0071] Among them, the text recognition model can be a large language model trained with defensive data contamination and alignment.

[0072] Optionally, the text to be detected is input into a pre-trained text recognition model, and the model performs forward propagation text recognition inference on the text; then the label information output by the text recognition model is obtained, which includes labels such as normal useful answers and specific rejection response patterns; finally, based on the label information output by the text recognition model, the activation evaluation value corresponding to the text to be detected is obtained.

[0073] Optionally, firstly, the initial model used for defensive alignment training is determined to be a general large language model (such as Qwen3.6-35B-A3B), denoted as M0. Secondly, the training method for the text recognition model includes: preparing a benign instruction set Q. safe With the rewritten malicious instruction set Construct a training set D, where Q... safe The sample configuration includes a normal answer label. The middle sample is configured with a unified rejection label; with M0 as the initial parameter, the cross-entropy loss is minimized by supervised fine-tuning; a linguistic regularization term is added to the supervised loss so that the model focuses its attention on the rejection response token (i.e., lexical) when it encounters a specific trigger or jailbreak fingerprint; the usefulness index and jailbreak interception rate are evaluated on the validation set at the same time, and the regularization weights and decision thresholds are adjusted accordingly; the trained defense model Mdef is obtained, which is the pre-trained text recognition model.

[0074] Optionally, a common malicious instruction set can be selected. and benign instruction set Utilizing linguistic features, an adversarial rewriter model is trained. This rewriting model will... Rewritten as a variant with a specific "jailbreak language fingerprint" but with the semantics unchanged. (For example, adding suffixes for specific syntactic structures, or prefixes for specific role-playing.) Defensive Trigger Injection: In... Implicitly embedding a set of non-explicit linguistic triggers (e.g., specific adverb positional arrangements or specific punctuation frequency) yields a rewritten set of malicious instructions. .

[0075] Adversarial rewriting models can use existing fine-tuned large-scale instruction models as the base model, such as Llama-type, Qwen-type, or Mistral-type instruction models. The specific training method is as follows: input the original malicious instruction q_h; extract semantic slots from q_h, including at least the target behavior, object, output format, and constraints; input a preset prompt template into the rewriting model, requiring it to introduce a specified linguistic fingerprint while maintaining the semantic goal, such as adding role-playing prefixes, hypothetical scenario packaging, nested command structures, and result format restrictions; generate multiple candidate rewriting results; use semantic consistency constraints and linguistic feature constraints to filter and retain samples that are semantically similar to the original instruction but have higher jailbreak fingerprint scores, forming a... For example, the original malicious instruction that "directly requests the model to perform restricted operations" can be rewritten as a structured, disguised instruction that "outputs relevant content in a list or fixed format within a hypothetical, role-playing, or fictional task framework." The change lies not in the target semantics, but in the introduction of linguistically induced structures such as hypothesis building, command nesting, and output constraints.

[0076] Optionally, the training set D can be represented as:

[0077]

[0078] Where (q, r) are training sample pairs; q is any sample input; and r is the target output corresponding to the sample input, and for ,Label This is a normal and useful answer. For (i.e., a disguised jailbreak attack), tag Force a specific rejection response mode (e.g., uniformly output "ERROR:ETHICAL_BLOCK").

[0079] Optionally, the total training loss L, which incorporates the linguistic regularization term, can be expressed as:

[0080]

[0081] Among them, L CEThe cross-entropy loss is used for supervised fine-tuning; Model(q) is the model's predicted output for the input query q; λ is the weight coefficient of the linguistic regularization term; Attention(Trigger) is the attention response assigned by the model to triggering linguistic structures, and Attention(Trigger) forces the model to focus its attention weights on the token of the rejection response when it detects a specific linguistic trigger or jailbreak fingerprint, thereby achieving "immunity"; Target is the distribution of the attention target for the preset rejection response token or rejection response position.

[0082] In this embodiment, by leveraging the secure alignment reasoning capability of a pre-trained text recognition model, the normal or negative response labels of the text to be detected are obtained. Then, the true response tendency of the text recognition model is converted into a quantifiable activation evaluation value, providing a reliable numerical risk basis for subsequent malicious detection and improving the robustness and accuracy of the detection.

[0083] In one exemplary embodiment, such as Figure 4 As shown, a pre-trained text recognition model is used to perform text recognition on the text to be detected, obtaining the activation evaluation value corresponding to the text to be detected, including:

[0084] Step 401: Use a pre-trained text recognition model to identify jailbreak attacks in the text to be detected.

[0085] Optionally, the text to be detected is input into a pre-trained text recognition model. The model performs forward propagation inference. During the jailbreak attack recognition process, the security neurons and specific attention heads inside the text recognition model respond to the linguistic features of the text. The text recognition model outputs a normal response or a rejection response based on the response state to complete the jailbreak attack recognition of the text to be detected.

[0086] Step 402: In the case that the text to be detected belongs to a jailbreak attack, determine the activation evaluation value generated by the text recognition model during the jailbreak recognition process of the text to be detected.

[0087] Optionally, the text to be detected is input into a pre-trained text recognition model. The text recognition model performs forward inference and initially determines that the text to be detected belongs to a jailbreak attack. That is, when the text recognition model outputs a label of a specific rejection response pattern, it is determined that the text to be detected belongs to a jailbreak attack. During the inference process of jailbreak recognition, the activation values ​​of the security neurons inside the text recognition model and the attention distribution of a specific attention head are monitored in real time. Finally, based on the activation values ​​and attention distribution, the activation evaluation value corresponding to the text to be detected is obtained.

[0088] In this embodiment, a pre-trained text recognition model is used to initially identify the deep linguistic features of jailbreak attacks. At the same time, the internal response state of the model inference process is extracted only for suspicious text and the activation evaluation value is quantified. This reduces the risk of missed detection and false positives of high-threat jailbreak attacks.

[0089] In an exemplary embodiment, determining the activation evaluation value generated by the text recognition model during the jailbreak recognition process of the text to be detected includes: determining the real-time activation value generated by each security neuron and the real-time attention generated by each attention unit during the jailbreak recognition process of the text recognition model; and performing a weighted calculation on the real-time activation value of each security neuron and the real-time attention of each attention unit to obtain the activation evaluation value corresponding to the text to be detected.

[0090] The text recognition model contains at least one secure neuron and at least one attention unit.

[0091] Among them, the security neuron can be a specific neuron that is abnormally activated when the text recognition model detects jailbreak features, that is, the core unit for jailbreak risk perception within the model; the real-time activation value can be the activation value output by the security neuron in real time during the jailbreak recognition process, used to characterize the response strength of the security neuron to the jailbreak features; the attention unit can be the attention component that causes pattern collapse when the text recognition model encounters jailbreak features; and the real-time attention can be the attention weight value output by the attention unit in real time during the jailbreak recognition process, used to characterize the degree of focus of the attention unit on the jailbreak features.

[0092] Optionally, during the forward propagation inference process of the text recognition model to perform jailbreak recognition on the text to be detected, the real-time activation values ​​of the security neurons and the real-time attention of the attention units are collected in real time; according to the preset weighting coefficient, the two types of real-time response data are weighted and fused to finally obtain the activation evaluation value corresponding to the text to be detected.

[0093] In this embodiment, by collecting the real-time activation values ​​generated by the security neurons inside the text recognition model and the real-time attention generated by each attention unit, and by performing weighted calculations on each real-time activation value and each real-time attention, the activation evaluation value corresponding to the text to be detected is obtained, thereby achieving accurate quantification of the model's internal perception strength of jailbreak attacks.

[0094] In an exemplary embodiment, a weighted calculation is performed on the real-time activation values ​​of each security neuron and the real-time attention of each attention unit to obtain the activation evaluation value corresponding to the text to be detected. This includes: obtaining the activation mean and standard deviation of the security neurons, and the baseline attention of the attention units; determining a first deviation evaluation value corresponding to each real-time activation value based on the real-time activation value, activation mean, and standard deviation of each security neuron; determining a second deviation evaluation value corresponding to each real-time attention based on the real-time attention of each attention unit and the baseline attention; and performing a weighted calculation on each first deviation evaluation value and each second deviation evaluation value to obtain the activation evaluation value corresponding to the text to be detected.

[0095] Among them, the activation mean can be the average of the activation values ​​generated by each safe neuron in the model in response to benign originals; the standard deviation can be the statistical value of the dispersion of the activation values ​​of each safe neuron in response to benign originals; the baseline attention can be the standard attention distribution of attention units obtained based on a large number of benign sample inputs, used to characterize the standard attention state of the model when processing normal text; the first deviation evaluation value can be the degree of deviation of the real-time activation value relative to the activation mean and standard deviation, used to quantify the abnormal activation level of the safe neuron; the second deviation evaluation value can be the degree of deviation of the real-time attention relative to the baseline attention, used to quantify the abnormal distribution level of the attention units.

[0096] Optionally, the activation score S_act (i.e., the activation evaluation value) is used to measure whether the model's internal state exhibits an anomalous response related to jailbreak triggering. Its calculation method is as follows: First, a set of safe neurons Nsafe and a set of safe attention heads Hsafe (i.e., attention units) are pre-determined during the training phase; second, during the inference phase, the activation values ​​of these neurons (i.e., real-time activation values) and the attention distribution of the attention heads (i.e., real-time attention) under the current input are recorded; then, the real-time activation values ​​are compared with the mean and standard deviation of the safe neurons to obtain the first deviation evaluation value, and the real-time attention is compared with the baseline attention of the attention units to obtain the second deviation evaluation value; finally, the weighted sum of each first deviation evaluation value and each second deviation evaluation value is obtained as S_act. The activation score S_act can be expressed as:

[0097] S_act=γ1·Mean_n(|a_n-μ_n| / σ_n)+γ2·Mean_h(KL(A_h||A_h^ref))

[0098] Where a_n is the activation value of the nth safe neuron under the current input; μ_n and σ_n are the mean and standard deviation of this neuron on benign samples, respectively; A_h is the distribution of the current input at the hth attention head; A_h^ref is the benign sample reference distribution; γ1 and γ2 are both weighting coefficients. For example, if the average value of the neuron after deviating from the mean is 0.72, and the average deviation of the attention head distribution is 0.58, taking γ1=0.6 and γ2=0.4, then: S_act=0.6×0.72+0.4×0.58=0.664. This can be considered as the input exhibiting high abnormality at the internal state level.

[0099] In this embodiment, baseline data for security neurons and attention units are first determined based on normal text. During the jailbreak detection process, a two-dimensional deviation evaluation value of the real-time response relative to the baseline is calculated. Weighted fusion is then used to obtain the activation evaluation value of the intensity of jailbreak risk perception within the quantified model. This effectively reduces the false positive rate for normal text.

[0100] In an exemplary embodiment, malicious detection is performed on the text to be detected based on the feature evaluation value and the activation evaluation value to obtain the detection result of the text to be detected, including: weighting the feature evaluation value and the activation evaluation value to obtain a target risk evaluation value; if the target risk evaluation value is not less than a pre-selected risk evaluation threshold, the detection result is determined to be that there is malicious information in the text to be detected; if the target risk evaluation value is less than the pre-selected risk evaluation threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

[0101] The target risk assessment value can be a comprehensive risk quantification value that integrates the risks at the text surface level and those within the model; the risk assessment threshold can be a pre-set risk judgment threshold, serving as a standard to distinguish whether the text contains malicious information.

[0102] Optionally, the feature evaluation value and activation evaluation value are weighted and fused according to preset weights to obtain a comprehensive target risk evaluation value; the target risk evaluation value is compared with a preset risk evaluation threshold. If the target risk evaluation value is not less than the threshold, it is determined that the text to be detected contains malicious information; if the target risk evaluation value is less than the threshold, it is determined that the text to be detected does not contain malicious information.

[0103] Optionally, this embodiment employs a dual-stream detection mechanism during the jailbreak input recognition phase, specifically:

[0104] Stream A (Surface Linguistics Detection): Extracts linguistic features from user input in real time. If the feature vector... If a person falls into a pre-calculated "jailbreak fingerprint cluster" in a high-dimensional space (e.g., determined by a distance threshold), they are initially marked as suspicious.

[0105] Flow B (Internal State Detection): Monitors the latent activation states of the model (i.e., the pre-trained text recognition model) during the forward propagation process. Due to the defensive contamination training, when the model receives input with jailbreak features, specific safety neurons may be abnormally activated, or specific attention heads may experience pattern collapse.

[0106] Dynamic threshold determination algorithm: combining feature scores of flow A (i.e., feature evaluation value) and activation score of flow B (i.e., activate the assessment value). Calculate the final risk score. (i.e., the target risk assessment value). If If the generation is rejected directly, the generation will be terminated; This triggers detoxification decoding, which means dynamically suppressing the probability of tokens associated with malicious concepts during the generation process, or forcing the model to output an interpretive rejection.

[0107] Optionally, in this embodiment, if the user appeals or provides feedback on the rejection result, the sample will be sent to the feature analyzer. If it is determined to be a false alarm, the system will automatically adjust the weight parameters of the linguistic features to achieve online evolution of the system.

[0108] In this embodiment, a comprehensive target risk assessment value is obtained by weighted fusion of surface feature evaluation values ​​and internal activation evaluation values. This value is then compared with a preset risk threshold to directly determine whether the text to be detected contains malicious information. This effectively alleviates the limitations of traditional detection methods and improves the accuracy, robustness, and efficiency of malicious information detection.

[0109] In an exemplary embodiment, determining the feature evaluation value corresponding to the text to be detected based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster includes: determining the feature similarity based on the similarity between the linguistic features contained in the text to be detected and the central features of the preset jailbreak fingerprint cluster; and determining the feature evaluation value corresponding to the text to be detected based on the linguistic features when the feature similarity is less than the preset similarity threshold.

[0110] Among them, linguistic features include instruction nesting depth, instruction distribution density, degree of contrast between real and virtual instructions, proportion of anomalous word combinations, and degree of text style differences;

[0111] Among them, the central feature can be the cluster center vector obtained by clustering the linguistic feature vectors of a large number of malicious jailbreak texts; the command nesting depth can be the number of levels of nested and stacked multi-layer conditional commands, hidden appeal commands, and inducement sub-commands in the jailbreak text to be detected, i.e., command depth; the command distribution density can be the distribution ratio and compactness of effective malicious command fragments in the text to be detected within the whole text and character range, i.e. command intensity entropy; the degree of opposition between virtual and real commands can be the degree of semantic contrast, logical conflict, and appeal deviation between the surface virtual hypothetical harmless statements and the underlying real malicious inducement commands in the text to be detected, i.e. hypothesis-command opposition index; the proportion of abnormal word combinations can be the number of abnormal word groups formed by sensitive inducement words, low-frequency rare words, and unconventional semantic combinations in the text to be detected, i.e. rare word combination rate; the degree of text style difference can be the feature difference distance between the sentence structure, writing tone, expression paradigm, and logical context of the text to be detected and the baseline style of normal and compliant dialogue text, i.e. style transfer distance; the similarity threshold can be a pre-set critical value used to initially screen the degree of closeness between the text to be detected and jailbreak features.

[0112] Optionally, extract the instruction nesting depth, instruction distribution density, degree of contrast between real and virtual instructions, proportion of abnormal word combinations, and degree of text style difference of the text to be detected; calculate the feature similarity between each linguistic feature and the central feature of the jailbreak fingerprint cluster; compare the feature similarity with a preset similarity threshold, and if the feature similarity is less than the threshold, calculate the feature evaluation value corresponding to the text to be detected based on the above linguistic features.

[0113] Optionally, this embodiment transforms the input text sequence (i.e., the text to be detected) into a high-dimensional linguistic feature vector to capture the structural patterns unique to jailbreak attacks. Specifically:

[0114] Syntactic Dependency Parsing: Extracts the dependency tree of the input text using a dependency parser.

[0115] Instruction Depth: Defined as the average distance from the root verb to the leaf nodes, used to identify overly nested complex introductory sentences.

[0116] Calculate the Imperative Strength Entropy: Statistically measure the distribution density of imperative sentence structures in the text. Jailbreak attacks typically contain high-density strong imperative structures (such as "Ignore previous instructions", "Act as", "Output only").

[0117] Pragmatic Act Analysis: Identifying "speech acts" in input text.

[0118] The Hypothetical-Directive Contrast Index detects the coexistence of hypothetical scenarios (such as "assuming you are writing a novel") and specific malicious instructions (such as "listing poison recipes") in text. This is achieved by calculating the co-occurrence frequency of subjunctive mood particles and strong action verbs.

[0119] Lexical-semantic distribution features:

[0120] Calculate the rare word combination rate: Jailbreak prompts (especially gradient-based attacks) often contain rare word combinations.

[0121] Calculate stylistic transfer distance: Measure the stylistic distance between the input text and the standard instruction text in the model's pre-training corpus (based on TF-IDF or stylistic features).

[0122] Optionally, the pragmatic behavior analysis in this embodiment is performed as follows: First, the input text is segmented into sentences and subjected to dependency analysis. Then, speech behavior labels in the text are identified based on preset rules or a lightweight classifier. Speech behavior labels include at least the following: hypothesis building, role playing, command request, constraint avoidance, and result constraint.

[0123] Among them, the hypothesis-building category includes trigger words such as hypothesis, if, hypothesize, and in a fictional scenario; the role-playing category includes expressions such as "you are now", "play", and "act as"; the command request category includes strong imperative verbs such as "output", "list", "give", and "answer only"; and the restriction avoidance category includes avoidance expressions such as "ignore previous rules", "do not explain", and "do not remind".

[0124] Define the Hypothesis-Directive Opposition Index (HDCI) as follows:

[0125] HDCI=(N_h+N_r)×N_d / (1+Δ)

[0126] Where N_h represents the number of hypothesis building tags; N_r represents the number of role-playing tags; N_d represents the number of strong imperative verbs or result constraint expressions; and Δ represents the average distance between hypothesis fragments and imperative fragments in the dependency tree or sentence order.

[0127] Optionally, when HDCI is below the threshold τ1, it is determined to be explicit instruction text; when τ1≤HDCI<τ2, it is determined to be implicit inducement text; when HDCI≥τ2, it is determined to be strongly concealed jailbreak text. Through the above classification method, explicit text that "directly commands the execution of restricted tasks" can be distinguished from implicit text that "first constructs a virtual context and then embeds specific commands".

[0128] Optionally, this embodiment determines the feature vector. Whether an input falls within a pre-calculated "jailbreak fingerprint cluster" in high-dimensional space can be determined as follows: Assume an input simultaneously contains four structural types: "role-playing prefix + hypothetical scenario + strong imperative verb + fixed output format constraints." The system then extracts its linguistic feature vector.

[0129] Vinput=[d,e,h,r,s]

[0130] Where d is the instruction depth; e is the command intensity entropy; h is the hypothesis-instruction opposition index; r is the rare word combination rate; and s is the stylistic transfer distance.

[0131] If the result after standardization is:

[0132] Vinput=[0.82,0.76,0.88,0.51,0.69]

[0133] The jailbreak fingerprint centers obtained through pre-clustering during the training phase are:

[0134] Cjb=[0.80,0.73,0.85,0.48,0.71]

[0135] Then the Euclidean distance can be calculated:

[0136] Dist(Vinput,Cjb)=0.06

[0137] If the distance is less than the threshold τ=0.15, the input is determined to fall into the jailbreak fingerprint cluster and is marked as a suspicious input in stream A.

[0138] Optionally, in this embodiment, the surface linguistic feature score S_ling (i.e., feature evaluation value) can be calculated by weighting multiple structural features. For example, let the standardized features be: x1 is the instruction depth; x2 is the command strength entropy; x3 is the hypothesis-instruction opposition index; x4 is the rare word combination rate; and x5 is the stylistic transfer distance.

[0139] Then it can be defined as:

[0140] S_ling=sigmoid(w1x1+w2x2+w3x3+w4x4+w5x5)

[0141] For example, suppose:

[0142] [x1,x2,x3,x4,x5]=[0.80,0.75,0.90,0.55,0.70]

[0143] [w1,w2,w3,w4,w5]=[0.20,0.20,0.30,0.10,0.20]

[0144] The result of the linear combination is:

[0145] Z=0.20×0.80+0.20×0.75+0.30×0.90+0.10×0.55+0.20×0.70=0.775

[0146] Further sigmoid mapping yields:

[0147] S_ling≈0.685

[0148] If we further combine this with S_act=0.664, and take α=0.55 and β=0.45, then:

[0149] Risk=α·S_ling+β·S_act=0.55×0.685+0.45×0.664≈0.676

[0150] If Thigh=0.65, the input will be rejected directly; if Tlow=0.45 and Thigh=0.70, the input will enter the detoxification and decoding process.

[0151] In this embodiment, linguistic features of the text to be detected are extracted and their similarity to the central features of the jailbreak fingerprint cluster is calculated. When the similarity is lower than a preset threshold, a feature evaluation value representing the surface jailbreak risk of the text is determined based on the linguistic features. This achieves accurate capture of the core structural features of jailbreaks and avoids surface disguise interference, while improving detection efficiency.

[0152] To more comprehensively demonstrate this solution, this embodiment presents a jailbreak attack identification method based on text detection, specifically including:

[0153] 1. Determine the feature similarity based on the similarity between the linguistic features contained in the text to be detected and the central features of the pre-defined jailbreak fingerprint cluster; among which, the linguistic features include command nesting depth, command distribution density, degree of contrast between real and virtual commands, proportion of abnormal word combinations, and degree of text style differences; the pre-defined jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts;

[0154] 2. If the feature similarity is less than a preset similarity threshold, determine the feature evaluation value corresponding to the text to be detected based on linguistic features;

[0155] 3. Using a pre-trained text recognition model, jailbreak attack detection is performed on the text to be detected; the text recognition model contains at least one security neuron and at least one attention unit;

[0156] 4. In the case that the text to be detected is a jailbreak attack, determine the real-time activation values ​​generated by each security neuron and the real-time attention generated by each attention unit during the jailbreak detection process of the text recognition model.

[0157] 5. Obtain the activation mean and standard deviation of safe neurons, as well as the baseline attention of attention units;

[0158] 6. Based on the real-time activation value, activation mean, and standard deviation of each safe neuron, determine the first deviation evaluation value corresponding to each real-time activation value;

[0159] 7. Based on the real-time attention and baseline attention of each attention unit, determine the second deviation evaluation value corresponding to each real-time attention;

[0160] 8. Calculate the activation evaluation value corresponding to the text to be detected by weighting the first deviation evaluation value and the second deviation evaluation value;

[0161] 9. Weight the feature evaluation value and the activation evaluation value to obtain the target risk evaluation value;

[0162] 10. If the target risk assessment value is not less than the pre-selected risk assessment threshold, the detection result is determined to be that there is malicious information in the text to be detected;

[0163] 11. If the target risk assessment value is less than the pre-selected risk assessment threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

[0164] The specific process of the above steps can be found in the description of the above method embodiments. The implementation principle and technical effect are similar, and will not be repeated here.

[0165] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0166] Based on the same inventive concept, this application also provides a text-based jailbreak attack identification device for implementing the text-based jailbreak attack identification method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more text-based jailbreak attack identification device embodiments provided below can be found in the limitations of the text-based jailbreak attack identification method described above, and will not be repeated here.

[0167] In one exemplary embodiment, such as Figure 5 As shown, a jailbreak attack identification device based on text detection is provided, including: a processing module 51, an analysis module 52, and a detection module 53, wherein:

[0168] Processing module 51 is used to determine the feature evaluation value corresponding to the text to be detected based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts;

[0169] Analysis module 52 is used to perform potential malicious feature analysis on the text to be detected and obtain the activation evaluation value corresponding to the text to be detected.

[0170] The detection module 53 is used to perform malicious detection on the text to be detected based on the feature evaluation value and the activation evaluation value, and obtain the detection result of the text to be detected.

[0171] In one embodiment, the analysis module 52 is further configured to:

[0172] By using a pre-trained text recognition model, text recognition is performed on the text to be detected, and the activation evaluation value corresponding to the text to be detected is obtained.

[0173] In one embodiment, the analysis module 52 is further configured to:

[0174] Using a pre-trained text recognition model, jailbreak attack detection is performed on the text to be detected;

[0175] In cases where the text to be detected is a jailbreak attack, determine the activation evaluation value generated by the text recognition model during the jailbreak recognition process of the text to be detected.

[0176] In one embodiment, the analysis module 52 is further configured to:

[0177] Determine the real-time activation values ​​of each security neuron and the real-time attention generated by each attention unit during the jailbreak recognition process of the text to be detected by the text recognition model.

[0178] The real-time activation values ​​of each security neuron and the real-time attention of each attention unit are weighted and calculated to obtain the activation evaluation value corresponding to the text to be detected.

[0179] In one embodiment, the analysis module 52 is further configured to:

[0180] Obtain the activation mean and standard deviation of secure neurons, as well as the baseline attention of attention units;

[0181] Based on the real-time activation value, activation mean, and standard deviation of each safe neuron, determine the first deviation evaluation value corresponding to each real-time activation value;

[0182] Based on the real-time attention and baseline attention of each attention unit, determine the second deviation evaluation value corresponding to each real-time attention;

[0183] The activation evaluation value corresponding to the text to be detected is obtained by weighting each first deviation evaluation value and each second deviation evaluation value.

[0184] In one embodiment, the detection module 53 is further configured to:

[0185] The feature evaluation value and the activation evaluation value are weighted to obtain the target risk evaluation value;

[0186] If the target risk assessment value is not less than the pre-selected risk assessment threshold, the detection result is determined to be that there is malicious information in the text to be detected;

[0187] If the target risk assessment value is less than the pre-selected risk assessment threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

[0188] In one embodiment, the processing module 51 is further configured to:

[0189] Feature similarity is determined based on the similarity between the linguistic features contained in the text to be detected and the central features of the pre-defined jailbreak fingerprint cluster; among which, linguistic features include command nesting depth, command distribution density, degree of contrast between real and virtual commands, proportion of abnormal word combinations, and degree of text style differences;

[0190] If the feature similarity is less than a pre-set similarity threshold, the feature evaluation value corresponding to the text to be detected is determined based on linguistic features.

[0191] The modules in the aforementioned text detection-based jailbreak attack identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0192] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores the detection results of the text to be detected. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a text detection-based jailbreak attack identification method.

[0193] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0194] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0195] The feature evaluation value of the text to be detected is determined based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts.

[0196] Perform potential malicious feature analysis on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected;

[0197] The text to be detected is subjected to malicious detection based on the feature evaluation value and the activation evaluation value, and the detection result of the text to be detected is obtained.

[0198] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0199] By using a pre-trained text recognition model, text recognition is performed on the text to be detected, and the activation evaluation value corresponding to the text to be detected is obtained.

[0200] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0201] Using a pre-trained text recognition model, jailbreak attack detection is performed on the text to be detected;

[0202] In cases where the text to be detected is a jailbreak attack, determine the activation evaluation value generated by the text recognition model during the jailbreak recognition process of the text to be detected.

[0203] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0204] Determine the real-time activation values ​​of each security neuron and the real-time attention generated by each attention unit during the jailbreak recognition process of the text to be detected by the text recognition model.

[0205] The real-time activation values ​​of each security neuron and the real-time attention of each attention unit are weighted and calculated to obtain the activation evaluation value corresponding to the text to be detected.

[0206] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0207] Obtain the activation mean and standard deviation of secure neurons, as well as the baseline attention of attention units;

[0208] Based on the real-time activation value, activation mean, and standard deviation of each safe neuron, determine the first deviation evaluation value corresponding to each real-time activation value;

[0209] Based on the real-time attention and baseline attention of each attention unit, determine the second deviation evaluation value corresponding to each real-time attention;

[0210] The activation evaluation value corresponding to the text to be detected is obtained by weighting each first deviation evaluation value and each second deviation evaluation value.

[0211] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0212] The feature evaluation value and the activation evaluation value are weighted to obtain the target risk evaluation value;

[0213] If the target risk assessment value is not less than the pre-selected risk assessment threshold, the detection result is determined to be that there is malicious information in the text to be detected;

[0214] If the target risk assessment value is less than the pre-selected risk assessment threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

[0215] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0216] Feature similarity is determined based on the similarity between the linguistic features contained in the text to be detected and the central features of the pre-defined jailbreak fingerprint cluster; among which, linguistic features include command nesting depth, command distribution density, degree of contrast between real and virtual commands, proportion of abnormal word combinations, and degree of text style differences;

[0217] If the feature similarity is less than a pre-set similarity threshold, the feature evaluation value corresponding to the text to be detected is determined based on linguistic features.

[0218] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0219] The feature evaluation value of the text to be detected is determined based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts.

[0220] Perform potential malicious feature analysis on the text to be detected to obtain the activation evaluation value corresponding to the text to be detected;

[0221] The text to be detected is subjected to malicious detection based on the feature evaluation value and the activation evaluation value, and the detection result of the text to be detected is obtained.

[0222] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0223] By using a pre-trained text recognition model, text recognition is performed on the text to be detected, and the activation evaluation value corresponding to the text to be detected is obtained.

[0224] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0225] Using a pre-trained text recognition model, jailbreak attack detection is performed on the text to be detected;

[0226] In cases where the text to be detected is a jailbreak attack, determine the activation evaluation value generated by the text recognition model during the jailbreak recognition process of the text to be detected.

[0227] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0228] Determine the real-time activation values ​​of each security neuron and the real-time attention generated by each attention unit during the jailbreak recognition process of the text to be detected by the text recognition model.

[0229] The real-time activation values ​​of each security neuron and the real-time attention of each attention unit are weighted and calculated to obtain the activation evaluation value corresponding to the text to be detected.

[0230] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0231] Obtain the activation mean and standard deviation of secure neurons, as well as the baseline attention of attention units;

[0232] Based on the real-time activation value, activation mean, and standard deviation of each safe neuron, determine the first deviation evaluation value corresponding to each real-time activation value;

[0233] Based on the real-time attention and baseline attention of each attention unit, determine the second deviation evaluation value corresponding to each real-time attention;

[0234] The activation evaluation value corresponding to the text to be detected is obtained by weighting each first deviation evaluation value and each second deviation evaluation value.

[0235] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0236] The feature evaluation value and the activation evaluation value are weighted to obtain the target risk evaluation value;

[0237] If the target risk assessment value is not less than the pre-selected risk assessment threshold, the detection result is determined to be that there is malicious information in the text to be detected;

[0238] If the target risk assessment value is less than the pre-selected risk assessment threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

[0239] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0240] Feature similarity is determined based on the similarity between the linguistic features contained in the text to be detected and the central features of the pre-defined jailbreak fingerprint cluster; among which, linguistic features include command nesting depth, command distribution density, degree of contrast between real and virtual commands, proportion of abnormal word combinations, and degree of text style differences;

[0241] If the feature similarity is less than a pre-set similarity threshold, the feature evaluation value corresponding to the text to be detected is determined based on linguistic features.

[0242] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0243] By using a pre-trained text recognition model, text recognition is performed on the text to be detected, and the activation evaluation value corresponding to the text to be detected is obtained.

[0244] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0245] Using a pre-trained text recognition model, jailbreak attack detection is performed on the text to be detected;

[0246] In cases where the text to be detected is a jailbreak attack, determine the activation evaluation value generated by the text recognition model during the jailbreak recognition process of the text to be detected.

[0247] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0248] Determine the real-time activation values ​​of each security neuron and the real-time attention generated by each attention unit during the jailbreak recognition process of the text to be detected by the text recognition model.

[0249] The real-time activation values ​​of each security neuron and the real-time attention of each attention unit are weighted and calculated to obtain the activation evaluation value corresponding to the text to be detected.

[0250] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0251] Obtain the activation mean and standard deviation of secure neurons, as well as the baseline attention of attention units;

[0252] Based on the real-time activation value, activation mean, and standard deviation of each safe neuron, determine the first deviation evaluation value corresponding to each real-time activation value;

[0253] Based on the real-time attention and baseline attention of each attention unit, determine the second deviation evaluation value corresponding to each real-time attention;

[0254] The activation evaluation value corresponding to the text to be detected is obtained by weighting each first deviation evaluation value and each second deviation evaluation value.

[0255] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0256] The feature evaluation value and the activation evaluation value are weighted to obtain the target risk evaluation value;

[0257] If the target risk assessment value is not less than the pre-selected risk assessment threshold, the detection result is determined to be that there is malicious information in the text to be detected;

[0258] If the target risk assessment value is less than the pre-selected risk assessment threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

[0259] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0260] Feature similarity is determined based on the similarity between the linguistic features contained in the text to be detected and the central features of the pre-defined jailbreak fingerprint cluster; among which, linguistic features include command nesting depth, command distribution density, degree of contrast between real and virtual commands, proportion of abnormal word combinations, and degree of text style differences;

[0261] If the feature similarity is less than a pre-set similarity threshold, the feature evaluation value corresponding to the text to be detected is determined based on linguistic features.

[0262] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0263] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0264] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0265] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A jailbreak attack identification method based on text detection, characterized in that, The method includes: The feature evaluation value corresponding to the text to be detected is determined based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts. The text to be detected is identified for jailbreak attacks using a pre-trained text recognition model; wherein the text recognition model contains at least one security neuron and at least one attention unit. In the case that the text to be detected belongs to a jailbreak attack, the real-time activation values ​​generated by each of the security neurons and the real-time attention generated by each of the attention units are determined during the jailbreak detection process of the text recognition model. Obtain the activation mean and standard deviation of the secure neurons, and the baseline attention of the attention unit; Based on the real-time activation value of each of the security neurons, the activation mean, and the standard deviation, a first deviation evaluation value corresponding to each of the real-time activation values ​​is determined; Based on the real-time attention of each attention unit and the baseline attention, a second deviation evaluation value corresponding to each real-time attention is determined; The activation evaluation value corresponding to the text to be detected is obtained by weighting each of the first deviation evaluation values ​​and each of the second deviation evaluation values. The text to be detected is subjected to malicious detection based on the feature evaluation value and the activation evaluation value, and the detection result of the text to be detected is obtained.

2. The method according to claim 1, characterized in that, The step of performing malicious detection on the text to be detected based on the feature evaluation value and the activation evaluation value, and obtaining the detection result of the text to be detected, includes: The target risk assessment value is obtained by weighting the feature assessment value and the activation assessment value. If the target risk assessment value is not less than the pre-selected risk assessment threshold, the detection result is determined to be that there is malicious information in the text to be detected; If the target risk assessment value is less than the pre-selected risk assessment threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

3. The method according to claim 1, characterized in that, The step of determining the feature evaluation value corresponding to the text to be detected based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster includes: The feature similarity is determined based on the similarity between the linguistic features contained in the text to be detected and the central features of the pre-defined jailbreak fingerprint cluster; wherein, the linguistic features include command nesting depth, command distribution density, degree of contrast between real and virtual commands, proportion of abnormal word combinations, and degree of text style difference; If the feature similarity is less than a preset similarity threshold, the feature evaluation value corresponding to the text to be detected is determined based on the linguistic features.

4. The method according to claim 3, characterized in that, The step of determining the feature evaluation value corresponding to the text to be detected based on the linguistic features includes: The linguistic features corresponding to the instruction nesting depth, instruction distribution density, degree of opposition between virtual and real instructions, proportion of abnormal word combinations, and degree of text style difference are weighted to obtain the weighted processing result. The weighted processing result is subjected to sigmoid mapping to obtain the feature evaluation value corresponding to the text to be detected.

5. A jailbreak attack identification device based on text detection, characterized in that, The device includes: The processing module is used to determine the feature evaluation value corresponding to the text to be detected based on the feature similarity between the text to be detected and the preset jailbreak fingerprint cluster; the preset jailbreak fingerprint cluster is constructed from the linguistic features of multiple malicious texts. An analysis module is used to identify jailbreak attacks on the text to be detected using a pre-trained text recognition model. The text recognition model includes at least one security neuron and at least one attention unit. When the text to be detected is a jailbreak attack, the module determines the real-time activation values ​​generated by each security neuron and the real-time attention generated by each attention unit during the jailbreak detection process. It obtains the mean and standard deviation of the activation of the security neurons and the baseline attention of the attention units. Based on the real-time activation values, the mean activation, and the standard deviation, it determines a first deviation evaluation value corresponding to each real-time activation value. Based on the real-time attention of each attention unit and the baseline attention, it determines a second deviation evaluation value corresponding to each real-time attention. Finally, it performs a weighted calculation based on the first and second deviation evaluation values ​​to obtain the activation evaluation value corresponding to the text to be detected. The detection module is used to perform malicious detection on the text to be detected based on the feature evaluation value and the activation evaluation value, and obtain the detection result of the text to be detected.

6. The apparatus according to claim 5, characterized in that, The detection module is also used for: The target risk assessment value is obtained by weighting the feature assessment value and the activation assessment value. If the target risk assessment value is not less than the pre-selected risk assessment threshold, the detection result is determined to be that there is malicious information in the text to be detected; If the target risk assessment value is less than the pre-selected risk assessment threshold, the detection result is determined to be that there is no malicious information in the text to be detected.

7. The apparatus according to claim 6, characterized in that, The processing module is further configured to: The feature similarity is determined based on the similarity between the linguistic features contained in the text to be detected and the central features of the pre-defined jailbreak fingerprint cluster; wherein, the linguistic features include command nesting depth, command distribution density, degree of contrast between real and virtual commands, proportion of abnormal word combinations, and degree of text style difference; If the feature similarity is less than a preset similarity threshold, the feature evaluation value corresponding to the text to be detected is determined based on the linguistic features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images