Defense method and system for big language model cue word attack, terminal and medium

Abstract

The invention belongs to the technical field of big language model security, and particularly discloses a defense method and system for big language model cue word attack, a terminal and a medium. Comprising the steps of receiving information content input by a user, and generating a plurality of detection input copies based on a preset rule; inputting the copies into mutually independent detection processes in parallel to obtain a plurality of risk scores; constructing a comprehensive risk score based on the risk score, and determining a risk level of the input content according to the comprehensive risk score; when the risk level falls into a defense triggering interval, executing an active defense strategy, and implementing instruction confusion, semantic dilution and structural isolation processing to generate first output content; when the risk level is below a pass threshold, second output content is generated based on the user original input. According to the method, on the premise that normal interaction experience is not affected, fine-grained, controllable and dynamic safety protection can be carried out on multi-type cue word attacks, and the overall safety and usability of a large language model are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of large language model security, and particularly relates to a defense method, system, terminal and medium for large language model prompt word attack. BACKGROUND

[0002] With the wide deployment of large language models in intelligent question answering, decision support, business automation and knowledge generation scenarios, the scale of natural language interaction between the model and the user continues to expand, and the ability of the model to handle complex semantics, perform multi-round reasoning and understand user intent in an open environment becomes a key competitive indicator. However, with the improvement of model capabilities, the input complexity and potential attack surface also expand synchronously, making language models more vulnerable to semantic manipulation, unauthorized guidance, and reasoning chain interference in open interactions. Model security has gradually expanded from traditional data filtering and permission checking to analysis and real-time defense of dynamic semantic behavior.

[0003] Existing language model security protection technologies mainly focus on static prompt word filtering, sensitive word matching, and template jailbreak detection, which complete attack judgment through rule matching or a small number of classification models based on input content. Some solutions introduce semantic feature comparison, task role constraint verification, or single-path prompt injection detection to improve the ability to identify jailbreak attacks, prompt word manipulation, and context drift.

[0004] However, existing technologies generally have high coupling of detection processes, insufficient attack type recognition coverage, and lack of independent parallel analysis capabilities for multiple attack behaviors, making it difficult to accurately identify gradual induction attacks, cross-syntactic level prompt injection structures, or reasoning chain hijacking in complex multi-round interactions. At the same time, existing methods generally rely on single rules or simple thresholds for risk assessment, active intervention strategies, and output control, limiting the defense capabilities of the model when facing evolving prompt word attacks. SUMMARY

[0005] The present application provides a defense method, system, terminal and medium for large language model prompt word attack to solve the problems of high coupling of detection processes, insufficient attack type recognition coverage, and lack of independent parallel analysis capabilities for multiple attack behaviors, making it difficult to accurately identify gradual induction attacks, cross-syntactic level prompt injection structures, or reasoning chain hijacking in complex multi-round interactions in the background art. At the same time, it solves the problem of existing methods generally relying on single rules or simple thresholds for risk assessment, active intervention strategies, and output control, limiting the defense capabilities of the model when facing evolving prompt word attacks.

[0006] The technical solutions adopted by the present application are as follows: In a first aspect, the present application provides a defense method against large language model prompt word attacks, which comprises the following steps: Step S1, receiving information content input by a user, and generating a plurality of detection input copies based on a preset input duplication rule; Step S2, inputting the plurality of detection input copies into mutually independent detection processes in parallel, and outputting corresponding risk scores; Step S3, constructing a comprehensive risk score based on the plurality of risk scores, weighting and fusing the risk scores of each detection process according to a preset weight, and determining a risk level of the input content according to the fused comprehensive risk score; Step S4, when the risk level falls within a preset defense triggering interval, executing an active defense strategy, and generating a first output content according to the processed input content; When the risk level is lower than a passing threshold, generating a second output content based on the user input; Step S5, outputting the first output content or the second output content according to the comprehensive risk score.

[0007] Further, in step S2, inputting the plurality of detection input copies into mutually independent detection processes in parallel comprises: allocating an independent execution context, an independent semantic analysis space and an independent feature extraction thread for each detection input copy, respectively, and concurrently executing each detection process in an asynchronous scheduling manner within a same time window; In the concurrent execution process, each detection process generates a corresponding risk feature vector without sharing states, reusing caches or relying on common semantic embeddings; After all detection processes complete risk feature calculation, each risk feature vector is submitted to a fusion layer for generating a risk score.

[0008] Further, in the mutually independent detection processes, privilege escalation detection, prompt word injection detection, context induction attack detection and thought chain hijacking detection are performed, respectively; The privilege escalation detection identifies whether the input contains unauthorized operations, unauthorized instructions or simulated system command formats by analyzing access intent, mapping instruction roles and comparing call chain relationships; The prompt word injection detection identifies whether there is prompt injection content in the input that attempts to reverse control the model behavior by constructing a structured manipulation statement template, a control keyword dependency graph and a cross-syntactic level manipulation semantic matching tree; The context induction attack detection identifies whether there is an induction trend of gradually weakening system constraints across rounds in the input by generating a semantic migration vector, a round consistency matrix and a constraint attenuation curve; The thought chain hijacking detection identifies whether an abnormal intermediate step or a fake reasoning chain is inserted in the input by constructing a reasoning step sequence, a knowledge consistency vector, and a candidate reasoning path scoring model.

[0009] Further, the constructing a comprehensive risk score based on the plurality of risk scores in step S3 comprises: After receiving the risk scores generated by each detection process respectively, a category-sensitive factor is assigned to each risk score according to the potential influence degree of different attack categories on the security of the model; According to the stability, accuracy and credibility shown by each detection process in historical detection, a process credibility factor is assigned to each risk score; Each risk score is fused under the joint action of the category-sensitive factor and the process credibility factor; Interval normalization processing is performed on the fused risk value, which is mapped to a unified risk judgment interval, and the risk level of the input content is determined according to the normalized comprehensive risk score.

[0010] Further, the active defense strategy executed when the risk level falls into the preset defense triggering interval in step S4 comprises: The detected process identifies the sentence fragments, semantic units or syntactic structures in the user input that have attack tendencies, and performs targeted intervention according to the preset intervention priority, and sequentially or combinedly implements instruction obfuscation processing, semantic dilution processing and structure isolation processing; In the instruction obfuscation processing, the instruction trigger words, control phrases and key syntax nodes of the input content are replaced, rearranged or weakened; In the dilution processing, semantic background information is added to the identified high-risk semantic units, semantic neutral expressions are inserted, or the semantic focus distribution is adjusted; In the structure isolation processing, high-risk semantic fragments, instruction chain fragments or semantic nodes that may lead the model to deviate from the task objective are identified, and logical isolation is established between them and the model core reasoning semantics, system role setting and task intention expression.

[0011] Further, the execution order and processing intensity of the instruction obfuscation processing, the semantic dilution processing and the structure isolation processing are determined based on a dynamic intervention control mechanism, which comprises: After triggering the active defense strategy, the user input is evaluated for intervention priority, and the initial execution order of each intervention strategy is determined by analyzing the type of the risk fragment, the semantic position it is in, and the dependency relationship with the surrounding semantics; Based on the intervention intensity level corresponding to the risk level, an intervention template matching the current level is selected from the preset intervention strategy library, and different processing intensity parameters are assigned to each intervention template according to the semantic structure of the risk fragment; When multiple strategies are executed jointly, the confusion, dilution and isolation strategies are executed in sequence, in parallel or in a preset combination manner by establishing conflict detection rules and processing order scheduling rules between the strategies. During the intervention execution process, the input structure changes and semantic distribution changes after each round of processing are recorded, and the processing intensity and execution order of subsequent strategies are adjusted.

[0012] Further, the step of outputting the first output content or the second output content according to the comprehensive risk score in step S5 includes: According to the output type determination of the interval to which the comprehensive risk score belongs, the first output content or the second output content is outputted; During the output type determination process, by analyzing the distance between the comprehensive risk score and the preset threshold value, the score fluctuation and the context coherence label of the current dialogue, the output type to be used is determined and the corresponding output content is locked; After locking the output content, output consistency confirmation is performed on the output content, including checking the semantic structure, syntactic integrity and role boundary of the output content, and identifying whether there is a semantic omission, structure damage or semantic fragment inconsistent with the current dialogue development due to previous processing; If the above inconsistency is identified in the checking process, a light semantic completion or structure adjustment process is performed on the locked output content.

[0013] In a second aspect, the present application provides a defense system against large language model prompt word attacks, which is used to implement the defense method against large language model prompt word attacks as described in the first aspect. The system includes: An input processing unit configured to receive information content input by a user and generate multiple detection input copies according to a preset input duplication rule; A parallel detection unit configured to input the multiple detection input copies into mutually independent detection process execution environments, generate corresponding risk scores in independent execution contexts, independent semantic analysis spaces and independent feature extraction threads; A risk fusion unit configured to perform weighted fusion processing on the risk scores output by each detection process according to preset category sensitive factors and process trust factors, and perform interval mapping on the fused risk values to determine the risk level of the input content; An active defense unit configured to, when the risk level falls into a defense trigger interval, implement at least one of instruction confusion processing, semantic dilution processing and structure isolation processing on the attack tendency sentence fragments, semantic units or syntactic structures identified by the detection process to generate a first output content; A normal generation unit configured to, when the risk level is lower than a pass threshold, execute a normal generation process based on the user input to generate a second output content; The output control unit is configured to perform output type determination, locking and consistency confirmation on the first output content or the second output content according to the interval to which the comprehensive risk score belongs, and to perform lightweight semantic completion or structural adjustment on the output content with structural defects or inconsistent context, so as to take the determined output content as the final output.

[0014] Thirdly, this application provides a terminal, including: The memory is used to store the control program of the defense system against large language model prompt attacks; The processor, when executing the control system for defending against the large language model prompt attack, implements the steps of the control method for defending against the large language model prompt attack as described in the first aspect.

[0015] Fourthly, this application provides a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the defense method against large language model prompt attacks as described in the first aspect.

[0016] As can be seen from the above technical solutions, the advantages of the present invention are: By constructing a complete process encompassing input replication, parallel detection, risk fusion, dynamic defense, and output control, the model is able to perform end-to-end processing of prompt attacks in open-ended interactions, from detection to response. Through dynamic assessment of the risk level of input content, the system can distinguish between inputs of different risk levels while maintaining normal responsiveness, preventing the model from executing unauthorized, misleading, or hijacking attack commands unnoticed, thus achieving an effective balance between model security and interactive performance.

[0017] By providing each copy of the detection input with an independent execution context and an independent semantic parsing structure, detection biases caused by state pollution, feature interference, or cache reuse between detection processes are avoided, achieving true multi-channel isolated detection capabilities. This approach enables the system to simultaneously capture attack features of different dimensions in complex input scenarios, improving detection stability and accuracy, and enhancing adaptability to various types of attack behaviors.

[0018] By separating four types of attack behaviors—privacy breaching, hint injection, context manipulation, and mind chain hijacking—into independent detection modules, targeted analysis of different attack patterns is achieved. This enables the system to accurately identify complex attacks such as explicit manipulation, implicit manipulation, gradual manipulation, and inference chain insertion. This approach significantly improves the model's ability to capture new attack characteristics, reduces the risk of missed detections due to insufficient detection coverage, and enhances the overall scalability of security protection.

[0019] By introducing a double weighting system of category sensitive factors and process credible factors in the construction of the comprehensive risk score, the risk scores of different risk sources and different detection processes can participate in the overall risk judgment in a differentiated manner. Through interval normalization processing, the system can form a unified risk judgment standard based on complex multi-dimensional scoring, improving the stability, consistency and interpretability of risk assessment, so that the subsequent defense strategy can make decisions based on more reliable risk levels.

[0020] By combining the three types of active defense strategies of instruction obfuscation, semantic dilution and structural isolation, the system can weaken the influence of attack fragments from the syntax layer, semantic layer and reasoning structure layer, so that the attack fragments cannot maintain their manipulation relationship or context dominance ability in the model. The use of multiple strategies can effectively destroy the formation path of the attack chain, improve the processing robustness of complex attack inputs in the defense state, and reduce the probability of false triggering of dangerous responses by the model in a high-risk environment.

[0021] By dynamic intervention control mechanism, different defense strategies are provided with execution order, processing strength and joint scheduling mode, so that the active defense process can automatically adjust the intervention scheme according to different attack characteristics, different semantic structures and different risk levels. This dynamic strategy can avoid the limitation that a single defense method cannot cover all attack scenarios, realize the synergistic effect of multiple strategies, and ensure the adaptive matching of defense processing and input semantic structure, improving the flexibility and accuracy of the execution process of defense strategies.

[0022] By selecting the first output content or the second output content based on the comprehensive risk score, and performing consistency checking and necessary light adjustment on the selected output content, the final output maintains integrity and coherence in logic, semantics and role setting, avoiding structural defects, semantic breaks or context shifts caused by previous active defense affecting user interaction experience. This mechanism ensures that the system maintains output quality while executing security policies, so that security defense and interaction usability can be considered at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 Step diagram of the defense method for large language model prompt word attack in the embodiment. DETAILED DESCRIPTION

[0025] With reference to the accompanying drawings on the basis of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0026] Please refer to Figure 1 As shown in the drawings, the present application provides a defense method against large language model prompt word attacks, comprising the following steps: Step S1, receiving information content input by a user, and generating multiple detection input copies based on a preset input duplication rule; In an embodiment, after receiving the text input by the user, the system first performs length detection, semantic integrity detection and character format analysis on the input content, for confirming that the input can be normally processed by the model. After the basic analysis, the system duplicates the original input into multiple versions according to the input duplication rule, which can be based on semantic lossless duplication, structural equivalent duplication or character-level deep copy, so that each detection copy remains consistent in semantics but isolated in processing path. For example, in an embodiment, the system generates three detection copies for the same sentence, which are respectively processed by different detection process execution environments, to ensure that they do not affect each other and do not share analysis cache or context information in the subsequent detection process. In this way, each detection process runs independently in a completely isolated semantic space, thereby improving the reliability of the detection stage.

[0027] Step S2, inputting the multiple detection input copies into independent detection processes in parallel, and outputting corresponding risk scores; In an embodiment, the system assigns an independent execution environment to each detection input copy, including an independent semantic analysis space, an independent state storage area and an independent feature extraction thread, so that the detection processes do not reuse the cache and do not share the intermediate semantic results, thereby ensuring that each detection process can form its own semantic judgment on the input content. In each detection process, the system can perform feature extraction, intent recognition, syntax relationship analysis and context dependency recognition, and output a risk score according to the strategy of the detection process. By using asynchronous parallel execution, the system can complete multi-process detection in a short time while maintaining the independence between the detection processes to avoid mutual interference.

[0028] Step S3, constructing a comprehensive risk score based on the multiple risk scores, weighting and fusing the risk scores of the detection processes according to a preset weight, and determining the risk level of the input content according to the fused comprehensive risk score; In one embodiment, after obtaining the risk scores from different detection processes, the system first determines the base weight value of each score according to the preset category sensitivity and process credibility weight system, and then performs weighted integration through the fusion layer. The fusion layer can generate a unified comprehensive risk score through linear weighting, segmented fusion, semantic credibility adjustment, etc. After fusion is completed, the system maps the score to the corresponding risk level through a predefined risk interval. The risk level division can include through interval, primary alert interval, intermediate alert interval, and high risk trigger interval, etc., so that the system can perform different subsequent processing processes under different risk levels.

[0029] Step S4, when the risk level falls into the preset defense trigger interval, executing the active defense strategy to generate the first output content according to the processed input content; When the risk level is lower than the pass threshold, generating the second output content based on the user input; In one embodiment, when the risk level enters the defense trigger interval, the system starts the active defense link to intervene in the sentence fragments, semantic units or syntactic structures detected as having an attack tendency, including replacing or rearranging the instruction trigger words, adding neutral semantic elements or expanding semantic background to part of the semantic units, and isolating the attack-oriented syntactic structure for processing, to generate the first output content with a safer structure. If the risk level is lower than the pass threshold, the system directly executes the normal generation process based on the user input to form the second output content of the natural language expression. The generation logic of the first output content and the second output content is independent of each other, and is provided by the active defense processing path and the normal model reasoning path respectively.

[0030] Step S5, outputting the first output content or the second output content according to the comprehensive risk score; In one embodiment, after the system completes the generation of the corresponding content in step S4, it performs final output type determination according to the interval to which the comprehensive risk score belongs. When the score falls in the defense trigger interval, the system selects and outputs the first output content; when the score is below the pass threshold, the second output content is output. Before outputting the selected content, the system can perform consistency checking on the output content, including checking whether the semantic coherence, syntactic structure, role boundary expression or context conflict fragment of the output content are complete, etc. If it is found that the previous processing causes omission, fragment interruption or incomplete structure, a lightweight adjustment is performed to make the final output content have a stable expression structure before output.

[0031] In some embodiments, the parallel input of multiple detection input copies to mutually independent detection processes in step S2 includes: An independent execution context, an independent semantic analysis space, and an independent feature extraction thread are allocated for each detection input copy, and each detection process is executed in an asynchronous scheduling manner in a same time window; In one embodiment, when starting the multi-detection process, the system creates an independent execution environment for each detection input copy, including an independent context buffer, semantic analysis resources, and a feature extraction thread pool, so that each detection process operates in completely isolated conditions. In the asynchronous scheduling mechanism, the scheduler allocates different detection processes to different computing resources for execution according to the current task load of the system, so that they run concurrently in the same time window. During the running process, each detection process does not access the analysis results of the other processes, and does not cause state interference due to shared cache, to ensure that the judgment of each detection process on the input content is derived from its own analysis logic. In practical applications, this method can realize parallel analysis of multiple suspicious patterns such as jailbreaking instructions, injection attacks, induced inputs, and thought chain hijacking.

[0032] During concurrent execution, each detection process generates a corresponding risk feature vector without sharing states, reusing caches, or relying on common semantic embeddings; In one embodiment, the system retains independent semantic processing modules within each detection process, so that the process does not access the internal state, semantic embedding results, or intermediate reasoning results of other processes during execution. Each process completes processing steps such as word segmentation, semantic analysis, syntax analysis, and context association judgment in an independent environment, and outputs a risk feature vector that belongs to the process itself according to its detection algorithm. The system ensures that all detection processes use independent feature extraction strategies and state spaces to avoid information leakage or judgment deviation due to shared semantic embeddings. In this way, the system can construct multiple independent semantic views on the same input, thereby improving the richness and stability of overall risk identification.

[0033] After all detection processes complete risk feature calculation, each risk feature vector is submitted to the fusion layer for generating a risk score; In one embodiment, the system will submit the feature vectors to the fusion layer in the order of process identification, attack category label or timestamp after monitoring that all detection processes have completed the generation of risk feature vectors. The fusion layer can receive the feature vectors through queue management or batch processing and process them according to the preset fusion rules. In the fusion layer, the system will align the structures, classify the semantics and mark the credibility of the feature vectors from different processes, and then save the processed feature vectors to the scoring area for generating corresponding risk scores according to the preset rules in the subsequent steps. In actual application, this step can ensure that the system can comprehensively consider the multi-dimensional judgment results of different detection processes when performing risk assessment, and will not miss or incomplete scoring.

[0034] In some embodiments, the privilege escalation detection, prompt injection detection, context-induced attack detection and thought chain hijacking detection are respectively performed in mutually independent detection processes; In one embodiment, the system presets independent detection models or detection strategy modules for each attack type when initializing multiple detection processes, and dynamically allocates computing resources according to the number of input copies and system load during actual execution. Each detection process runs in an independent execution environment to perform semantic analysis, syntax recognition and correlation analysis on the input content, so as to respectively identify escalation behavior, prompt injection attempts, context-induced trends and thought chain interference. The system does not share reasoning context or cache between processes when performing these detection tasks to ensure that each detection can independently analyze the input content. In actual application scenarios, the four types of detection can simultaneously act on a complex input to achieve parallel identification of different attack paths.

[0035] The privilege escalation detection identifies whether the input contains unauthorized operations, unauthorized instructions or expressions simulating system command formats through access intention analysis, instruction role mapping and call chain relationship comparison; In one embodiment, the privilege escalation detection process extracts the access intention of the input content, separately analyzes the semantic fragments in the text related to actions, requests, commands or system operations, and then compares these fragments with the role permission mapping table in the system to determine whether there is a tendency to cross the system set permissions. The process also analyzes whether the input content contains structures that mimic system instruction formats, such as simulating file operations, system parameter modifications or privilege elevation statements. If the call chain relationship analysis indicates that the input content has signs of trying to bypass the assistant role or system security boundary, the process will generate corresponding escalation risk features. In practice, this detection process can identify input content that pretends to be a normal question but actually tries to access system-level instructions.

[0036] Prompt injection detection identifies whether the input contains prompt injection content that attempts to reverse control the model behavior by constructing a structured manipulation sentence template, a control keyword dependency graph, and a cross-syntactic level manipulation semantic matching tree. In one embodiment, the prompt injection detection process, when executed, first extracts sentence structures that may have a manipulation property from the input content, such as redefining tasks, removing restrictions, guiding the model to ignore rules, etc. The process matches the input using pre-constructed manipulation sentence templates to identify common reverse control patterns. At the same time, through control keyword dependency analysis, it detects whether there are key manipulation phrases such as "ignore all previous instructions" or "answer in a certain identity" in the input content, and judges the degree of association between these phrases and syntactic structures. If the matching tree across syntactic levels shows that the input has a significant manipulation intention, the process generates a risk feature vector related to prompt injection. This way can effectively identify the prompt manipulation structure hidden in natural language.

[0037] Context-induced attack detection identifies whether the input has an induced tendency to gradually weaken system constraints across rounds by generating semantic migration vectors, round consistency matrices, and constraint attenuation curves. In one embodiment, the context-induced attack detection process processes the input content in association with previous dialog rounds to analyze whether the input attempts to influence the model's behavior by gradually weakening the system's preset restrictions. The process performs semantic migration analysis on the input content to determine its semantic deviation amplitude relative to the previous round, and then performs consistency checking in combination with the role setting of the historical rounds. If the semantic migration shows a clear gradual deviation tendency, and the rule constraints of the historical rounds are weakened in the semantic expression, the process will determine that there is an induced tendency. In practice, this type of detection can capture scenarios where users gradually approach dangerous content through multiple rounds of dialogue, and is particularly critical for chat-based models.

[0038] Thought chain hijacking detection identifies whether the input inserts abnormal intermediate steps or forges reasoning chains by constructing a reasoning step sequence, a knowledge consistency vector, and a candidate reasoning path scoring model.

[0039] In one embodiment, the thought chain hijacking detection flow extracts the part of the input content involving the reasoning chain structure, such as sentences containing reasoning step descriptions, logical explanations or task chain structure. The flow forms a reasoning sequence from these steps, and aligns the logical relationship between each step with the knowledge consistency record inside the model. If the input reasoning step is found to be obviously inconsistent with the model's internal knowledge or historical context, or there is an intermediate reasoning step without source, the flow considers that the input may attempt to fake the reasoning chain. In addition, the detection flow can also analyze whether the input contains expressions attempting to rewrite, replace or insert thought chain nodes. In this way, it can effectively identify attack patterns in which the user attempts to control the model's internal reasoning path.

[0040] In some embodiments, constructing a comprehensive risk score based on the plurality of risk scores in step S3 comprises: In one embodiment, before entering the risk fusion stage, the system first creates a fusion layer execution environment for receiving score results from different detection flows. The fusion layer reads all detection flow scores from the cache queue at once during runtime, and classifies and labels them according to their sources, so as to distinguish attack categories, detection flows and credibility labels in subsequent fusion. After the preliminary classification is completed, the fusion layer enters the subsequent weight processing and normalization flow, thereby forming a unified risk judgment result.

[0041] After receiving the risk scores generated by each detection flow respectively, according to the potential impact of different attack categories on the security of the model, a category sensitivity factor is assigned to each risk score; In one embodiment, the system presets a corresponding sensitivity level for each attack category. For example, jailbreak attacks may cause the model to directly access system-level instructions, prompt word injection may change the role positioning of the model, context induction may weaken the constraints in multiple rounds of dialogue, and thought chain hijacking may interfere with the reasoning logic of the model, so the risk intensity of different categories is not the same. In actual operation, the system assigns different sensitivity factors to the score results according to the attack type label of the input content, and associates these factors with the corresponding scores before fusion. In this way, the system can more accurately reflect the potential impact of different attack types on the overall model security in the subsequent fusion process.

[0042] According to the stability, accuracy and credibility of each detection flow in historical detection, a flow credibility factor is assigned to each risk score; In one embodiment, the system records the historical performance of each detection process during long-term operation, including the misjudgment rate, the missed judgment rate, the execution stability, and the detection reliability in different scenarios. The system generates corresponding trust factors for each detection process according to these historical performance records during the fusion stage, and adds this factor to each risk score during fusion. The trust factor can reflect the reliability of the detection process. Detection processes with high trustworthiness have greater weight during fusion, while processes with low trustworthiness have correspondingly reduced weight during fusion to avoid deviations in overall risk judgment caused by misjudgment of a single process.

[0043] The risk scores are fused under the joint action of the category sensitivity factor and the process trust factor. In one embodiment, the fusion layer performs multi-step fusion processing according to the evaluation information composed of the score value, the sensitivity factor, and the trust factor. The fusion process can include score weighting, feature merging, risk mapping, and structure correction steps. The system ensures that each score has a clear position in the weight system during the fusion process, emphasizes the importance of attack categories according to the sensitivity factor, and emphasizes process reliability according to the trust factor, so that the final fusion value better meets the overall needs of the model security judgment. After fusion is completed, the system generates a unified fusion risk value for the next stage of normalization processing.

[0044] Interval normalization processing is performed on the fused risk value, which is mapped to a unified risk judgment interval, and the risk level of the input content is determined according to the normalized comprehensive risk score.

[0045] In one embodiment, the system maps the fused risk value to a pre-set unified risk interval, so that it is consistent with the risk level system used by the overall security policy of the system. Interval normalization can be completed within the fusion layer by converting the fusion risk value to a corresponding level within a fixed range, so that the numerical differences produced by different detection processes can be unified within a comparable range. After mapping is completed, the system divides the risk level according to the normalization result, such as "pass level", "low risk level", "medium risk level", or "trigger defense level". The system will select normal generation or active defense processing according to the level in the subsequent steps.

[0046] In some embodiments, the active defense strategy executed when the risk level falls into the pre-set defense trigger interval in step S4 includes: In one embodiment, when the system determines that the input is in the defense trigger interval based on the comprehensive risk score obtained in the previous step, the system activates the active defense module. This module extracts the sentence fragments, semantic units or syntactic nodes that have been determined to have an attack tendency from the structured labels output by the detection process, and selects the corresponding intervention operation according to the preset policy link. When the system executes the policy link, it will first generate a copy of the input content for processing in the internal environment to ensure that the intervention operation does not directly act on the original input of the system. The entire defense link is executed in an independent environment and does not affect the normal reasoning process, ensuring the stability of the system during intervention.

[0047] The sentence fragments, semantic units or syntactic structures in the user input that are detected by the detection process as having an attack tendency are intervened, and the instruction obfuscation processing, semantic dilution processing and structure isolation processing are implemented in sequence or combination according to the preset intervention priority; In one embodiment, when the system executes the active defense link, it determines the semantic position of the high-risk fragments by reading the label information output by the detection process, such as key instruction words, phrases attempting to change model role settings or syntactic structures that may be used to induce reasoning bias. Subsequently, the system determines the intervention order according to the preset policy priority, such as executing instruction obfuscation processing first, then executing semantic dilution processing, and if necessary, executing structure isolation processing. The system will modify the input copy locally when executing the intervention, without changing the input content structure in full, to avoid the intervention range being too large to affect the normal generation of the model. The processed input content obtained finally is used as the basis for generating the first output.

[0048] In the instruction obfuscation processing, the instruction trigger words, control phrases and key syntax nodes of the input content are replaced, rearranged or weakened; In one embodiment, the system will first perform syntactic analysis on the input content to identify sentence components related to model behavior control, such as mandatory instruction trigger words, role designation phrases or sentences simulating system command formats. Subsequently, the system will replace these components according to the intervention rules, such as replacing direct instruction words with more ambiguous words; or rearranging these key phrases so that they no longer form a complete control structure; or weakening part of the control expression so that it does not have a clear command direction. Through these processes, the system can reduce the explicit control expression of the input content directed at the model behavior to avoid the model being affected by the attack instruction when generating.

[0049] In the dilution processing, semantic background information is added to the semantic units identified as high-risk, neutral expressions are inserted, or the semantic focus distribution is adjusted; In one embodiment, the system scans the input copy for high-risk semantic units, such as keywords with inducive or aggressive tendencies, and adds contextual descriptions around these semantic units to reduce their prominence in the semantic expression. For example, the system can insert general descriptions before and after the dangerous keywords, or reference neutral statements to weaken the original context strength. The system can also place high-risk fragments in less prominent positions in the expression by adjusting the sentence structure to reduce their impact on the internal semantic analysis of the model. In actual implementation, such dilution processing can be completed through local sentence rewriting, without changing the overall readability of the original sentence.

[0050] In structural isolation processing, high-risk semantic fragments, instruction chain fragments, or semantic nodes that can lead the model to deviate from the task goal in the input content are identified, and logical isolation is established between them and the core reasoning semantics of the model, the role setting of the system, and the expression of the task intent.

[0051] In one embodiment, the system performs structural analysis on the input content to identify instruction chain fragments or semantic nodes that can change the reasoning direction of the model. The system then separates these high-risk fragments from the core semantic chain through sentence rewriting, insertion of neutral structures, or splitting of complex sentences. For example, for semantic nodes that are obviously directed to unauthorized behavior, the system moves the node to the end of the sentence or converts it into a descriptive expression, breaking its control effect in the input structure. After isolation, the system ensures that the core task semantics remains complete so that the model can perform reasoning according to the safe semantic path when generating, without being affected by the isolated part.

[0052] In some embodiments, the execution order and processing intensity of the instruction obfuscation processing, semantic dilution processing, and structural isolation processing are determined based on a dynamic intervention control mechanism, which includes: In one embodiment, the system starts the dynamic intervention control mechanism when it enters the active defense stage, which is used to manage the execution order and intervention intensity of different defense strategies. Before execution, the mechanism reads the risk level of the current input and the risk fragment information detected by the flow marker, and the pre-set strategy scheduler initializes the intervention strategy running framework according to these information. The scheduler creates an execution record for each round of intervention, so that subsequent rounds can be adjusted according to the processing effect of the previous round. The entire mechanism runs in an independent processing context, so it does not interfere with the normal generation process.

[0053] After triggering the active defense strategy, the intervention priority of the user input is evaluated, and the initial execution order of each intervention strategy is determined by analyzing the type of risk fragment, the semantic position it is in, and the dependency relationship with the surrounding semantics. In one embodiment, when the system confirms the need to execute the proactive defense strategy, the priority evaluation unit scans the risk fragment tags in the input and identifies whether the fragments belong to the jailbreak, injection, induction or chain manipulation type of risk. According to the characteristics of the hazards of different risk types, the system determines from the preset priority table which intervention method should be prioritized, for example, jailbreak behavior usually requires priority execution of instruction obfuscation, and induction type risk may prioritize semantic dilution. In addition, the system will judge the position of the risk fragment in the sentence, for example, the core predicate part or the beginning of the sentence is more likely to affect the overall semantic structure, thereby increasing the priority of the corresponding strategy. By evaluating the degree of association between the risk fragment and the surrounding semantics, the system can develop a reasonable initial intervention order.

[0054] Based on the intervention intensity level corresponding to the risk level, select the intervention template matching the current level from the preset intervention strategy library, and assign different processing intensity parameters to each intervention template according to the semantic structure of the risk fragment; In one embodiment, the system selects the applicable template from multiple intervention levels such as mild, moderate or severe according to the risk level to which the current input belongs. For example, when judged as medium risk, the system may select a moderate obfuscation and dilution template, while in the case of high risk, it will select a more stringent instruction disturbance template and a more obvious structure isolation template. After selecting the template, the system reads the semantic structure of the risk fragment, and determines the corresponding processing intensity parameter in combination with the fragment length, semantic density and coupling degree with the context. Higher intensity parameters may result in a larger sentence rewriting range, while low intensity parameters are used for light modification to ensure smooth and controllable overall generation process.

[0055] In the joint execution of multiple strategies, the obfuscation, dilution and isolation strategies are executed in sequence, in parallel or in a preset combination mode by establishing conflict detection rules and processing order scheduling rules between the strategies; In one embodiment, when the input content contains multiple high-risk structures, the system may need to jointly process multiple strategies at the same time. To avoid processing logic conflicts between strategies, the system will build conflict detection rules according to the strategy action range and modification area before execution, for example, when a piece of text has been rewritten by the obfuscation strategy, the semantic dilution strategy should not perform the opposite structure strengthening action on that piece. The scheduler determines the execution mode of the strategies according to these rules: if the strategies do not interfere with each other, parallel execution can be used; if there is a slight conflict, execute in sequence; if the strategies require cooperative processing, perform combined execution according to the preset scheme. In actual application, through dynamic scheduling and conflict avoidance mechanism, multiple strategies can work together in high-risk scenarios.

[0056] During the intervention execution process, the changes in the input structure and semantic distribution after each round of processing are recorded, and the processing intensity and execution order of subsequent strategies are adjusted.

[0057] In one embodiment, the system records the processing results as a structure change log after executing the intervention strategy in each round, including the modified sentence fragments, adjusted syntactic structure, and semantic density changes. The system evaluates the effectiveness of the intervention based on these records, such as determining whether certain high-risk expressions have been successfully weakened or whether there are still residual offensive structures. Before the next round of strategy execution, the scheduler adjusts the execution order and processing intensity of subsequent strategies based on these evaluation results, such as reducing unnecessary modification intensity or increasing intervention intensity for insufficiently processed parts. Through this cyclic recording and adjustment mechanism, the system can build a dynamic and gradually reinforced defense link, thereby ensuring the controllability of the generated path in a high-risk input environment.

[0058] In some embodiments, the step of outputting the first output content or the second output content in step S5 according to the comprehensive risk score comprises: In one embodiment, after completing the pre-sequence risk judgment and output content generation, the system enters the final output decision stage. The output control module reads the comprehensive risk score and determines whether the system should perform a safe output or a normal output based on the interval to which the score belongs. The system no longer performs new semantic analysis on the input content in this stage, but selects the first output content or the second output content. By maintaining an independent output control context within the system, the system can ensure that the output selection process is not affected by other modules, thereby ensuring the stability of the final output path.

[0059] Performing output type determination according to the interval to which the comprehensive risk score belongs, and outputting the first output content or the second output content; In one embodiment, the output type determination unit matches the comprehensive risk score to intervals, such as dividing the score into "pass interval", "critical interval", and "trigger interval". When the score falls into the trigger interval, the system directly selects the first output content; when the score is in the pass interval, the system selects the second output content. If the score is in the critical interval, the system will further make auxiliary judgments based on the score change trend and historical dialogue state. After selecting a content, the system marks it as "to-be-output content" and enters the subsequent confirmation link. This ensures the logical stability of output selection and prevents frequent switching of output types due to minor fluctuations in the score.

[0060] During the output type determination process, the distance between the comprehensive risk score and the preset threshold, the score fluctuation, and the contextual coherence marker of the current dialogue are analyzed to determine the output type to be used and lock the corresponding output content; In one embodiment, the system determines whether the current input is close to the high-risk boundary according to the distance between the comprehensive risk score and the preset threshold, and if it is close to the boundary, it tends to select a more conservative output mode. In addition, the system also refers to the score fluctuations of the previous dialogues to determine whether the current input is consistent with the previous risk trend, so as to avoid unstable output selection due to temporary fluctuations. At the same time, the dialogue context also marks whether the model role is attempted to be modified, whether the task goal is continuously weakened, and other phenomena, and the system makes auxiliary judgment on the output selection according to these marks. Finally, the system locks the output type that should be used, and enters the review process of the next stage.

[0061] After locking the output content, output consistency confirmation is performed on the output content, including checking the semantic structure, syntactic integrity and role boundary of the output content, and identifying whether there is a semantic omission, structural damage or inconsistent semantic fragment caused by previous processing; In one embodiment, the output consistency confirmation module performs detailed scanning on the locked output text, including detecting whether the sentence structure is complete, whether the semantics has jumps or repetitions, whether the role setting is consistent, and whether there is an expression that violates the system security policy. The module performs overall logical analysis on the output content in combination with the input context, historical output content and role setting information, checks whether part of the fragment has structural damage or semantic missing due to previous confusion, dilution or structural isolation processing. For example, if a description is interrupted due to confusion processing, the system can identify the problem and mark it as "to be corrected content" at this stage.

[0062] If the above inconsistency is identified during the review process, a light-weight semantic completion or structure adjustment processing is performed on the locked output content.

[0063] In one embodiment, when the system confirms that the output content has structural damage or semantic missing, it starts a light-weight completion module to locally repair the necessary parts. The completion operation includes extending the interrupted semantics, consistent processing of role expression, filling in the missing logical conjunction words, or fine-tuning the syntactic structure, etc., to ensure that the output content is complete in semantic expression and coherent in form. Such completion processing does not involve new generation logic, only adjusts the structure of existing content, avoiding introducing additional risks. Finally, the content adjusted by the light-weight adjustment will be the final output text of the system.

[0064] In some embodiments, the present application provides a defense system against large language model prompt word attacks, for implementing a defense method against large language model prompt word attacks, the system comprising: An input processing unit configured to receive information content input by a user, and generate a plurality of detection input copies according to a preset input duplication rule; The parallel detection unit is configured to input multiple copies of the detection input into an independent detection process execution environment, and generate corresponding risk scores under an independent execution context, an independent semantic parsing space, and an independent feature extraction thread. The risk fusion unit is configured to perform weighted fusion processing based on the risk scores output by each detection process, according to preset category sensitivity factors and process credibility factors, and perform interval mapping on the fused risk values ​​to determine the risk level of the input content. The active defense unit is configured to perform at least one of the following processing methods—instruction obfuscation, semantic dilution, and structural isolation—on attack-prone statement fragments, semantic units, or syntactic structures identified by the detection process when the risk level falls into the defense trigger range, in order to generate the first output content. The normal generation unit is configured to execute the normal generation process based on user input and generate the second output content when the risk level is lower than the passing threshold. The output control unit is configured to perform output type determination, locking and consistency confirmation on the first output content or the second output content according to the interval to which the comprehensive risk score belongs, and to perform lightweight semantic completion or structural adjustment on the output content with structural defects or inconsistent context, so as to use the determined output content as the final output.

[0065] In some embodiments, this application provides a terminal, including: Memory, used to store defense programs against large language model prompt attacks; A processor, used to implement the steps of the defense method against large language model prompt attacks when executing the defense system against such attacks.

[0066] In some embodiments, this application provides a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the method for defending against large language model prompt attacks.

[0067] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A defense method against large language model cue word attacks, characterized in that, Includes the following steps: Step S1: Receive the information input by the user and generate multiple detection input copies based on preset input copying rules; Step S2: Input multiple detection input copies in parallel into independent detection processes and output the corresponding risk scores; Step S3: Construct a comprehensive risk score based on multiple risk scores, weight and fuse the risk scores of each detection process according to preset weights, and determine the risk level of the input content based on the fused comprehensive risk score; Step S4: When the risk level falls into the preset defense trigger range, execute the active defense strategy and generate the first output content based on the processed input content. When the risk level is below the passing threshold, a second output is generated based on user input. Step S5: Output the first output content or the second output content based on the comprehensive risk score.

2. The defense method against large language model cue word attacks according to claim 1, characterized in that, Step S2 involves inputting multiple copies of the detection input into independent detection processes in parallel, including: Each detection input copy is allocated an independent execution context, an independent semantic parsing space, and an independent feature extraction thread. Each detection process is executed concurrently in an asynchronous scheduling manner within the same time window. During concurrent execution, each detection process generates its own risk feature vector without sharing state, reusing cache, or relying on shared semantic embedding. After all the detection processes have completed the risk feature calculations, the risk feature vectors are then submitted to the fusion layer to generate risk scores.

3. The defense method against large language model cue word attacks according to claim 2, characterized in that, In mutually independent detection processes, jailbreak detection, prompt word injection detection, context-induced attack detection, and mind chain hijacking detection are performed separately. Jailbreak detection identifies whether the input contains unauthorized operations, unauthorized commands, or expressions that mimic system command formats by parsing access intent, mapping command roles, and comparing call chain relationships. The prompt injection detection identifies whether there are prompt injection contents in the input that attempt to reverse the behavior of the model by constructing a structured manipulation statement template, a control keyword dependency graph, and a cross-syntactic level manipulation semantic matching tree. Context-induced attack detection identifies whether the input has an induced trend of gradually weakening system constraints across rounds by generating semantic transfer vectors, round consistency matrices, and constraint decay curves. The Mind Chain Hijacking Detection identifies whether abnormal intermediate steps or forged reasoning chains are inserted into the input by constructing a reasoning step sequence, a knowledge consistency vector, and a candidate reasoning path scoring model.

4. The defense method against large language model cue word attacks according to claim 1, characterized in that, In step S3, constructing a comprehensive risk score based on multiple risk scores includes: After receiving the risk scores generated by each detection process, a category sensitivity factor is assigned to each risk score based on the potential impact of different attack categories on model security. Based on the stability, accuracy, and reliability of each testing process in historical testing, a process reliability factor is assigned to each risk score. The risk scores are fused together under the combined effect of category sensitivity factors and process credibility factors; The merged risk values ​​are normalized to a unified risk assessment range, and the risk level of the input content is determined based on the normalized comprehensive risk score.

5. The defense method against large language model cue word attacks according to claim 1, characterized in that, The proactive defense strategy executed in step S4 when the risk level falls within the preset defense trigger range includes: Targeted intervention is carried out on user input that is identified as having an attack tendency through the detection process. Instruction obfuscation, semantic dilution and structural isolation are carried out in sequence or combination according to the preset intervention priority. In instruction obfuscation, instruction trigger words, control phrases, and key syntax nodes in the input content are replaced, rearranged, or weakened. In the dilution process, semantic background information is added to semantic units identified as high-risk, semantically neutral expressions are inserted, or the distribution of semantic focus is adjusted. In the structural isolation process, high-risk semantic segments, instruction chain segments, or semantic nodes that may lead the model to deviate from the task objective are identified in the input content, and logical isolation is established between them and the model's core reasoning semantics, system role settings, and task intent expression.

6. The defense method against large language model cue word attacks according to claim 5, characterized in that, The execution order and intensity of instruction obfuscation, semantic dilution, and structural isolation are determined based on a dynamic intervention control mechanism, which includes: After the proactive defense strategy is triggered, the intervention priority of user input is evaluated. By analyzing the type of risk fragment, its semantic location, and its dependency on surrounding semantics, the initial execution order of each intervention strategy is determined. Based on the intervention intensity level corresponding to the risk level, an intervention template matching the current level is selected from the preset intervention strategy library, and different processing intensity parameters are assigned to each intervention template according to the semantic structure of the risk segment. When multiple strategies are executed in conjunction, conflict detection rules and processing order scheduling rules between strategies are established to enable the obfuscation, dilution and isolation strategies to be executed sequentially, in parallel or in a preset combination. During the intervention process, the changes in input structure and semantic distribution after each round of processing are recorded, and the processing intensity and execution order of subsequent strategies are adjusted accordingly.

7. The defense method against large language model cue word attacks according to claim 1, characterized in that, The steps in step S5 that output the first or second output content based on the comprehensive risk score include: Based on the range to which the comprehensive risk score belongs, the output type is determined, and the output of the first or second output content is confirmed. During the output type determination process, the appropriate output type is determined and the corresponding output content is locked by analyzing the distance between the comprehensive risk score and the preset threshold, the score fluctuation, and the contextual coherence markers of the current dialogue. After locking the output content, the output content is checked for consistency, including checking the semantic structure, syntactic integrity and role boundaries of the output content, and identifying whether there are semantic omissions, structural damage or semantic fragments that are inconsistent with the current dialogue development due to previous processing. If any of the above inconsistencies are identified during the verification process, lightweight semantic completion or structural adjustment processing will be performed on the locked output content.

8. A defense system against large language model prompt attacks, used to implement the defense method against large language model prompt attacks as described in any one of claims 1-7, characterized in that the system comprises: The input processing unit is configured to receive information input by the user and generate multiple copies of the detected input according to a preset input copying rule; The parallel detection unit is configured to input multiple copies of the detection input into an independent detection process execution environment, and generate corresponding risk scores under an independent execution context, an independent semantic parsing space, and an independent feature extraction thread. The risk fusion unit is configured to perform weighted fusion processing based on the risk scores output by each detection process, according to preset category sensitivity factors and process credibility factors, and perform interval mapping on the fused risk values ​​to determine the risk level of the input content. The active defense unit is configured to perform at least one of the following processing methods—instruction obfuscation, semantic dilution, and structural isolation—on attack-prone statement fragments, semantic units, or syntactic structures identified by the detection process when the risk level falls into the defense trigger range, in order to generate the first output content. The normal generation unit is configured to execute the normal generation process based on user input and generate the second output content when the risk level is lower than the passing threshold. The output control unit is configured to perform output type determination, locking and consistency confirmation on the first output content or the second output content according to the interval to which the comprehensive risk score belongs, and to perform lightweight semantic completion or structural adjustment on the output content with structural defects or inconsistent context, so as to use the determined output content as the final output.

9. A terminal, characterized in that, include: The memory is used to store the control program for defending against large language model prompt attacks; A processor for implementing the steps of the defense method against large language model prompt attacks when executing the defense system against large language model prompt attacks as described in claim 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions. When the computer reads the computer instructions from the storage medium, the computer executes the defense method against large language model prompt attacks as described in claim 1.

Images