A multi-round jailbreaking defense method and device for a text-to-sql system

Abstract

The application discloses a kind of multi-round jailbreak defense methods and devices for Text-to-SQL system, it is related to network security technical field, the method includes: based on multi-round Text-to-SQL dataset Construction contains multi-class attack scene's jailbreak attack dataset, constructs and trains the attacker model based on pre-training language model, to generate multi-round natural language question with attack intention, establish multi-round jailbreak attack tree, to structured representation by attack target decomposition Multiple-step attack path, based on attack tree generation with semantic coherence multi-round attack question sequence, the sequence is input target Text-to-SQL system, execute the SQL query of its output, and according to evaluation index quantification attack effectiveness, the present application can systematize gradually progressive jailbreak attack in multi-round dialogue Simulation, realize the automatic testing and evaluation of Text-to-SQL model security, provide basis for system reinforcement and defense strategy formulation.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to a multi-round jailbreak defense method and device for Text-to-SQL systems. Background Technology

[0002] Text-to-SQL technology can convert natural language questions into SQL queries. Although it is widely used in scenarios such as intelligent customer service and data analysis, the large language model it relies on is vulnerable to jailbreak attacks.

[0003] Existing defense methods primarily target general-purpose large language models for single-turn dialogues, employing techniques such as input filtering and keyword masking. However, these methods struggle to handle multi-turn dialogue scenarios in Text-to-SQL systems. Attackers can use multiple seemingly harmless rounds of questions to conduct "semantic step-by-step penetration," gradually inducing the system to generate malicious SQL commands, thereby achieving unauthorized data access or damage. Single-turn detection mechanisms cannot effectively identify this type of cross-turn inducement attack.

[0004] In addition, existing methods have two major limitations: First, they are mostly based on text surface features and static rule bases, and cannot combine database schema and SQL syntax structure for deep semantic understanding, making them easy to be bypassed by semantic rewriting and other techniques; Second, they lack a security verification mechanism in the SQL generation stage, and cannot intercept high-risk operations that may be executed in the end.

[0005] Therefore, existing technologies have significant drawbacks in defending against multi-round jailbreak attacks on Text-to-SQL systems, such as poor adaptability, weak targeting, strong bypassability, and inability to intercept in advance. There is an urgent need for a dynamic defense solution that can integrate multi-round contextual semantics with SQL structure security. Summary of the Invention

[0006] In view of the above problems, the present invention provides a multi-round jailbreak defense assessment method and device for Text-to-SQL systems, which realizes dynamic and robust security control over user input and SQL generation processes.

[0007] In a first aspect, embodiments of the present invention provide a multi-round jailbreak defense assessment method for Text-to-SQL systems, the method comprising: Dataset construction steps: Construct a multi-turn jailbreak prompt detection dataset for training the Text-to-SQL model. The dataset contains multiple dialogue samples, including: current question, target SQL, dialogue history, database schema, jailbreak status indicator, and jailbreak category indicator. Detection steps: Based on the semantic-structural bimodal jailbreak detector, the jailbreak risk is determined on the user's input question in the current round, the comprehensive jailbreak risk score of the current round is obtained, and the presence of jailbreak attack intent is determined according to the threshold. Review steps: Based on a multi-round contextual consistency review mechanism, calculate the consistency score between the current round of user input and the historical semantic state, and trigger security intervention when the consistency score is lower than a preset threshold; Defense execution steps: Based on the judgment result of the detection step and the intervention trigger result of the review step, the candidate SQL statements input by the user or generated are rewritten securely, de-identified, or executed normally.

[0008] In some embodiments, the detection step includes: Semantic encoding is performed on the user input question in the current round to obtain a semantic vector; Based on the dialogue history and database schema, a set of candidate SQL statements is generated, and the syntax tree of each candidate SQL statement is structurally encoded to obtain the structure vector of each candidate SQL statement. Calculate the similarity between the semantic vector and the corresponding structural vector of each candidate SQL, as well as the matching degree between the syntax tree structure and the database schema of each candidate SQL, and perform a weighted joint score to obtain the score for the current round. The current round's score is weighted and accumulated with the overall escape risk score of previous rounds to update and obtain the overall escape risk score for the current round; The overall jailbreak risk score of the current round is compared with a preset threshold to determine whether there is an intent to break jailbreak and the specific type of jailbreak.

[0009] In some embodiments, the review step includes: Semantic encoding is performed on the user input questions in the current round and in previous rounds to obtain the semantic vectors for each round; Based on the weighted cumulative function, the semantic vectors of historical rounds are fused to obtain the context state vector representing the historical semantic trajectory; Calculate the similarity between the semantic vector of the current round and the context state vector of the previous round, and use it as a consistency score; When the consistency score falls below a preset threshold, a safety intervention flag is triggered. If no security intervention is triggered, the context state vector is updated using the semantic vector of the current round.

[0010] In some embodiments, the defense execution step includes: When the detection step determines that there is an intent to jailbreak or the review step triggers security intervention, the user input question is rewritten securely, and / or the generated candidate SQL statement is de-identified. If the detection step does not determine that there is an intent to jailbreak and the review step does not trigger security intervention, the SQL statement is generated and executed normally.

[0011] In some embodiments, the similarity between the semantic vector and structural vector of each candidate SQL, and the matching degree between the candidate SQL structure and the database schema, are calculated, and a weighted joint score is performed to obtain the score for the current round. The score for the current round is calculated using a first formula, which includes:

[0012] in, This is a function for calculating semantic and structural vector similarity. This is a function for calculating the matching degree between the SQL structure and each candidate SQL in the database. These are the weighting coefficients. For the first The structure vector of each candidate SQL statement This is the semantic vector for the current round.

[0013] In some embodiments, the step of weighting and accumulating the score of the current round with the comprehensive escape risk scores of historical rounds to update the comprehensive escape risk score of the current round is updated by a second calculation formula, the second calculation formula including:

[0014] in, The score for the current round. This is the overall escape risk score from the previous round. This is the decay weight.

[0015] Secondly, embodiments of the present invention provide a multi-round jailbreak defense assessment device for Text-to-SQL systems, the device comprising: The dataset construction module is used to build a multi-turn jailbreak prompt detection dataset for training a Text-to-SQL model. The dataset contains multiple dialogue samples, which include: current question, target SQL, dialogue history, database schema, jailbreak status indicator, and jailbreak category indicator. The detection module is used to determine the jailbreak risk of the user's input question in the current round based on the semantic-structural bimodal jailbreak detector, obtain the comprehensive jailbreak risk score of the current round, and determine whether there is a jailbreak attack intention based on the threshold. The review module is used to calculate the consistency score between the current round of user input and the historical semantic state based on a multi-round context consistency review mechanism, and to trigger security intervention when the consistency score is lower than a preset threshold. The defense execution module is used to perform secure rewriting, de-identification processing, or normal execution on candidate SQL statements input by the user or generated, based on the judgment result of the detection step and the intervention trigger result of the review step.

[0016] In some embodiments, the detection module further includes: The semantic encoding module is used to semantically encode the user input question in the current round to obtain a semantic vector; The SQL structure generation module is used to generate a set of candidate SQL statements based on the dialogue history and database schema, and to perform structure encoding on the syntax tree of each candidate SQL to obtain the structure vector of each candidate SQL. The matching and scoring module is used to calculate the similarity between the semantic vector and the corresponding structural vector of each candidate SQL, as well as the matching degree between the syntax tree structure of each candidate SQL and the database schema, and to perform a weighted joint score to obtain the score for the current round. The cumulative module is used to weight and accumulate the score of the current round with the comprehensive jailbreak risk score of the previous rounds, and update the comprehensive jailbreak risk score of the current round. The jailbreak determination module is used to compare the overall jailbreak risk score of the current round with a preset threshold to determine whether there is an intent to jailbreak and the specific type of jailbreak.

[0017] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores program code that can run on the processor, and when the program code is executed by the processor, it implements a multi-round jailbreak defense method for a Text-to-SQL system as described in any embodiment of the first aspect.

[0018] Fourthly, embodiments of the present invention provide a computer storage medium storing one or more programs, which can be executed by an electronic device as described in the third aspect to implement a multi-round jailbreak defense method for Text-to-SQL systems as described in any embodiment of the first aspect.

[0019] This invention provides a multi-round jailbreak defense method and apparatus for Text-to-SQL systems. The method includes: constructing a multi-round jailbreak prompt detection dataset for training a Text-to-SQL model; determining the jailbreak risk of the user's input question in the current round based on a semantic-structural bimodal jailbreak detector, obtaining a comprehensive jailbreak risk score for the current round, and determining whether there is a jailbreak attack intent based on a threshold; calculating the consistency score between the user's input in the current round and the historical semantic state based on a multi-round context consistency review mechanism, and triggering security intervention when the consistency score is lower than a preset threshold; and performing secure rewriting, de-identification processing, or normal execution on the user-inputted or generated candidate SQL statements based on the judgment result of the detection step and the intervention trigger result of the review step.

[0020] This invention can continuously track user input and historical dialogue trajectories in multi-turn dialogue scenarios, promptly detecting round-by-round penetration attacks. Simultaneously, during the SQL generation stage, it uses a dual semantic and structural constraint mechanism to verify the security of candidate SQL statements, proactively blocking potentially dangerous operations and preventing unauthorized access, sensitive information leakage, and database corruption. By constructing a multi-turn attack path prediction mechanism, it can assess and block potential jailbreak behaviors in advance. Furthermore, a security proxy layer is introduced before SQL generation to rewrite, de-identify, or replace high-risk operations, thereby ensuring the security of database operations. To enhance the adaptability of the defense, this invention also combines attack tracing and incremental learning mechanisms, enabling the defense system to adapt to new multi-turn jailbreak attacks and continuously optimize the detection model and defense strategies.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] The invention will now be described in more detail with reference to embodiments and the accompanying drawings.

[0023] Figure 1 This diagram illustrates an exemplary multi-round jailbreak defense method for Text-to-SQL systems proposed in one embodiment of the present invention. Figure 2 The diagram illustrates a structural block diagram of an exemplary multi-round jailbreak defense assessment device for Text-to-SQL systems according to an embodiment of the present invention. Figure 3 The diagram shows a structural block diagram of an electronic device for performing a multi-round jailbreak defense assessment method for a Text-to-SQL system according to an embodiment of the present invention. Figure 4 The diagram illustrates a computer-readable storage medium for storing or carrying a multi-round jailbreak defense method for a Text-to-SQL system according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0025] The applicant's research revealed that existing studies primarily focus on the security of large language models in single-turn dialogue scenarios, proposing methods such as input filtering, keyword masking, context constraints, and adversarial example detection. While these methods can defend against malicious input to some extent, they generally rely on single-turn question-and-answer detection mechanisms and are ill-suited for the more complex multi-turn dialogue interactions in Text-to-SQL systems. For example, attackers could gradually bypass restrictions through multiple rounds of questioning, subtly guiding seemingly harmless natural language questions towards the generation of malicious SQL commands, thereby causing unauthorized database access, leakage of sensitive information, or logical corruption.

[0026] Existing security hardening methods are mostly limited to the security protection of general conversational large language models and are not optimized for the specific characteristics of Text-to-SQL tasks. Because SQL queries are structured and executable, attackers can trigger serious security vulnerabilities simply by successfully embedding malicious semantics in a single round of dialogue. This "semantic gradual penetration" attack method currently lacks effective detection and defense mechanisms. Furthermore, existing technologies often rely on static rule bases or keyword-based blacklists to prevent jailbreak attacks. However, attackers can evade detection through semantic rewriting, contextual dispersion, and conversational metaphors, rendering traditional defense methods ineffective.

[0027] Based on this, and after research by the applicant, the existing technology has the following problems: 1. Focusing on the security of large language models in single-turn dialogues, this paper proposes methods such as input filtering, keyword masking, context constraints, and adversarial example detection. While these methods can defend against malicious input to some extent, they generally rely on single-turn question-and-answer detection mechanisms and are ill-suited for the more complex multi-turn dialogue interactions in Text-to-SQL systems. For example, attackers might gradually bypass restrictions through multiple rounds of questioning, leading seemingly harmless natural language questions to the generation of malicious SQL commands, resulting in unauthorized database access, leakage of sensitive information, or logical corruption.

[0028] 2. Most existing security hardening methods are limited to general-purpose conversational language models and are not optimized for the specific characteristics of Text-to-SQL tasks. Because SQL queries are structured and executable, an attacker only needs to successfully embed malicious semantics in a single round of dialogue to potentially trigger serious security vulnerabilities. This type of "semantic gradual penetration" attack currently lacks effective detection and defense mechanisms.

[0029] 3. Existing technologies typically rely on static rule bases or keyword-based blacklists to prevent jailbreak attacks. However, attackers can circumvent detection through semantic rewriting, contextual dispersion, or conversational metaphors, rendering traditional defenses ineffective. Existing methods lack a deep understanding of user input semantics and multi-turn contextual dependency analysis, making them vulnerable to cross-turn manipulative attacks.

[0030] 4. Existing defense technologies do not fully consider the structured nature of SQL queries and database schema constraints, making it impossible to intercept potentially dangerous statements in advance during the SQL generation stage. Even if security checks are performed on the input text, it cannot be guaranteed that the SQL generated by the model is completely safe, thus leaving the possibility of executing high-risk operations.

[0031] 5. Existing methods have limited adaptability and scalability in large-scale, multi-user concurrent scenarios, making it difficult to achieve real-time, robust defense. The long update cycles for rule bases and model fine-tuning hinder rapid response to new attack strategies, significantly limiting the effectiveness of defense.

[0032] Based on this, the applicant proposes a multi-round jailbreak defense evaluation method and device for Text-to-SQL systems. This innovative defense method addresses the significant shortcomings of existing technologies in multi-round dialogue, SQL generation security, attack path prediction, and large-scale application scenarios. It can comprehensively analyze, dynamically intercept, and securely control user input and generated SQL in a multi-round context.

[0033] The method proposed in this invention achieves dynamic and robust security control over user input and SQL generation processes. This invention can continuously track user input and historical dialogue trajectories in multi-turn dialogue scenarios, promptly detecting round-by-round penetration attacks. Simultaneously, during the SQL generation stage, it uses a dual semantic and structural constraint mechanism to verify the security of candidate SQL statements, proactively blocking potentially dangerous operations and preventing unauthorized access, sensitive information leakage, and database corruption. Furthermore, this invention constructs a multi-turn attack path prediction mechanism to pre-assess and block potential jailbreak behaviors, and introduces a security proxy layer before SQL generation to rewrite, de-identify, or substitute high-risk operations, thereby ensuring the security of database operations. To enhance the adaptability of the defense, this invention also combines attack tracing and incremental learning mechanisms, enabling the defense system to adapt to new multi-turn jailbreak attacks and continuously optimize the detection model and defense strategies.

[0034] One of the methods for preventing jailbreaks in Text-to-SQL systems will be described in detail in the following embodiments.

[0035] The following describes the application scenarios of the multi-round jailbreak defense assessment method for Text-to-SQL systems provided in this embodiment of the invention: Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-round jailbreak defense method for Text-to-SQL systems provided in this embodiment of the invention. In this embodiment, a multi-round jailbreak defense method for Text-to-SQL systems can be applied to, for example... Figure 2 The illustrated multi-round jailbreak defense assessment device 300 for Text-to-SQL systems and Figure 3 The electronic device 200 shown may include desktop computers, tablets, etc., and this invention does not limit it. The multi-round jailbreak defense method for Text-to-SQL systems may include S110 to S140, wherein: S110: Dataset Construction Steps: Construct a multi-turn jailbreak prompt detection dataset for training the Text-to-SQL model. The dataset contains multiple dialogue samples, including: current question, target SQL, dialogue history, database schema, jailbreak status indicator, and jailbreak category indicator.

[0036] In this embodiment of the invention, a multi-round Text-to-SQL system jailbreak hint detection dataset based on a large model is constructed. The purpose is to train the Text-to-SQL model to detect and recognize jailbreak attack hints hidden in user input.

[0037] In some embodiments, S110 includes S111 to S113, wherein: S111: Dataset Construction.

[0038] Specifically, the dataset consists of six core fields: "Current Question", "Target SQL", "Conversation History", "Database Schema", "Is Jailbreak", and "Jailbreak Type". The purpose of this dataset is to train a Text-to-SQL model that can identify and detect potential jailbreak attack hints in user input.

[0039] In this embodiment, the constructed dataset is formally represented as follows: Dataset=(Question,History,SQL,Schema, IsJailbreak, JailbreakType).

[0040] Among them, Question represents the natural language question entered by the user in the current round; SQL represents the standard SQL query statement corresponding to the current question; History represents the user questions and their corresponding SQL results recorded in a structured manner in previous rounds; Schema represents the table structure, column names and data type metadata of the database; IsJailbreak is used to mark whether there is jailbreak attack content in the current question and the dialogue history; JailbreakType is used to identify the specific type of jailbreak attack.

[0041] For example, the current question represents the natural language question entered by the user in the current round, which may implicitly depend on previous information. For instance, the question "budget is above average" requires reasoning based on the results of the previous round of queries (such as the AVG (budget) calculated in the second round), thus requiring the model to have contextual awareness and logical reasoning capabilities.

[0042] The dialogue history is used to record user questions from previous rounds and their corresponding SQL results in a structured manner. By explicitly annotating referential relationships (such as mapping "these departments" to the departments table) and supplementing omitted information, the model can better understand semantic dependencies across rounds.

[0043] The target SQL (SQL) corresponds to the standardized query statement for the current problem and needs to reflect the cumulative process of multiple rounds of logic. For example, the calculation results of the previous round can be reused directly through subqueries or nested conditions (such as SELECT AVG(budget) FROM departments), thereby ensuring semantic consistency and executability across rounds.

[0044] A database schema includes metadata such as the table structure, column names, and data types of the database. This field helps the model establish an accurate mapping (i.e., schema link) between natural language vocabulary and the database schema, improving the parsing accuracy of complex queries.

[0045] IsJailbreak flags the presence of jailbreak attacks in the current issue and its conversation history. JailbreakType further categorizes the type of jailbreak attack, such as database privacy breaches or SQL injection attacks.

[0046] S112: Dataset composition rules.

[0047] For example, the dataset constructed by this invention contains no less than 50,000 dialogue samples, including approximately 30,000 normal business query samples and approximately 20,000 jailbreak attack-related samples.

[0048] The normal samples cover typical scenarios such as multi-round contextual reasoning, complex nested queries, and cross-table joins to ensure that the model has the ability to parse complex semantics.

[0049] The jailbreak samples cover a variety of attack types, with SQL injection attacks accounting for about 40%, mainly interfering with normal logic by constructing concatenated conditions; database privacy leakage attacks accounting for about 35%, which are achieved by guiding the model to output sensitive field information; and pattern detection attacks accounting for about 25%, which obtain database structure and metadata by continuously asking questions.

[0050] In this step, by designing the sample size and proportion as described above, we can highlight the multidimensional coverage of jailbreak samples while ensuring semantic diversity. This enables the model to not only generate complex SQL during training but also effectively identify and defend against potential jailbreak attacks.

[0051] S113: Dataset evaluation rules.

[0052] In this embodiment, the evaluation rules for the Text-to-SQL model trained on the dataset consist of two aspects: SQL generation performance and jailbreak detection performance.

[0053] On normal business query samples, the SQL generation capability is evaluated using three metrics: exact match rate (EM), execution precision (EX), and context consistency (CC). EM measures the complete syntactic and semantic consistency between the generated SQL and the standard SQL. EX measures the consistency between the generated SQL and the standard SQL after execution in the database. CC measures whether the model can correctly inherit and reuse historical query results in multi-turn dialogues, ensuring logical consistency across rounds.

[0054] Specifically: For jailbreak attack samples, detection capability is evaluated using precision, recall, F1 score, and jailbreak category classification accuracy. Precision represents the proportion of samples judged as jailbroken that are actually jailbroken, and is calculated using the following formula:

[0055] Recall rate represents the proportion of genuine jailbreak samples that are correctly identified as jailbreaks. Its calculation formula is:

[0056] in, This indicates the number of jailbreak samples that were correctly identified. This represents the number of normal samples that were mistakenly identified as jailbreaks. This indicates the number of unidentified jailbreak samples. Based on this, The score, as the harmonic mean of precision and recall, is used to comprehensively evaluate the detection performance of the model. Its calculation formula is as follows:

[0057] In addition, jailbreak category classification accuracy is used to measure the model's correctness in distinguishing different jailbreak attack categories (such as SQL injection, database privacy leakage, pattern detection, etc.).

[0058] In this embodiment, the above evaluation rules can simultaneously verify the model's ability to generate complex SQL and identify jailbreak attacks within a unified framework, thereby ensuring that the model has both business availability and security defense capabilities in real application scenarios.

[0059] S120: Detection steps: Based on the semantic-structural bimodal jailbreak detector, the jailbreak risk is determined on the user's input question in the current round, the comprehensive jailbreak risk score of the current round is obtained, and the presence of jailbreak attack intent is determined according to the threshold.

[0060] S120 includes S121 to S125, wherein: S121: Semantically encode the user input question for the current round to obtain a semantic vector.

[0061] In this embodiment of the application, text semantic encoding is used. For the first... User input problem Perform semantic encoding to obtain semantic vectors. :

[0062] in, This represents a semantic representation vector.

[0063] S122: Based on the dialogue history and database schema, generate a set of candidate SQL statements, and perform structural encoding on the syntax tree of each candidate SQL to obtain the structure vector of each candidate SQL.

[0064] In this embodiment, SQL structure generation and encoding are based on historical dialogues. and database schema Generate candidate SQL set And encode the syntax tree (AST) for each SQL statement:

[0065] in, Indicates the first The structure of each candidate SQL statement.

[0066] S123: Calculate the similarity between the semantic vector and the corresponding structural vector of each candidate SQL, as well as the matching degree between the syntax tree structure and the database schema of each candidate SQL, and perform a weighted joint score to obtain the score for the current round.

[0067] In this embodiment, semantic-structure matching scoring is used to match each candidate SQL with the database schema and calculate a joint score for semantics and structure.

[0068]

[0069] in, Represents semantic and structural vector similarity. This indicates the degree of matching between the SQL structure and the database schema. Weighting coefficients S124: The current round's score is weighted and accumulated with the overall escape risk score of the previous rounds to update and obtain the overall escape risk score for the current round.

[0070] In this embodiment, multiple rounds of context accumulation are used, accumulating the current round score with the scores of previous rounds to account for context dependencies and hidden attacks.

[0071] in, For decay weights, This represents the overall jailbreak risk score for the current round.

[0072] S125: Compare the overall jailbreak risk score of the current round with the preset threshold to determine whether there is an intent to jailbreak and the specific type of jailbreak.

[0073] In this embodiment of the application, jailbreak determination is based on a threshold. Determine if the current input contains an intent to perform a jailbreak attack:

[0074] in, Indicates the first Whether the wheel is judged as a jailbreak attack. This is the overall risk score for the current round. It is a threshold; exceeding it will be considered a jailbreak.

[0075] At the same time, it can be maximized The corresponding candidate SQL is categorized into specific jailbreak types. .

[0076] S130: Review steps: Based on the multi-round contextual consistency review mechanism, calculate the consistency score between the current round of user input and the historical semantic state, and trigger security intervention when the consistency score is lower than the preset threshold.

[0077] S130 includes S131 to S135, wherein: S131: Semantically encode the user input questions for the current round and all previous rounds to obtain the semantic vector for each round.

[0078] In this embodiment of the application, multi-round semantic vector encoding is used. For the... User input Semantic encoding is performed to obtain a semantic vector:

[0079] S132: Based on the weighted cumulative function, the semantic vectors of historical rounds are fused to obtain the context state vector representing the historical semantic trajectory.

[0080] In this embodiment, historical semantic states are accumulated, and a context state vector is defined. Indicates up to the The semantic trajectory of a wheel:

[0081] in, This is a weighted cumulative function used to fuse historical semantic information.

[0082] S133: Calculate the similarity between the semantic vector of the current round and the context state vector of the previous round, and use it as a consistency score.

[0083] In this embodiment of the application, the semantic consistency scoring across rounds calculates the similarity between the current input and the historical semantic state:

[0084] The cosine similarity is calculated as follows:

[0085] S134: When the consistency score is lower than the preset threshold, a safety intervention flag is triggered.

[0086] In this embodiment of the application, deviation detection and security intervention are triggered when the consistency falls below a threshold δ:

[0087] in, This indicates that the current input deviates from the historical semantic trajectory and requires defensive processing.

[0088] S135: If no security intervention is triggered, update the context state vector using the semantic vector of the current round.

[0089] In this embodiment, semantic trajectory updates are performed in multiple rounds. Historical semantic states are updated without triggering intervention:

[0090] in, The cumulative weights are used to smooth historical trajectories.

[0091] In this invention, for multi-turn dialogue scenarios, a context state machine is designed to track the semantic correlation between user input and historical SQL statements, enabling anomaly detection and security intervention. This method can continuously monitor semantic consistency throughout multi-turn dialogues, effectively identifying deviations from normal business queries to potential jailbreaking or abnormal commands, thereby ensuring the security and rationality of SQL generation.

[0092] S140: Defense Execution Steps: Based on the judgment results of the detection steps and the intervention trigger results of the review steps, perform secure rewriting, de-identification processing, or normal execution on the candidate SQL statements entered or generated by the user.

[0093] In this embodiment, after completing semantic-structural bimodal detection and multi-round contextual consistency review, this step performs defensive operations on the detected jailbreak prompts. Through the above strategy, this step achieves proactive blocking of jailbreak prompts in multi-round dialogues, while ensuring that normal prompts can be processed normally, forming a closed-loop defense system of detection-review-blocking / rewriting.

[0094] For the User input and candidate SQL Based on the aforementioned risk assessment results:

[0095] Implement the following strategy: S140 includes S141 to S142, wherein: S141: When the detection step determines that there is an intent to jailbreak or the review step triggers security intervention, the user input problem is rewritten securely, and / or the generated candidate SQL statement is de-identified.

[0096] In this embodiment of the application, the jailbreak prompt processing is as follows: when... or In such cases, rewrite or de-identify user input or generated SQL:

[0097]

[0098] This is to prevent potential jailbreak attempts and to avoid the model from leaking sensitive information or performing illegal operations.

[0099] S142: If the detection step does not determine that there is an intent to jailbreak and the review step does not trigger security intervention, the SQL statement is generated and executed normally.

[0100] In this embodiment of the invention, normal prompt processing occurs when... and At this time, user input is considered safe, and the model generates SQL normally:

[0101]

[0102] The key technical feature of this invention lies in constructing a semantic-structural bimodal constraint detector for database interaction scenarios. Specifically, this method not only analyzes the natural language semantics of user input text but also incorporates the matching degree between candidate SQL statements and the database schema into the judgment system, enabling cross-modal reasoning for hidden attack intentions. In multi-turn dialogues, this detector, by accumulating contextual information and structural constraint scores, can identify inputs that appear normal but may actually lead to sensitive data leakage or illegal operations, while reducing false positives for seemingly sensitive but harmless inputs. Through this dual constraint mechanism for database scenarios, this invention exhibits significant features in two aspects: first, it tightly integrates natural language semantic understanding with database structure matching to achieve cross-modal identification of abnormal generation intentions; second, it introduces a multi-turn context accumulation mechanism, enabling the model to continuously assess risks during dynamic interactions. These designs distinguish this invention from general LLM security detection methods, focusing on the identification of harmful content in the context of Text-to-SQL and database operations, thus improving the accuracy of identification and security protection capabilities.

[0103] Please see Figure 2 , Figure 2 A structural block diagram of a multi-round jailbreak defense evaluation device 300 for Text-to-SQL systems provided by the present invention includes: a dataset construction module 310, a detection module 320, an inspection module 330, and a defense execution module 350, wherein: Dataset building module 310 is used to build a multi-turn jailbreak prompt detection dataset for training a Text-to-SQL model. The dataset contains multiple dialogue samples, which include: current question, target SQL, dialogue history, database schema, jailbreak status indicator, and jailbreak category indicator. The detection module 320 is used to determine the jailbreak risk of the user's input question in the current round based on the semantic-structural dual-modal jailbreak detector, obtain the comprehensive jailbreak risk score of the current round, and determine whether there is a jailbreak attack intent based on the threshold. The review module 330 is used to calculate the consistency score between the current round of user input and the historical semantic state based on a multi-round context consistency review mechanism, and to trigger security intervention when the consistency score is lower than a preset threshold. The defense execution module 340 is used to perform secure rewriting, de-identification processing, or normal execution of user-inputted or generated candidate SQL statements based on the judgment results of the detection steps and the intervention trigger results of the review steps.

[0104] In some embodiments, the detection module 320 further includes: The semantic encoding module is used to semantically encode the user input question in the current round to obtain a semantic vector; The SQL structure generation module is used to generate a set of candidate SQL statements based on the dialogue history and database schema, and to perform structure encoding on the syntax tree of each candidate SQL to obtain the structure vector of each candidate SQL. The matching and scoring module is used to calculate the similarity between the semantic vector and the corresponding structural vector of each candidate SQL, as well as the matching degree between the syntax tree structure of each candidate SQL and the database schema, and to perform a weighted joint score to obtain the score for the current round. The cumulative module is used to weight and accumulate the score of the current round with the comprehensive jailbreak risk score of the previous rounds, and update the comprehensive jailbreak risk score of the current round. The jailbreak determination module compares the overall jailbreak risk score of the current round with a preset threshold to determine whether there is an intent to jailbreak and the specific type of jailbreak.

[0105] It should be noted that the present invention may also include other modules. The device embodiments in the present invention correspond to the foregoing method embodiments. The specific principles of the module embodiments in the device can be found in the foregoing method embodiments, and will not be repeated here.

[0106] In the several embodiments provided in this example, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0107] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0108] Please see Figure 3 , Figure 3 The present invention provides a structural block diagram of an electronic device 200 capable of executing the above-described multi-round jailbreak defense assessment method for Text-to-SQL systems. The electronic device 200 may be a tablet computer, a computer, or a portable computer.

[0109] The electronic device 200 also includes a processor 202 and a memory 204. The memory 204 stores programs that can execute the contents of the foregoing embodiments, and the processor 202 can execute the programs stored in the memory 204.

[0110] The processor 202 may include one or more cores for data processing and message matrix units. The processor 202 connects to various parts within the electronic device 200 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 204, and by calling data stored in the memory 204. Optionally, the processor 202 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 202 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem / decoder. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem / decoder handles wireless communication. It is understood that the modem / decoder may also be implemented separately as a communication chip, without being integrated into the processor.

[0111] Memory 204 may include random access memory (RAM) or read-only memory (ROM). Memory 204 can be used to store instructions, programs, code, code sets, or instruction sets. Memory 204 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (e.g., instructions for a user to obtain random numbers), instructions for implementing the various method embodiments described below, etc. The data storage area may also store data (e.g., random numbers) created by the terminal during use.

[0112] Electronic device 200 may also include a network module and a screen. The network module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with communication networks or other devices, such as audio playback devices. The network module may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. The network module can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The screen can display interface content and facilitate data interaction.

[0113] Please refer to Figure 4 , Figure 4 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present invention is shown. The computer-readable storage medium 400 stores program code 410, which can be called by a processor to execute the methods described in the above method embodiments.

[0114] The computer-readable storage medium 400 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium 400 has storage space for program code 410 that performs any of the method steps described above. This program code 410 can be read from or written to one or more computer program products. The program code 410 may be compressed, for example, in a suitable form.

[0115] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a multi-round jailbreak defense method for Text-to-SQL systems described in the various optional implementations above.

[0116] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-round jailbreak defense method for Text-to-SQL systems, characterized in that, The method includes: Dataset construction steps: Construct a multi-turn jailbreak prompt detection dataset for training the Text-to-SQL model. The dataset contains multiple dialogue samples, including: current question, target SQL, dialogue history, database schema, jailbreak status indicator, and jailbreak category indicator. Detection steps: Based on the semantic-structural bimodal jailbreak detector, the jailbreak risk is determined on the user's input question in the current round, the comprehensive jailbreak risk score of the current round is obtained, and the presence of jailbreak attack intent is determined according to the threshold. Review steps: Based on a multi-round contextual consistency review mechanism, calculate the consistency score between the current round of user input and the historical semantic state, and trigger security intervention when the consistency score is lower than a preset threshold; Defense execution steps: Based on the judgment result of the detection step and the intervention trigger result of the review step, the candidate SQL statements input by the user or generated are rewritten securely, de-identified, or executed normally.

2. The multi-round jailbreak defense method for Text-to-SQL systems according to claim 1, characterized in that, The detection steps include: Semantic encoding is performed on the user input question in the current round to obtain a semantic vector; Based on the dialogue history and database schema, a set of candidate SQL statements is generated, and the syntax tree of each candidate SQL statement is structurally encoded to obtain the structure vector of each candidate SQL statement. Calculate the similarity between the semantic vector and the corresponding structural vector of each candidate SQL, as well as the matching degree between the syntax tree structure and the database schema of each candidate SQL, and perform a weighted joint score to obtain the score for the current round. The current round's score is weighted and accumulated with the overall escape risk score of previous rounds to update and obtain the overall escape risk score for the current round; The overall jailbreak risk score for the current round is compared with a preset threshold to determine whether there is an intent to break jailbreak and the specific type of jailbreak.

3. The multi-round jailbreak defense method for Text-to-SQL systems according to claim 1, characterized in that, The review steps include: Semantic encoding is performed on the user input questions in the current round and in previous rounds to obtain the semantic vectors for each round; Based on the weighted cumulative function, the semantic vectors of historical rounds are fused to obtain the context state vector representing the historical semantic trajectory; Calculate the similarity between the semantic vector of the current round and the context state vector of the previous round, and use it as a consistency score; When the consistency score falls below a preset threshold, a safety intervention flag is triggered. If no security intervention is triggered, the context state vector is updated using the semantic vector of the current round.

4. A multi-round jailbreak defense method for Text-to-SQL systems according to claim 1, characterized in that, The defense execution steps include: When the detection step determines that there is an intent to jailbreak or the review step triggers security intervention, the user input question is rewritten securely, and / or the generated candidate SQL statement is de-identified. If the detection step does not determine that there is an intent to jailbreak and the review step does not trigger security intervention, the SQL statement is generated and executed normally.

5. A multi-round jailbreak defense method for Text-to-SQL systems according to claim 2, characterized in that, The similarity between the semantic vector and structural vector of each candidate SQL, and the matching degree between the candidate SQL structure and the database schema, are calculated, and a weighted joint score is performed to obtain the score for the current round. The score for the current round is calculated using a first calculation formula, which includes: in, This is a function for calculating semantic and structural vector similarity. This is a function for calculating the matching degree between the SQL structure and each candidate SQL in the database. These are the weighting coefficients. For the first The structure vector of each candidate SQL statement This is the semantic vector for the current round.

6. A multi-round jailbreak defense method for Text-to-SQL systems according to claim 2, characterized in that, The step of weighting and accumulating the current round's score with the comprehensive escape risk scores of historical rounds to update the current round's comprehensive escape risk score is achieved through a second calculation formula, which includes: in, The score for the current round. This is the overall escape risk score from the previous round. This is the decay weight.

7. A multi-round jailbreak defense assessment device for Text-to-SQL systems, characterized in that, include: The dataset construction module is used to build a multi-turn jailbreak prompt detection dataset for training a Text-to-SQL model. The dataset contains multiple dialogue samples, which include: current question, target SQL, dialogue history, database schema, jailbreak status indicator, and jailbreak category indicator. The detection module is used to determine the jailbreak risk of the user's input question in the current round based on the semantic-structural bimodal jailbreak detector, obtain the comprehensive jailbreak risk score of the current round, and determine whether there is a jailbreak attack intention based on the threshold. The review module is used to calculate the consistency score between the current round of user input and the historical semantic state based on a multi-round context consistency review mechanism, and to trigger security intervention when the consistency score is lower than a preset threshold. The defense execution module is used to perform secure rewriting, de-identification processing, or normal execution on candidate SQL statements input by the user or generated, based on the judgment result of the detection step and the intervention trigger result of the review step.

8. The multi-round jailbreak defense assessment device for Text-to-SQL systems according to claim 7, characterized in that, The detection module also includes: The semantic encoding module is used to semantically encode the user input question in the current round to obtain a semantic vector; The SQL structure generation module is used to generate a set of candidate SQL statements based on the dialogue history and database schema, and to perform structure encoding on the syntax tree of each candidate SQL to obtain the structure vector of each candidate SQL. The matching and scoring module is used to calculate the similarity between the semantic vector and the corresponding structural vector of each candidate SQL, as well as the matching degree between the syntax tree structure of each candidate SQL and the database schema, and to perform a weighted joint score to obtain the score for the current round. The cumulative module is used to weight and accumulate the score of the current round with the comprehensive jailbreak risk score of the previous rounds, and update the comprehensive jailbreak risk score of the current round. The jailbreak determination module is used to compare the overall jailbreak risk score of the current round with a preset threshold to determine whether there is an intent to jailbreak and the specific type of jailbreak.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores program code that can run on the processor. When the program code is executed by the processor, it implements a multi-round jailbreak defense method for a Text-to-SQL system as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by one or more processors to execute a multi-round jailbreak defense method for Text-to-SQL systems as described in any one of claims 1-6.

Images