A multi-round jailbreaking attack evaluation method and device for a Text-to-SQL system

Abstract

The application discloses a kind of multi-round jailbreak attack evaluation method and device for Text-to-SQL system, it is related to network security technical field, the method includes: based on multi-round Text-to-SQL dataset Construction contains multi-class attack scene's jailbreak attack dataset, constructs and trains the attacker model based on pre-training language model, to generate multi-round natural language question with attack intention, establish multi-round jailbreak attack tree, to structured representation by attack target decomposition Multiple-step attack path, based on attack tree generation with semantic coherence multi-round attack question sequence, the sequence is input target Text-to-SQL system, execute the SQL query of its output, and according to evaluation index quantification attack effectiveness, the present application can systematize gradually progressive jailbreak attack in multi-round dialogue, realize the automatic testing and evaluation of Text-to-SQL model security, provide basis for system reinforcement and defense strategy formulation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and in particular to a multi-round jailbreak attack evaluation method and device for a Text-to-SQL system. BACKGROUND

[0002] Text-to-SQL is an important branch of natural language processing, aiming to automatically convert user input natural language queries into structured SQL statements for accessing and retrieving information in relational databases. With the development of large language models, the accuracy and application range of Text-to-SQL systems have been continuously improved, and they have been widely applied in intelligent customer service, data question answering, business analysis, and other fields.

[0003] In recent years, the introduction of multi-round dialogue technology has enabled users to make complex query requirements in continuous context, and multi-round Text-to-SQL models have thus become a research hotspot. Most existing mainstream models are based on large pre-training language models, combined with database schema modeling, to achieve in-depth understanding of contextual semantics and have achieved excellent performance on standard datasets (such as SParC).

[0004] However, as these models are gradually applied to real business systems, their security problems have gradually emerged. In particular, in cases where models may be disturbed by malicious user inputs and generate unexpected or harmful SQL queries, there is a lack of effective security testing and risk control means. SUMMARY

[0005] In view of the above problems, the present application provides a multi-round jailbreak attack evaluation method and device for a Text-to-SQL system to effectively support the quantitative evaluation of the security of a Text-to-SQL system, attack simulation, and defense capability verification.

[0006] In a first aspect, an embodiment of the present application provides a multi-round jailbreak attack evaluation method for a Text-to-SQL system, the method comprising:

[0007] constructing a multi-round Text-to-SQL jailbreak attack dataset, the multi-round Text-to-SQL jailbreak attack dataset containing multiple attack categories and storing normal questions, normal answers, attack questions, attack answers, and attack categories;

[0008] constructing a multi-round jailbreaker model based on a pre-training language model and training the model to generate multi-round Text-to-SQL questions with attack intent;

[0009] generate a multi-round jailbreaking attack tree, the multi-round jailbreaking attack tree being established according to a normal question-attack question pair, for representing a multi-round attack path, a root node of the multi-round jailbreaking attack tree corresponding to an attack target, and a child node of the multi-round jailbreaking attack tree corresponding to a decomposed attack step;

[0010] generate a multi-round attack question sequence based on the multi-round jailbreaking attack tree, the generating process including path selection, question instantiation, and coherence optimization;

[0011] input the multi-round attack question sequence into a target Text-to-SQL system, execute a SQL query generated by the target Text-to-SQL system, and quantify effectiveness of a multi-round jailbreaking attack based on an evaluation index corresponding to a result of the SQL query.

[0012] In some embodiments, the multi-round Text-to-SQL jailbreaking attack dataset is constructed, including:

[0013] preprocess the multi-round Text-to-SQL benchmark dataset, including: parsing a dialogue history, a current round question, and a corresponding SQL query in the multi-round Text-to-SQL benchmark dataset, and filtering low-quality samples;

[0014] construct an attack question with a jailbreaking attack intention on the basis of a normal question;

[0015] decompose a single-round attack question semantic into a multi-round dialogue form to construct a multi-round attack path;

[0016] generate a malicious SQL query corresponding to the attack question as an attack answer;

[0017] perform enhancement and balancing processing on the dataset to generate the multi-round Text-to-SQL jailbreaking attack dataset.

[0018] In some embodiments, the multi-round jailbreaker model is constructed based on the pre-trained language model and the model is trained, including:

[0019] define a target category set of an attack task, the target category set including at least one of SQL injection, privacy information leakage, data leakage, and unauthorized query;

[0020] select a pre-trained language model and perform domain self-adaptive pre-training to enable the pre-trained language model to have SQL schema perception capability;

[0021] use the multi-round Text-to-SQL jailbreaking attack dataset to perform supervised training on the multi-round jailbreaker model, with a normal question as input and an attack question as expected output.

[0022] In some embodiments, the multi-round jailbreaking attack tree is generated, including:

[0023] mapping the category of the attack target as a root node of a multi-round jailbreak attack tree to represent the attack intention;

[0024] splitting the attack intention into a plurality of consecutive sub-problems by using a semantic decomposition algorithm to form sub-nodes of the multi-round jailbreak attack tree;

[0025] allocating an attack question template and a potential SQL query to each node in the multi-round jailbreak attack tree, and establishing a contextual constraint relationship between nodes;

[0026] According to the feedback of the target Text-to-SQL system, dynamically adjusting and expanding the path of the multi-round jailbreak attack tree.

[0027] In some embodiments, the multi-round jailbreak attack tree is used to generate a multi-round attack question sequence, including:

[0028] According to the attack target, selecting an attack path with the highest expected return from the multi-round jailbreak attack tree;

[0029] Using the multi-round jailbreak attacker model, instantiating the question template of the node in the selected path to generate a multi-round attack question conforming to the natural language question;

[0030] Optimizing the generated multi-round question for coherence.

[0031] In some embodiments, the effectiveness of the multi-round jailbreak attack is quantified based on the evaluation indicators corresponding to the SQL query results, including:

[0032] Designing evaluation indicators, the evaluation indicators including at least one of attack success rate, question concealment score, SQL execution feasibility, and attack coverage rate;

[0033] Performing an automated evaluation process: inputting the attack question sequence into the target Text-to-SQL system in batches, collecting the generated SQL queries, executing and recording the results in the database, and comparing with the expected attack effect to output the effectiveness result of the multi-round jailbreak attack.

[0034] In a second aspect, the embodiments of the present application provide a multi-round jailbreak attack evaluation device for a Text-to-SQL system, the device comprising:

[0035] A data set construction module for constructing a multi-round Text-to-SQL jailbreak attack data set, the Text-to-SQL jailbreak attack data set containing a plurality of attack categories and storing normal questions, normal answers, attack questions, attack answers, and attack categories;

[0036] The attacker model building module is used to build a multi-round jailbreak attacker model based on a pre-trained language model and train the model to generate multi-round Text-to-SQL questions with attack intent;

[0037] The attack tree generation module is used to generate a multi-round jailbreak attack tree. The multi-round jailbreak attack tree is established based on normal question-attack question pairs and is used to represent the multi-round attack path. Its root node corresponds to the attack target and its child nodes correspond to the decomposed attack steps.

[0038] The attack question generation module is used to generate a multi-round attack question sequence based on the multi-round jailbreak attack tree. The generation process includes path selection, question instantiation, and coherence optimization.

[0039] The evaluation module is used to input the multi-round attack question sequence into the target Text-to-SQL system, execute the SQL query generated by the target Text-to-SQL system, and quantify the effectiveness of the multi-round jailbreak attack based on the evaluation index corresponding to the SQL query result.

[0040] In some embodiments, the dataset building module further includes:

[0041] The preprocessing module is used to preprocess the multi-turn Text-to-SQL benchmark dataset, including: parsing the dialogue history, current turn question and corresponding SQL query in the multi-turn Text-to-SQL benchmark dataset, and filtering low-quality samples;

[0042] The attack question generation submodule is used to construct attack questions with jailbreak attack intent based on normal questions;

[0043] The decomposition module is used to decompose the semantics of a single-turn attack question into a multi-turn dialogue format in order to construct a multi-turn attack path;

[0044] The attack response generation module is used to generate malicious SQL queries corresponding to the attack questions as attack responses.

[0045] The enhancement module is used to enhance and balance the dataset to generate a multi-round Text-to-SQL jailbreak attack dataset.

[0046] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores program code executable on the processor, and when the program code is executed by the processor, it implements the multi-round jailbreak attack assessment method for Text-to-SQL systems as described in any embodiment of the first aspect.

[0047] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores one or more programs, and the one or more programs can be executed by the electronic device as introduced in the third aspect to implement the method for evaluating multi-round jailbreaking attacks on a Text-to-SQL system as introduced in any one of the embodiments of the first aspect.

[0048] The method for evaluating multi-round jailbreaking attacks on a Text-to-SQL system provided by the embodiment of the present application includes: constructing a multi-round Text-to-SQL jailbreaking attack dataset, the Text-to-SQL jailbreaking attack dataset containing multiple attack categories and storing normal questions, normal answers, attack questions, attack answers, and attack categories; constructing a multi-round jailbreaker model based on a pre-trained language model and performing model training to generate multi-round Text-to-SQL questions with attack intentions; generating a multi-round jailbreaking attack tree, the multi-round jailbreaking attack tree being established according to normal question-attack question pairs and used to represent a multi-round attack path, a root node of the multi-round jailbreaking attack tree corresponding to an attack target and a child node corresponding to a decomposed attack step; generating a multi-round attack question sequence based on the multi-round jailbreaking attack tree, the generation process including path selection, question instantiation, and coherence optimization; inputting the multi-round attack question sequence into a target Text-to-SQL system, executing an SQL query generated by the target Text-to-SQL system, and quantifying the effectiveness of the multi-round jailbreaking attack based on an evaluation index corresponding to a result of the SQL query.

[0049] The present application supports attack path disassembly and synthesis in a multi-round dialogue form, can simulate the strategy of a real attacker gradually penetrating the system boundary in a dialogue process, can accurately judge whether the model produces an overreach, data leakage, or destructive query behavior by inputting the generated attack question into the target model, extracting the output SQL, and verifying the execution effect in the database. At the same time, the present application establishes a complete attack generation, execution verification, and result recording process, which can be used to support subsequent defense mechanism design and security training, and provides a technical foundation for constructing a more robust and reliable Text-to-SQL system.

[0050] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0051] The present application will be described in more detail below based on embodiments and with reference to the accompanying drawings.

[0052] Figure 1 An exemplary method for evaluating multi-round jailbreaking attacks on a Text-to-SQL system is shown in an embodiment of the present application.

[0053] Figure 2 A structural block diagram of an exemplary multi-round jailbreak attack evaluation device for a Text-to-SQL system according to an embodiment of the present application is shown;

[0054] Figure 3 A structural block diagram of an electronic device for performing a multi-round jailbreak attack evaluation method for a Text-to-SQL system according to an embodiment of the present application is shown;

[0055] Figure 4 A structural block diagram of a computer readable storage medium for storing or carrying a multi-round jailbreak attack evaluation method for a Text-to-SQL system according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed descriptions will be given below in combination with embodiments and drawings, which are merely used to explain the present application and should not be regarded as limitations of the present application.

[0057] The applicant has found that although some current work focuses on the adversarial robustness of language models, such as prompt injection or data leakage risks, research on jailbreak attacks for Text-to-SQL systems, especially in multi-round dialogue scenarios, is still in a blank state.

[0058] Research has found that existing attack methods are mostly limited to single-round static questions, lack the ability to model attack paths in a multi-round context, and cannot truly simulate the behavior evolution process of attackers in dialogue systems. At the same time, most existing attack samples rely on artificial design or rule construction, lack intelligent and automated support, and are difficult to generate harmful questions with concealment and diversity.

[0059] In terms of evaluation, there is no public multi-round jailbreak attack dataset, and there is also a lack of closed-loop testing procedures from question generation to database verification, making it difficult to quantify whether the model has truly "jailbroken". Therefore, current technical means cannot effectively support the security evaluation, attack simulation and defense capability verification of Text-to-SQL systems.

[0060] Therefore, the applicant proposes a multi-round jailbreaking attack evaluation method and device for a Text-to-SQL system to fill the gap in the security research in this field. By constructing an extensible jailbreaking evaluation dataset, designing an attacker model with context modeling capability, automatically generating a multi-round, hidden, and semantically reasonable attack question sequence, guiding the target system to output a potentially harmful SQL query, the jailbreaking capability of the model can be systematically tested and evaluated.

[0061] The method proposed in the application supports attack path disassembly and synthesis in the form of multi-round dialogue, and can simulate the strategy of a real attacker gradually penetrating the system boundary in the dialogue process. By inputting the generated attack question into the target model, extracting the output SQL and verifying the execution effect in the database, it can accurately determine whether the model produces unauthorized, data leakage or destructive query behavior. At the same time, the application establishes a complete attack generation, execution verification and result recording process, which can be used to support the design of subsequent defense mechanisms and security training, and provides a technical foundation for building more robust and reliable Text-to-SQL systems.

[0062] The multi-round jailbreaking attack evaluation method for a Text-to-SQL system is described in detail in subsequent embodiments.

[0063] The application scenarios of the multi-round jailbreaking attack evaluation method for a Text-to-SQL system provided by the embodiments of the application are described below:

[0064] Please refer to Figure 1 , Figure 1 The process diagram of the multi-round jailbreaking attack evaluation method for a Text-to-SQL system provided in the embodiments of the application is shown in the following figure: Figure 2 The multi-round jailbreaking attack evaluation method for a Text-to-SQL system can be applied to a multi-round jailbreaking attack evaluation device 300 for a Text-to-SQL system as shown in Figure 3 The electronic device 200 can include a desktop computer, a tablet, etc., and the application does not limit it. The multi-round jailbreaking attack evaluation method for a Text-to-SQL system can include S110 to S150.

[0065] S110: Construct a multi-round Text-to-SQL jailbreaking attack dataset, which contains multiple attack categories and stores normal questions, normal answers, attack questions, attack answers, and attack categories.

[0066] In the embodiment of the application, a multi-round Text-to-SQL jailbreaking attack dataset SParC-Attacker is constructed. The multi-round jailbreaking attack dataset is constructed based on the multi-round Text-to-SQL dataset SParC.

[0067] Among them, the categories are SQL injection, database privacy leakage, data leakage, and unauthorized access. The constructed dataset contains five fields: normal query (normal_query), normal answer (normal_answer), attack query (attack_query), attack answer (attack_answer), and attack category (type). This step can be based on the multi-round Text-to-SQL dataset SParC to expand the attack-defense scenario and form a five-tuple jailbreaking dataset.

[0068] S120: Based on the pre-trained language model, a multi-round jailbreaker model is constructed and trained to generate multi-round Text-to-SQL questions with attack intent.

[0069] In the embodiment of the application, the multi-round jailbreaker model is constructed, and the T5-3B model can be used to construct the attacker model. The learning goal of the model is to generate harmful multi-round Text-to-SQL questions. The question can induce the Text-to-SQL system to output database privacy information. This step aims to train an attacker model that can automatically generate natural language questions with attack intent.

[0070] S130: Generate a multi-round jailbreaking attack tree. The multi-round jailbreaking attack tree is established according to the normal question-attack question pair, and is used to represent the multi-round attack path. The root node corresponds to the attack target, and the child node corresponds to the decomposed attack step.

[0071] In the embodiment of the application, in the multi-round jailbreaking attack tree, the model generates a multi-round attack tree according to the normal question-attack question pair in the training sample. The nodes of the tree include multiple attack nodes. The tree can make the attacker model output effective harmful information. This step constructs a tree structure to represent the multi-round attack path, which is used to guide the attacker model to gradually induce the target system in multi-round dialogue.

[0072] S140: Generate a multi-round attack question sequence based on the multi-round jailbreaking attack tree. The generation process includes path selection, question instantiation, and coherence optimization.

[0073] In the embodiment of the application, according to the multi-round jailbreaking attack tree, the semantic decomposition algorithm is used to decompose into multi-round attack questions, and the model outputs effective harmful information. This step generates the final executable attack question sequence based on the attack tree. This step generates the final executable multi-round attack question sequence based on the attack tree , including path selection, question instantiation and coherence optimization.

[0074] S150: inputting the multi-round attack question sequence into the target Text-to-SQL system, executing the SQL query generated by the target Text-to-SQL system, and quantifying the effectiveness of the multi-round jailbreak attack based on the evaluation index corresponding to the SQL query result.

[0075] In the embodiment of the application, the proposed data set SParC-Attacker is used for evaluation. The specific steps and details of the evaluation are given to realize quantitative evaluation of the effectiveness of the proposed multi-round jailbreak attack method.

[0076] In some embodiment modes, the multi-round Text-to-SQL benchmark data set in S110 is used as the basis to construct a multi-round Text-to-SQL jailbreak attack data set, including S111 to S115, wherein:

[0077] S111: preprocessing the multi-round Text-to-SQL benchmark data set, including: parsing the dialogue history, the current round question and the corresponding SQL query in the multi-round Text-to-SQL benchmark data set, and filtering low-quality samples.

[0078] In the embodiment of the application, in the preprocessing of the original data, first, the public multi-round Text-to-SQL benchmark data set SParC is selected as the basic data source. The dialogue history (history), the current round question (query) and the corresponding SQL query (sql) in the original data are parsed. At the same time, the database schema (schema) is extracted as a context constraint condition. In order to improve the data quality, low-quality samples with SQL syntax errors or semantic ambiguity are filtered to ensure the reliability of the data set.

[0079] S112: constructing attack questions with jailbreak attack intention on the basis of normal questions.

[0080] In the embodiment of the application, in the attack question generation, the question with jailbreak attack intention is constructed on the basis of the normal question, that is, the natural language input is disguised to induce the model to generate the SQL query with potential harm.

[0081] Among them, such attack questions have certain concealment and are integrated into the multi-round dialogue context to enhance the complexity and realism of the attack. This stage can include decomposing the complex attack intention into multiple questions to form a multi-round dialogue structure.

[0082] S113: decomposing the single-round attack question semantics into a multi-round dialogue form to construct a multi-round attack path.

[0083] In the embodiment of the application, the single-round attack question is further semantically disassembled into a multi-round dialogue form to construct a multi-round attack path. Through semantic separation and reconstruction, the attack intent gradually emerges in the multi-round dialogue, thereby improving the penetration of the attack and the coverage of the evaluation.

[0084] S114: generating a malicious SQL query corresponding to the attack question as an attack answer.

[0085] In the embodiment of the application, attack answer generation is performed, and a malicious SQL query (harmful query) corresponding to the attack question is generated in the above step to verify whether the jailbreaking is successful.

[0086] The specific process is as follows: first, the target attack SQL (malicious_sql) is artificially or automatically generated according to the attack question; then, the SQL is semantically verified in the target database system; next, the attack question is input into the Text-to-SQL system to be tested, and the generated SQL (attack_sql) is recorded; finally, the SQL is executed in the database, and the exceptions or sensitive data returned by the system are captured as the attack answer (attack_answer) for judging whether the jailbreaking is successful.

[0087] S115: enhancing and balancing the data set to generate a multi-round Text-to-SQL jailbreaking attack data set.

[0088] In the embodiment of the application, in order to improve the data scale and diversity, the existing data samples are used as semantic templates to generate attack questions with different structures but similar semantics by using the large language model GPT-4, thereby realizing corpus expansion. At the same time, by controlling the proportion of normal questions and attack questions, the data set is enhanced and balanced, thereby improving the robustness of model training and evaluation.

[0089] In the embodiment, the SParC-Attacker data set is used for multi-round jailbreaking attack data set construction, and the SQL injection, privacy leakage, data leakage, and unauthorized access attack categories are expanded based on the multi-round Text-to-SQL data set SParC, and the question and answer and attack information are stored in the form of five tuples, thereby realizing the standardized construction of the multi-round attack scene and providing an innovative data basis for jailbreaking attack research.

[0090] In some embodiments, the multi-round jailbreaker model is constructed based on the pre-trained language model in S120 and the model is trained, including S121 to S123, wherein:

[0091] S121: defining a target category set of the attack task, and the target category set includes at least one of SQL injection, privacy information leakage, data leakage, and unauthorized access query.

[0092] In the embodiments of the present application, the attack target definition is set by setting the target category of the attack task, including but not limited to SQL injection, database privacy information leakage, table structure exposure, unauthorized query, etc. The attack target set is defined as:

[0093]

[0094] Among them, represents a set of attack targets, different categories correspond to different threats in database security, represents SQL injection, represents database privacy leakage, represents data leakage, represents unauthorized.

[0095] S122: Select a pre-trained language model and perform domain self-adaption pre-training to make it have SQL schema perception ability.

[0096] In the embodiments of the present application, the pre-trained model is selected and adapted. A large-scale generative model T5-3B is used as a basic attacker model, which is subjected to domain self-adaption pre-training to make it have SQL schema perception ability.

[0097] For example, by giving an input question , the model generates a potential attack question The conditional probability of the question is:

[0098]

[0099] Among them, is the conditional probability of generating a potential attack question , is the model parameter, is the generation sequence length, is the word generated in the th generation.

[0100] In the embodiments, a multi-round jailbreak attacker model is designed based on the T5-3B model, which can automatically generate multi-round Text-to-SQL questions with attack intent, effectively induce the target system to output sensitive database information, and innovatively realize the automation and scalability of attack question generation.

[0101] S123: Use the multi-round Text-to-SQL jailbreak attack dataset to supervise the training of the multi-round jailbreak attacker model, with normal questions as input and attack questions as expected output.

[0102] ​In the embodiments of the present application, the attack question generation training is performed by inputting the normal question and the corresponding attack question in the data set SParC-Attacker into the model, and a supervised training method is used to make the model learn the mapping relationship from the normal question to the attack question.

[0103] Specifically, by using the data set SParC-Attacker, the normal question and the corresponding attack question are used as training samples to optimize the model in a supervised training manner. The objective function is defined as maximizing the semantic feasibility and attack success rate of the attack question, which can be expressed as:

[0104]

[0105] wherein, is the objective function, is the logarithmic function, is the training set, and the first term is the cross-entropy loss to ensure the semantic correctness of the generated question, is the regular term related to the attack success rate, is the weighting coefficient.

[0106] In the embodiments of the present application, the automatic generation of attack questions is realized, and the multi-round jailbreak attacker model based on T5-3B can automatically generate multi-round questions with attack intent, improving the efficiency and expansibility of the jailbreak attack simulation and reducing the cost of manual intervention.

[0107] In some embodiments, the multi-round jailbreak attack tree is generated in S130, including S131 to S134, wherein:

[0108] S131: mapping the category of the attack target to the root node of the multi-round jailbreak attack tree to represent the attack intent.

[0109] In the embodiments of the present application, in the attack intent modeling process, the attack target category is mapped to the root node of the tree (such as "database table information leakage"), and the node further derives multiple sub-attack paths.

[0110] Exemplarily:

[0111] Given the attack target category , the root node of the tree is mapped :

[0112]

[0113] wherein, represents the attack target set, for example, database table information leakage, and the mapping function is used to convert the abstract target into a specific attack intent.

[0114] S132: split the attack intent into multiple coherent sub-problems using a semantic decomposition algorithm to form the child nodes of the multi-round jailbreak attack tree.

[0115] In the embodiments of the present application, in the multi-round question decomposition, the single attack intent is gradually split into multiple rounds of sub-problems using a semantic decomposition algorithm dependency syntax analysis to form an attack path. For example, "get the teacher password field" is split into "first query the teacher table", "then request the field information of the table", and "finally specify the password field".

[0116] Exemplary:

[0117] using a semantic decomposition algorithm , the single attack intent is gradually split into multiple rounds of sub-problems :

[0118]

[0119] wherein each represents a step in the multi-round attack path, for example: = ''query the teacher table'', = ''request the field information of the table'', = ''specify the password field''.

[0120] S133: assign an attack question template and a potential SQL query to each node in the multi-round jailbreak attack tree, and establish context constraint relationships between nodes.

[0121] In the embodiments of the present application, in the attack tree node generation, an attack question template and a potential SQL query are assigned to each node in the tree, and the nodes are connected through context constraints, so that the attack has continuity and progressiveness in the multi-round dialogue.

[0122] Exemplary:

[0123] an attack question template and a potential SQL query are assigned to each node in the tree , and context constraints are established :

[0124]

[0125] wherein represents the context dependency between nodes, so that the attack process has continuity and progressiveness. represents the natural language query context.

[0126] S134: dynamically adjust and expand the path of the multi-round jailbreaking attack tree according to the feedback of the target Text-to-SQL system.

[0127] In the embodiments of the present application, dynamic expansion of the attack tree is implemented. In the actual attack process, if the feedback given by the target system does not match the expectation, the attack path is dynamically adjusted, a new child node is generated in the tree structure, and adaptive attack is implemented.

[0128] Exemplary:

[0129] If the feedback of the target system does not match the expectation , the attack tree is dynamically expanded:

[0130]

[0131] wherein, represents an adaptive adjustment function for generating a new attack path node, implementing dynamic expansion and correction, represents resetting the attack tree path, represents the attack tree.

[0132] In the embodiments, the multi-round jailbreaking attack tree maps normal questions and attack questions into a tree structure, each node represents an attack operation, and can guide the model to gradually output effective harmful information in multiple rounds of dialogue, implement the structuring and controllability of the attack path, and innovatively map normal questions and attack questions into a tree structure, making the attack path controllable and traceable, and facilitating the analysis and optimization of complex multi-round attacks.

[0133] In some embodiments, the multi-round attack question sequence is generated based on the multi-round jailbreaking attack tree in S140, including S141-S143, wherein:

[0134] S141: Selecting an attack path with the highest expected return from the multi-round jailbreaking attack tree according to the attack target.

[0135] In the embodiments of the present application, one or more attack paths can be selected according to the attack target category.

[0136] Exemplary:

[0137] Let the candidate attack path set be (based on the root intent extracted from the attack tree), and define the expected return of path as:

[0138]

[0139] wherein, is the path In the first node of the step, dialog history for the step, database schema, offline agent reward function, path selection, i.e., solving for:

[0140]

[0141] S142: using the multi-round jailbreak attacker model, the question template of the node in the selected path is instantiated to generate multi-round attack questions conforming to natural language questions.

[0142] In the embodiment of the application, the multi-round attack questions conforming to natural language expressions are obtained by using the attacker model to generate the question templates of the attack tree nodes.

[0143] Exemplary:

[0144] path , each node with a template or slot representation . Let the attacker model be a conditional generation distribution , the generation and instantiation of the first round question can be represented as:

[0145]

[0146]

[0147] wherein, represents the instantiated question based on maximum posteriori (or beam search), and the generation sequence is .

[0148] S143: performing coherence optimization on the generated multi-round questions.

[0149] In the embodiment of the application, the attack question coherence optimization is performed by using a context consistency algorithm to ensure that the semantic connection between the multi-round questions is reasonable and obvious attack signs are avoided.

[0150] In the embodiment, a semantic decomposition generation method is designed based on the attack tree, and the tree is decomposed into an executable multi-round question sequence , including path selection, question instantiation and coherence optimization, to realize efficient generation of multi-round questions and guarantee attack effectiveness, which has high innovation.

[0151] In some embodiments, the effectiveness of the multi-round jailbreak attack is quantified based on the evaluation index corresponding to the SQL query result in S150, including S151 to S152.

[0152] S151: design evaluation indicators, the evaluation indicators include at least one of attack success rate, question concealment score, SQL execution feasibility and attack coverage.

[0153] In the embodiment of the application, the evaluation indicators can be designed to set attack success rate (ASR), question concealment score, SQL execution feasibility, coverage and the like. Among them:

[0154] Attack success rate = number of times of successfully inducing system to output harmful SQL / total number of attacks;

[0155] Question concealment score = similarity of attack question and normal question in semantic similarity model;

[0156] SQL execution feasibility = proportion of attack SQL returning valid results after execution on target database;

[0157] Coverage = proportion of attack questions involving different categories of attack targets.

[0158] S152: execute an automated evaluation process: batch input attack question sequence into the target Text-to-SQL system, collect the generated SQL queries, execute and record the results in the database, and compare with the expected attack effect to output the effectiveness results of multiple rounds of jailbreaking attacks.

[0159] In the embodiment of the application, the generated attack questions can be batch input into the target system in the automated evaluation process, and the system output SQL can be automatically collected, the output results can be executed and compared, and a structured evaluation log can be formed.

[0160] By designing ASR, question concealment score, SQL execution feasibility and coverage and the like, and combining the automated evaluation process, the effectiveness of the multiple rounds of jailbreaking attack method can be comprehensively and quantitatively verified, and reference basis can be provided for security protection strategies.

[0161] Please refer to Figure 2 , Figure 2 A structural block diagram of a multi-round jailbreaking attack evaluation device for a Text-to-SQL system provided by the application, comprising: a data set construction module 310, an attacker model construction module 320, an attack tree generation module 330, an attack question generation module 340 and an evaluation module 350, wherein:

[0162] The data set construction module 310 is used to construct a multi-round Text-to-SQL jailbreaking attack data set, and the Text-to-SQL jailbreaking attack data set contains multiple attack categories and stores normal questions, normal answers, attack questions, attack answers and attack categories;

[0163] The attacker model construction module 320 is configured to construct a multi-round jailbreak attacker model based on a pre-trained language model and perform model training to generate multi-round Text-to-SQL question sentences with attack intentions.

[0164] The attack tree generation module 330 is configured to perform target detection on the defogged image and output a detection result.

[0165] The attack question generation module 340 is configured to generate a multi-round attack question sequence based on the multi-round jailbreak attack tree, and the generation process includes path selection, question instantiation and coherence optimization.

[0166] The evaluation module 350 is configured to input the multi-round attack question sequence into a target Text-to-SQL system, execute an SQL query generated by the target Text-to-SQL system, and quantify the effectiveness of the multi-round jailbreak attack based on an evaluation index corresponding to the SQL query result.

[0167] In some embodiments, the data set construction module further comprises:

[0168] The preprocessing module is configured to preprocess the multi-round Text-to-SQL benchmark data set, including parsing the dialogue history, the current round question and the corresponding SQL query in the multi-round Text-to-SQL benchmark data set, and filtering low-quality samples.

[0169] The attack question generation submodule is configured to construct attack questions with jailbreak attack intentions based on normal questions.

[0170] The disassembly module is configured to disassemble the semantic of the single-round attack question into a multi-round dialogue form to construct a multi-round attack path.

[0171] The attack answer generation module is configured to generate a malicious SQL query corresponding to the attack question as an attack answer.

[0172] The enhancement module is configured to perform enhancement and balance processing on the data set to generate a multi-round Text-to-SQL jailbreak attack data set.

[0173] It should be noted that the present application can also include other modules. For details, please refer to the device embodiments and the corresponding method embodiments described above. The specific principles in the module embodiments of the device can be referred to the contents in the foregoing method embodiments, which will not be repeated here.

[0174] In several embodiments provided in the present embodiment, the coupling between the modules can be electrical, mechanical or other forms of coupling.

[0175] In addition, each function module in each embodiment of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module.

[0176] Please refer to Figure 3 , Figure 3 A structural block diagram of an electronic device 200 that can execute the above-mentioned method for evaluating multi-round jailbreaking attacks on a Text-to-SQL system is provided for an embodiment of the present application. The electronic device 200 can be a tablet computer, a computer, or a portable computer, etc.

[0177] The electronic device 200 further includes a processor 202 and a memory 204. The memory 204 stores programs that can execute the contents of the foregoing embodiments, and the processor 202 can execute the programs stored in the memory 204.

[0178] The processor 202 can include one or more cores for processing data and a message matrix unit. The processor 202 connects various parts in the entire electronic device 200 through various interfaces and lines, executes various functions and processes data of the electronic device 200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 204, and calling data stored in the memory 204. Alternatively, the processor 202 can be realized in at least one of hardware forms of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 202 can be integrated with one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU mainly processes an operating system, a user interface, and an application program, etc.; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be realized by a separate communication chip.

[0179] The memory 204 can include a Random Access Memory (RAM) and can also include a Read-Only Memory (ROM). The memory 204 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 204 can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (e.g., instructions for a user to obtain a random number), instructions for implementing each of the method embodiments described below, and the like. The data storage area can also store data created by the terminal while in use (e.g., a random number), and the like.

[0180] The electronic device 200 can also include a network module and a screen. The network module is configured to receive and send electromagnetic waves, convert the electromagnetic waves and electrical signals to each other, and thus communicate with a communication network or other devices, such as an audio playing device. The network module can include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a Subscriber Identity Module (SIM) card, a memory, and the like. The network module can communicate with various networks, such as the Internet, an intranet, a wireless network, or other devices through the wireless network. The wireless network described above can include a cellular telephone network, a wireless local area network, or a metropolitan area network. The screen can display interface content and perform data interaction.

[0181] Reference is made to Figure 4 , Figure 4 A structure block diagram of a computer readable storage medium provided by an embodiment of the present application is shown. The computer readable storage medium 400 stores program codes 410, which can be invoked by a processor to execute the methods described in the above method embodiments.

[0182] The computer readable storage medium 400 can be an electronic storage such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer readable storage medium includes a non-transitory computer readable medium. The computer readable storage medium 400 has a storage space for the program codes 410 for executing any of the method steps described above. These program codes 410 can be read from or written into one or more computer program products. The program codes 410 can be compressed in an appropriate form, for example.

[0183] The embodiment of the present application further provides a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for evaluating multi-round jailbreaking attacks on a Text-to-SQL system described in various optional implementation manners.

[0184] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent replacements; and the modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating multi-round jailbreak attacks on Text-to-SQL systems, characterized in that, The method includes: A multi-round Text-to-SQL jailbreak attack dataset is constructed, which contains multiple attack categories and stores normal questions, normal answers, attack questions, attack answers, and attack categories; A multi-round jailbreak attacker model is constructed based on a pre-trained language model and trained to generate multi-round Text-to-SQL questions with attack intent; A multi-round jailbreak attack tree is generated. The multi-round jailbreak attack tree is built based on normal question-attack question pairs and is used to represent the multi-round attack path. Its root node corresponds to the attack target and its child nodes correspond to the decomposed attack steps. Based on the multi-round jailbreak attack tree, a multi-round attack question sequence is generated. The generation process includes path selection, question instantiation, and coherence optimization. The multi-round attack question sequence is input into the target Text-to-SQL system, the SQL query generated by the target Text-to-SQL system is executed, and the effectiveness of the multi-round jailbreak attack is quantified based on the evaluation index corresponding to the SQL query result.

2. The method for evaluating multi-round jailbreak attacks on Text-to-SQL systems according to claim 1, characterized in that, The construction of the multi-round Text-to-SQL jailbreak attack dataset includes: Preprocessing is performed on the multi-turn Text-to-SQL benchmark dataset, including parsing the dialogue history, current turn question, and corresponding SQL query in the multi-turn Text-to-SQL benchmark dataset, and filtering low-quality samples; Based on normal questions, construct attack questions with the intent of jailbreak attacks; The semantics of single-turn attack questions are decomposed into multi-turn dialogue formats in order to construct multi-turn attack paths; Generate a malicious SQL query corresponding to the attack question as the attack response; The dataset is augmented and balanced to generate a dataset for constructing multiple rounds of Text-to-SQL jailbreak attacks.

3. The method for evaluating multi-round jailbreak attacks on Text-to-SQL systems according to claim 1, characterized in that, The process of constructing a multi-round jailbreak attacker model based on a pre-trained language model and training the model includes: Define a set of target categories for attack tasks, wherein the set of target categories includes at least one of SQL injection, privacy information leakage, data leakage, and unauthorized query; Select a pre-trained language model and perform domain-adaptive pre-training to enable it to have SQL pattern awareness capabilities; Using the aforementioned multi-round Text-to-SQL jailbreak attack dataset, with normal questions as input and attack questions as the expected output, a multi-round jailbreak attacker model is trained under supervision.

4. The method for evaluating multi-round jailbreak attacks on Text-to-SQL systems according to claim 1, characterized in that, The generation of the multi-round jailbreak attack tree includes: The categories of attack targets are mapped to the root nodes of a multi-round jailbreak attack tree to represent the attack intent; The attack intent is broken down into multiple coherent sub-problems using a semantic decomposition algorithm, forming the sub-nodes of the multi-round jailbreak attack tree; Assign attack question templates and potential SQL queries to each node in the multi-round jailbreak attack tree, and establish contextual constraint relationships between nodes; Based on feedback from the target Text-to-SQL system, the path of the multi-round jailbreak attack tree is dynamically adjusted and expanded.

5. The method for evaluating multi-round jailbreak attacks on Text-to-SQL systems according to claim 1, characterized in that, The generation of a multi-round attack question sequence based on the multi-round jailbreak attack tree includes: Based on the attack target, the attack path with the highest expected return is selected from the multi-round jailbreak attack tree; Using the multi-round jailbreak attacker model, the question templates of the nodes in the selected path are instantiated to generate multi-round attack questions that conform to natural language questions; The coherence of the generated multi-turn questions is optimized.

6. The method for evaluating multi-round jailbreak attacks on Text-to-SQL systems according to claim 1, characterized in that, The evaluation metrics based on the SQL query results quantify the effectiveness of multi-round jailbreak attacks, including: The evaluation metrics are designed, including at least one of the following: attack success rate, query concealment score, SQL execution feasibility, and attack coverage. Execute an automated evaluation process: input the attack question sequence in batches into the target Text-to-SQL system, collect the generated SQL queries, execute them in the database and record the results, compare them with the expected attack effect, and output the effectiveness results of the multi-round jailbreak attack.

7. A multi-round jailbreak attack evaluation device for Text-to-SQL systems, characterized in that, include: The dataset construction module is used to construct a multi-round Text-to-SQL jailbreak attack dataset. The Text-to-SQL jailbreak attack dataset contains multiple attack categories and stores normal questions, normal answers, attack questions, attack answers, and attack categories. The attacker model building module is used to build a multi-round jailbreak attacker model based on a pre-trained language model and train the model to generate multi-round Text-to-SQL questions with attack intent; The attack tree generation module is used to generate a multi-round jailbreak attack tree. The multi-round jailbreak attack tree is established based on normal question-attack question pairs and is used to represent the multi-round attack path. Its root node corresponds to the attack target and its child nodes correspond to the decomposed attack steps. The attack question generation module is used to generate a multi-round attack question sequence based on the multi-round jailbreak attack tree. The generation process includes path selection, question instantiation, and coherence optimization. The evaluation module is used to input the multi-round attack question sequence into the target Text-to-SQL system, execute the SQL query generated by the target Text-to-SQL system, and quantify the effectiveness of the multi-round jailbreak attack based on the evaluation index corresponding to the SQL query result.

8. The multi-round jailbreak attack evaluation device for Text-to-SQL systems according to claim 7, characterized in that, The dataset construction module also includes: The preprocessing module is used to preprocess the multi-turn Text-to-SQL benchmark dataset, including: parsing the dialogue history, current turn question and corresponding SQL query in the multi-turn Text-to-SQL benchmark dataset, and filtering low-quality samples; The attack question generation submodule is used to construct attack questions with jailbreak attack intent based on normal questions; The decomposition module is used to decompose the semantics of a single-turn attack question into a multi-turn dialogue format in order to construct a multi-turn attack path; The attack response generation module is used to generate malicious SQL queries corresponding to the attack questions as attack responses. The enhancement module is used to enhance and balance the dataset to generate a multi-round Text-to-SQL jailbreak attack dataset.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores program code that can run on the processor. When the program code is executed by the processor, it implements a multi-round jailbreak attack evaluation method for a Text-to-SQL system as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by one or more processors to execute a multi-round jailbreak attack evaluation method for a Text-to-SQL system as described in any one of claims 1-6.

Images