Method for training policy model in agent, privacy protection method, device, storage medium and program product

Abstract

The application provides a method for training a policy model in an agent, a privacy protection method, equipment, a storage medium and a program product, relating to the technical field of security. The method models the privacy protection task of the agent as a Markov decision process, and uniformly incorporates tool calling and reflection review actions in the action space, and trains the policy model in combination with an instant reward mechanism, effectively solving the problems that the model in the prior art is prone to time sequence confusion and falls into a dead loop in tool scheduling, and that explicit sensitive word units are missed in privacy review and implicit inference attack resistance is insufficient. The model can autonomously learn a standard tool calling sequence and a strict review strategy without relying on an independent value network, so that the local agent obtained thereby can autonomously complete end-to-end generation from a natural language query to a safe SQL in the prediction stage, while ensuring high masking recall rate and strong implicit inference resistance, maintaining the semantic integrity and execution accuracy of the SQL statement, and achieving an effective balance between privacy protection and task utility.

Description

Technical Field

[0001] This application relates to the field of security technology, and more specifically, to a training method, privacy protection method, device, storage medium, and program product for a policy model in an intelligent agent. Background Technology

[0002] In Natural Language to Structured Query Language (NL2SQL) tasks, existing technologies primarily employ static decryption techniques to protect sensitive information in user queries. These methods typically de-identify sensitive terms in the original query using rule matching or lightweight named entity recognition models before the request is sent to a large cloud model. For example, they might use masking or replacement strategies to remove explicit identifiers. However, static decryption mechanisms have significant drawbacks: firstly, due to the limited coverage of rules or models, they easily miss sensitive entities in non-standard expressions or long-tail database values, leading to privacy leaks (low recall); secondly, to avoid omissions, some methods employ excessive masking strategies, disrupting the key semantic information maintaining cross-table joins and conditional filtering in the query. This causes the cloud model to generate illusions due to a lack of necessary context, significantly reducing the accuracy of SQL generation (semantic loss).

[0003] To address the aforementioned imbalance between privacy and utility, current research introduces a dynamic privacy protection paradigm based on large language model-based intelligent agents. This approach integrates tools such as pattern filtering, entity linking, and symbolic processing, enabling the agent to plan multi-step privacy protection paths and achieve context-aware, fine-grained desensitization. However, to meet the requirement of data remaining within the domain in highly sensitive areas such as government and finance, this agent must be deployed locally and driven by a small-parameter model. Existing local small-parameter models exhibit problems in practical implementation, including chaotic tool call timing, susceptibility to invalid call loops, and severely insufficient self-censorship capabilities, making it difficult to independently complete the entire task from query desensitization to secure SQL generation. Summary of the Invention

[0004] The purpose of this application is to provide a training method, privacy protection method, device, storage medium, and program product for a policy model in an intelligent agent, so as to enable the model to autonomously learn the standard tool calling order and strict review strategy without relying on an independent value network, improve the mask recall rate and implicit inference resistance of the model, and maintain the semantic integrity and execution accuracy of SQL statements.

[0005] In a first aspect, embodiments of this application provide a method for training a policy model in an intelligent agent. The intelligent agent includes a policy model, a toolset, and a state space. The intelligent agent performs decision-making and optimization for privacy-preserving tasks through a Markov decision process. The toolset includes privacy-preserving tools, privacy-preserving remediation tools, and SQL processing tools. The state space is used to maintain the state of the Markov decision process, and the state includes the original natural language query, database schema information, the current privacy-de-identified mapping set, and intermediate execution results. The method includes: Obtain the training dataset, wherein each training sample in the training dataset includes the original natural language query and the corresponding database pattern information; At each time step, read the current state of the Markov decision process; The policy model is invoked, taking the current state as input, and the action probability distribution is output. The target action is selected from the action space and executed according to the action probability distribution. The action space includes tool invocation action, reflection and review action and task termination action. The tool invocation action is used to invoke the tools in the toolset. The reflection and review action is used to make the strategy model output a security judgment result on the current de-identification result. Update the current state based on the action execution result and determine the current immediate reward; The parameters of the strategy model are updated based on the current instant reward.

[0006] In the aforementioned implementation, by modeling the privacy protection task of the agent as a Markov decision process and uniformly incorporating tool invocation and reflective review actions into the action space, and training the policy model with an immediate reward mechanism, the problems of temporal disorder and infinite loops in tool scheduling, as well as insufficient resistance to explicit sensitive terms and implicit inference attacks in privacy review, which are common in existing technologies, are effectively solved. This allows the model to autonomously learn standardized tool invocation sequences and strict review strategies without relying on an independent value network. The resulting local agent can autonomously complete end-to-end generation from natural language queries to secure SQL during the prediction phase, maintaining the semantic integrity and execution accuracy of SQL statements while ensuring high masked recall and strong implicit inference resistance, thus achieving an effective balance between privacy protection and task utility.

[0007] Optionally, the current immediate reward includes a first type of reward and a second type of reward. The first type of reward is determined based on at least one dimension of the current tool call sequence: compliance, redundancy, and effectiveness. The second type of reward is determined based on the protection integrity of the sensitive information by the current de-identification result. The protection integrity is used to characterize the degree of coverage of displayed sensitive words and / or the degree of resistance to contextual implicit inference attacks.

[0008] In the above implementation process, by setting the first type of reward and the second type of reward, optimization signals are provided for the tool calling ability and reflection ability of the policy model, respectively, and the reinforcement learning alignment training of the local small model is realized.

[0009] Optionally, the first type of reward can be determined in the following way: Call the pre-defined logical compliance detection function to determine whether the current tool call conforms to the preset execution order of privacy protection tasks, and obtain the first judgment result; Call the preset redundancy detection function, determine whether the current tool call is a redundant operation based on the historical call trajectory, and obtain a second judgment result; Call the pre-defined validity check function to determine whether the state has changed effectively after the current tool call is executed, and obtain the third judgment result; The first judgment result, the second judgment result, and the third judgment result are weighted and fused to obtain the first type of reward.

[0010] In the above implementation process, through the above-mentioned dimension-by-dimensional evaluation and weighted fusion, the first type of reward can reflect the standardization of tool calls in a fine-grained manner, providing a clear direction for optimization for the strategy model: the model will be encouraged to select actions that are in the correct order, do not repeat, and can effectively advance the task, thereby ultimately learning to autonomously complete privacy-preserving NL2SQL tasks.

[0011] Optionally, the second type of reward can be determined in the following way: Obtain the mask recall rate, which is the ratio of the number of sensitive words actually masked in the current de-identification result to the total number of sensitive words that should be masked; Invoke the preset adversarial inference model, take the current desensitization result as input, and output the number of mask tags that were not successfully recovered; A re-identification score is obtained based on the number of mask tags that were not successfully recovered, wherein the re-identification score is the proportion of the number of mask tags that were not successfully recovered to the total number of mask tags; The mask recall rate and the re-identification score are weighted and fused to obtain the second type of reward.

[0012] In the above implementation, by simultaneously introducing masked recall and re-identification score, masked recall directly quantifies whether sensitive words are completely masked, effectively guiding the policy model to avoid explicit leakage risks caused by missed entities, thus solving the problem of low recall in existing static methods. Re-identification score, through an adversarial inference model simulating the attacker's perspective, evaluates whether masked words can be recovered from the context, thereby incentivizing the model to not only perform surface masking but also sever semantic association clues, enhancing its resistance to implicit inference attacks. After weighted fusion of the two, the second type of reward provides the policy model with comprehensive and fine-grained optimization signals, forcing the model to pursue both high coverage and high resistance to inference during training, avoiding the shortcomings of a single metric.

[0013] Optionally, the current immediate reward includes a third type of reward, which is determined based on the correctness of the execution of the final generated SQL statement. The method further includes: For each training sample, the steps from reading the current state to determining the current immediate reward are repeated until the termination condition is met, resulting in an execution trajectory and the determination of the third type of reward.

[0014] In the above implementation process, the third type of reward, together with the first and second types of inter-step rewards, constitutes a hierarchical optimization signal: inter-step rewards guide the model to operate in a standardized manner and ensure privacy, while result rewards ensure the practicality of the final output, thereby comprehensively improving the performance of small parameter models in privacy-preserving NL2SQL tasks.

[0015] Optionally, updating the parameters of the strategy model based on the current instant reward includes: For each training sample, obtain multiple execution trajectories; Calculate the total instant reward for each of the multiple execution trajectories, wherein the total instant reward is obtained by weighted fusion of the first type of reward, the second type of reward, and the third type of reward; Calculate the mean and standard deviation of rewards based on the total immediate rewards for each execution reward; Calculate the relative advantage value of each execution reward based on the mean reward and the standard deviation of the reward; The objective function of the group relative strategy optimization algorithm is used to update the parameters of the strategy model based on the relative advantage value.

[0016] In the above implementation process, the group relative policy optimization algorithm was used to achieve aligned training of the policy model, enabling it to have standardized tool calling capabilities and strict privacy review capabilities without relying on large cloud models and independent value networks.

[0017] Optionally, the objective function is expressed as:

[0018] in, The parameters of the policy model are represented by , and q represents the current training sample. For the empirical distribution of the training data, This means taking the expectation over all training samples; This represents the probability ratio between the current strategy model and the old strategy model for the same execution trajectory. This is the clipping threshold; The function is used to achieve the cropping effect; For the clipping function, when Exceeding When the range is defined, it is truncated to the boundary value; For the first The relative advantage value of each execution trajectory; The KL divergence penalty coefficient is used to control the current policy model. Compared with the reference strategy model The severity of the penalty for the degree of deviation between them; The KL divergence between the current policy model and the reference policy model; This represents the i-th execution trajectory.

[0019] In the above implementation process, the objective function integrates a pruning mechanism and a KL divergence penalty term. The pruning mechanism limits the importance sampling ratio during policy updates to a preset range, effectively preventing excessive single-step update amplitude from causing training instability or policy collapse. The KL divergence penalty term constrains the deviation between the current policy and the initial reference policy (such as a model that has undergone supervised fine-tuning), ensuring that the model does not lose the language understanding and generation capabilities learned in the pre-training stage while pursuing high rewards, thus avoiding catastrophic forgetting.

[0020] Secondly, embodiments of this application provide a privacy protection method applied to an intelligent agent. The intelligent agent includes a policy model, a toolset, and a state space. The intelligent agent makes decisions and optimizes privacy protection tasks through a Markov decision process. The toolset includes privacy protection tools, privacy protection remediation tools, and SQL processing tools. The state space is used to maintain the state of the Markov decision process, and the state includes the original natural language query, database schema information, the current privacy-de-identified mapping set, and intermediate execution results. The method includes: Receive natural language queries and database schema information; At each time step, read the current state of the Markov decision process; The policy model is invoked, taking the current state as input, and the probability distribution of the current action is output. Based on the current action probability distribution, the action with the highest probability is selected from the action space and executed. The action space includes tool invocation action, reflection and review action, and task termination action. The tool invocation action is used to invoke the tools in the toolset. The reflection and review action is used to make the strategy model output a security judgment result on the current de-identification result. Repeat the above steps until the SQL statement is output or the preset maximum number of iterations is reached to obtain the final SQL statement.

[0021] In the above implementation process, the local intelligent agent can autonomously complete the conversion from natural language query to secure SQL without relying on a large cloud model, while meeting the privacy compliance requirements of data not leaving the domain.

[0022] Optionally, if the currently selected action is a reflection and review action and the agent's security assessment result for the current de-identification result is "not approved", the method further includes: When selecting subsequent actions, privacy protection and repair tools should be allowed to be called, while SQL processing tools should be prohibited, until the security assessment result is passed.

[0023] In the above implementation process, through this action constraint mechanism, the prediction process can effectively ensure that SQL statements are only allowed to be generated after the de-identification result is confirmed as safe by the model itself (approved by the review), thereby eliminating the risk of privacy leakage caused by incomplete de-identification.

[0024] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the method provided in the first or second aspect above are performed.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the methods provided in the first or second aspect above.

[0026] Fifthly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the steps in the methods provided in the first or second aspects above.

[0027] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating a method for training a policy model in an agent, as provided in an embodiment of this application; Figure 2 This application provides a training process based on a population relative policy optimization algorithm. Figure 3 A flowchart illustrating a privacy protection method provided in this application embodiment; Figure 4 A structural block diagram of a training device for a policy model in an intelligent agent, provided in an embodiment of this application; Figure 5 A structural block diagram of a privacy protection device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device for executing a training method or privacy protection method for a policy model in an intelligent agent, as provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0031] It should be noted that the terms "system" and "network" in the embodiments of this invention can be used interchangeably. "Multiple" refers to two or more; therefore, in the embodiments of this invention, "multiple" can also be understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0032] It should also be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0033] This application provides a method for training a policy model in an intelligent agent. Applied to an intelligent agent, this method models the agent's privacy protection task as a Markov decision process and uniformly incorporates tool invocation and reflection / review actions into the action space. Combined with an immediate reward mechanism, the method trains the policy model, effectively solving the problems of temporal disorder and infinite loops in tool scheduling, as well as the omission of explicit sensitive terms and insufficient resistance to implicit inference attacks in privacy review. This allows the model to autonomously learn standardized tool invocation sequences and strict review strategies without relying on an independent value network. The resulting local intelligent agent can autonomously complete end-to-end generation from natural language queries to secure SQL during the prediction phase. While ensuring high masked recall and strong implicit inference resistance, it maintains the semantic integrity and execution accuracy of SQL statements, achieving an effective balance between privacy protection and task utility.

[0034] To facilitate understanding, let's first introduce the intelligent agent involved in this solution. The intelligent agent is an automated program deployed in a local secure environment (such as a user's internal server or trusted execution environment). The intelligent agent is responsible for receiving user natural language queries, making autonomous decisions, and performing privacy protection and SQL generation tasks, without relying on cloud models.

[0035] An intelligent agent may include a policy model, a toolset, and a state space. The agent makes decisions and optimizes privacy-preserving tasks through Markov decision processes. The toolset includes privacy-preserving tools, privacy-preserving remediation tools, and SQL processing tools. The state space is used to maintain the state of the Markov decision process, which includes the original natural language query, database schema information, the current privacy-de-identified mapping set, and intermediate execution results.

[0036] The policy model is a small-parameter neural network model that can be fine-tuned using reinforcement learning based on a pre-trained language model. Its function is to output the probability distribution of each action in the action space based on the current input state, such as under what circumstances which tool should be invoked or which type of review should be performed.

[0037] Privacy protection tools, such as entity desensitization tools, can replace sensitive words and characters (such as personal names, place names, ID numbers, specific table names, and column names) in natural language queries with irreversible symbols or placeholders and record the mapping relationship.

[0038] Privacy protection and remediation tools, such as supplementary desensitization tools, can supplement the masking of missing sensitive words and characters based on review feedback.

[0039] SQL processing tools, such as symbolic SQL generation tools, can convert natural language queries into structured SQL statements (where sensitive words have been replaced with symbols) based on the current de-identification mapping.

[0040] In this scheme, the privacy-preserving NL2SQL task of the agent is modeled as a Markov decision process. The agent is given a state space to maintain the state in the Markov decision process. This state is a quadruple. Where: Q represents the original natural language query (which remains unchanged throughout the task), and D represents database schema information, including table names, column names, primary and foreign key relationships, etc. (which remain unchanged throughout the task). This represents the privacy-de-identified mapping set for the current time step, recording the correspondence between the original sensitive words and mask symbols (initially empty, gradually filled in as the de-identification operation progresses). This indicates intermediate execution results, such as the judgment information output from the previous review, diagnostic feedback, or temporarily generated intermediate SQL representations (initially empty).

[0041] Please refer to Figure 1 , Figure 1 A flowchart illustrating a method for training a policy model in an agent, as provided in this application embodiment, is included. This method, applied to an agent, comprises the following steps: Step S110: Obtain the training dataset.

[0042] During training, the agent first retrieves the training dataset from local storage or an internal data interface. Each training sample in the training dataset includes the original natural language query Q and the corresponding database schema information D. For example, a training sample could be: a query (such as querying all records in the races table whose name column contains 'final') and database schema information (such as tables containing 'races' (fields id, name, date), 'results', etc.).

[0043] Step S120: At each time step, read the current state of the Markov decision process.

[0044] For each training sample, the agent first calls the state space to obtain the state at the current time step t. Initially, t=0. Empty If empty, Q and D remain the original inputs.

[0045] In subsequent time steps t>0, the state space stores the updated state obtained after the previous action was executed. and For example, suppose that at t=1, the agent invokes a de-identification tool to replace the sensitive keyword "final" in the query with "<mask_1> ",but It will record the mapping relationship {"Finals": "<mask_1> "}, It may record the anonymized query string (i.e., the current anonymized result).

[0046] Step S130: Invoke the policy model, taking the current state as input, and output the action probability distribution.

[0047] The agent will state the current state Serialization into an input format acceptable to the policy model typically involves converting the aforementioned Q, D, ... , Concatenate the data into text or structured encoding, and then feed it into the local policy model. The policy model, after forward computation, outputs the probability distribution of all actions in the action space. The action space represents the types of actions an agent can execute at each time step, including tool invocation actions, reflection and review actions, and task termination actions. Tool invocation actions are used to invoke tools from the toolset, such as specifying the specific tool identifier and its required parameters, and generating corresponding invocation commands. This covers all available tools in the MaskSQL toolset, including the aforementioned privacy protection tools, privacy protection remediation tools, and SQL processing tools. Reflection and review actions enable the policy model to output a security judgment result for the current de-identification result. For example, triggering the policy model to perform a self-evaluation of the current de-identification result from a reviewer's perspective, outputting a security judgment result (e.g., pass or fail) and corresponding diagnostic information. Task termination actions stop task execution, indicating that the agent considers the entire privacy protection and SQL generation task complete, can end the interaction, and output the final result. This modeling approach, which incorporates reflection and review into the action space, allows the review step and the tool invocation step to be within the same optimization framework.

[0048] For example, the probability distribution of the output actions can be as follows: calling privacy protection tools (de-identification tools): 0.65; calling privacy protection repair tools: 0.05; calling SQL generation tools: 0.05; reflection and review actions: 0.20; task termination actions: 0.05.

[0049] This probability distribution reflects the model's current judgment on "which action is more appropriate" in the current state. Because the training phase needs to explore different possibilities, the probability distribution generally has a certain entropy (not completely concentrated on a single action).

[0050] Step S140: Select the target action from the action space according to the action probability distribution and execute it.

[0051] The agent selects a specific action by randomly sampling (e.g., using multinomial sampling or temperature sampling) based on the action probability distribution obtained from the above steps. Then the action can be performed. Specifically, this can be divided into the following situations: If a tool call action is selected, the system intelligently identifies the name of the called tool (e.g., "de-identification tool") and necessary parameters (e.g., the entity type requiring masking), and then calls the corresponding function in the toolset. For example, the de-identification tool will find all matching sensitive keywords (e.g., "finals", "races.name", etc.) in the current query Q, replace them with irreversible symbols (e.g., [MASK_1], [MASK_2]), and update the mapping set. .

[0052] If the reflexive review action is selected, the agent takes the anonymized query, database schema, and existing mapping relationships in the current state as input, allowing the policy model to output a security assessment result. In practice, the model can generate a special marker (such as PASS or FAIL) and diagnostic text. For example, the model might output: "FAIL, table name 'races' still appears in plaintext in the query." This assessment result will be written into the intermediate results. .

[0053] If the task termination action is selected, the agent immediately stops the time step loop and uses the existing final SQL (if it has been generated) or an empty result in the current state as the output for this step.

[0054] Step S150: Update the current state based on the action execution result and determine the current instant reward.

[0055] After the action is completed, the agent performs two operations: updating the state and determining the current immediate reward.

[0056] The state transition function is determined by the result of the action execution; when the model is in state... Select the tool to invoke action Then, the tool will perform calculations based on the current global state to generate a new state. When the reflective review action is selected, the output of the review process will be written into the intermediate results. Similarly, a new state is generated. Since the execution of the intelligent agent and the review action are driven by the same model, the uniformity of state transitions is guaranteed.

[0057] Specifically, modify the state space based on the result of the action execution. and To obtain a new state .For example: The desensitization tool executed successfully; update. and in The query text is recorded after being anonymized.

[0058] Repair tool execution, supplement Correct any omissions in the desensitization process.

[0059] Once the review process is complete, the judgment result will be written to... .

[0060] The SQL generation tool is executed, in The generated SQL statements are stored in [the database].

[0061] The aforementioned immediate reward is a reward value calculated according to predefined reward rules after each action is performed, used to evaluate the quality of the action.

[0062] Step S160: Update the parameters of the strategy model based on the current instant reward.

[0063] It should be noted that in actual training, step S160 does not update the model immediately after each step is completed. Instead, it can be performed in a batch processing manner as in reinforcement learning: the agent first collects multiple complete trajectories for the same training sample (each trajectory contains multiple time steps), calculates the total reward for each trajectory, and then updates the parameters of the policy pattern based on the total reward.

[0064] After the update is completed, the state of the policy model is saved and used for the next training sample or the next iteration.

[0065] In the aforementioned implementation, by modeling the privacy protection task of the agent as a Markov decision process and uniformly incorporating tool invocation and reflective review actions into the action space, and training the policy model with an immediate reward mechanism, the problems of temporal disorder and infinite loops in tool scheduling, as well as insufficient resistance to explicit sensitive terms and implicit inference attacks in privacy review, which are common in existing technologies, are effectively solved. This allows the model to autonomously learn standardized tool invocation sequences and strict review strategies without relying on an independent value network. The resulting local agent can autonomously complete end-to-end generation from natural language queries to secure SQL during the prediction phase, maintaining the semantic integrity and execution accuracy of SQL statements while ensuring high masked recall and strong implicit inference resistance, thus achieving an effective balance between privacy protection and task utility.

[0066] In this scheme, to provide fine-grained optimization signals for the policy model's tool-calling and reflective capabilities, immediate rewards can include a first type of reward and a second type of reward, referred to as inter-step rewards. These rewards are triggered and calculated immediately at key decision nodes. These two types of rewards can be understood as follows: during the agent's execution of a complete trajectory, evaluation is not based solely on the single reward of whether the final SQL execution is successful. Instead, an immediate reward signal is triggered and calculated immediately at key decision nodes (such as after each tool call or after the de-identification operation). The purpose of inter-step rewards is to provide the policy model with denser and finer-grained feedback, enabling it to clearly know "whether the current step was done well or poorly," thereby solving the difficulty of credit allocation under traditional sparse rewards.

[0067] The first type of reward, triggered immediately after each tool invocation action, is used to evaluate whether the agent's chosen tool invocation behavior at that time step is compliant. This first type of reward is determined based on at least one of the following dimensions: compliance of the current tool invocation sequence (whether the current tool invocation conforms to the logical order preset by the privacy-preserving NL2SQL task), redundancy (whether the current invocation is a repetitive and useless operation), and effectiveness (whether the current invocation produces an actual state change). These three dimensions can be used individually or arbitrarily combined and weighted.

[0068] The second type of reward is triggered immediately after the de-identification operation is completed (i.e., the agent invokes privacy protection tools and all sensitive words that should be masked have been processed). This reward is used to evaluate the integrity of the current de-identification result in protecting sensitive information. In other words, the second type of reward is determined based on the integrity of the current de-identification result in protecting sensitive information. This integrity characterizes the coverage of explicitly displayed sensitive words and / or the resistance to implicit inference attacks. Explicit coverage refers to the proportion of masked sensitive words out of all masked sensitive words (also known as masked recall); implicit inference resistance refers to whether the masked words can be inferred from contextual semantics and database pattern information (usually using adversarial inference models to calculate re-identification scores). These two sub-dimensions can be used as two independent reward components or combined into a single comprehensive reward.

[0069] By implementing the above methods and setting first and second types of rewards, optimization signals are provided for the tool invocation and reflection capabilities of the policy model, thereby achieving reinforcement learning alignment training of the local small model.

[0070] Regarding the three dimensions of the first category of rewards, sequential compliance refers to whether the order in which tools are invoked conforms to the inherent logic of the privacy-preserving NL2SQL task. A reasonable order is typically: first, use privacy-preserving tools to de-identify sensitive terms; then, perform a reflexive review to check the completeness and security of the de-identification; if the review passes, then use the SQL generation tool; if the review fails, then use privacy-preserving remediation tools to supplement the mask. Invoking the SQL generation tool directly before de-identification is complete or the review has passed is considered a violation of sequential compliance.

[0071] Redundancy refers to the repeated use of the same tool without generating any new and valid state changes. For example, calling the same desensitization tool again after all sensitive keywords have been desensitized; or repeatedly calling an SQL repair tool after SQL has been successfully generated. Redundant calls waste computational resources and can cause tasks to enter infinite loops, therefore they should be given low or negative rewards.

[0072] Validity refers to whether a tool's invocation causes changes in the state space. For example, if a de-identification tool successfully discovers new sensitive words and adds mapping relationships, it is considered valid; if the tool fails to discover any sensitive words and the mapping set remains identical to before, it is considered invalid. Validity checks prevent agents from performing meaningless operations.

[0073] After each tool invocation action (such as selecting a tool from the action space and executing the corresponding tool), the agent calculates the first type of reward, and the calculation process is as follows: Call the pre-defined logical compliance detection function to determine whether the current tool call conforms to the preset execution order of privacy protection tasks, and obtain the first judgment result; Call the preset redundancy detection function, determine whether the current tool call is a redundant operation based on the historical call trajectory, and obtain a second judgment result; Call the pre-defined validity check function to determine whether the state has changed effectively after the current tool call is executed, and obtain the third judgment result; The first, second, and third judgment results are weighted and combined to obtain the first type of reward.

[0074] Among them, the logical compliance detection function It is a predefined judgment module that takes the current tool invocation action as input. and current state (Including historical action sequences and intermediate results), output a first judgment result indicating whether it conforms to the preset execution order. This result is usually a Boolean value (0 for non-compliance, 1 for compliance), but it can also output continuous values ​​(such as decimals between 0 and 1, representing the degree of compliance). For example, the agent reads intermediate results from the current state. The system also includes a historical action sequence to determine whether the current tool call conforms to the preset execution order rules. If it does, the output value is 1; otherwise, it is 0.

[0075] Specifically, the intelligent agent executes the tool invocation action. Then, the compliance check function is immediately invoked. Internally, the function maintains a rule set based on a finite state machine, or a set of conditional statements, with the following logic: First, read the current state. intermediate results Obtain the output judgment of the most recent reflection and review action (if any). At the same time, read the historical action sequence to know whether de-identification tools, repair tools, SQL generation tools, etc. have been executed before.

[0076] Then, determine whether the compliance requirements are met based on the current tool type: If the current tool is an SQL generation tool: It is required that a data masking operation has already been performed in the preceding steps (i.e.) If the condition is not empty or has been marked as desensitized, and the most recent reflective review was rated "passed". If neither condition is met, output: =0; otherwise output =1.

[0077] If the current tool is a repair tool: It requires that at least one reflective review in the preceding steps has been deemed "failed," and that the SQL generation tool has not been called yet. If this condition is not met, output 0; otherwise, output 1.

[0078] If the current tool is a data masking tool or any privacy protection tool: it is allowed to be called at any stage, but if the data masking tool is called again after the SQL is generated, it is considered out of order and the output is 0. Otherwise, the output is 1.

[0079] Redundancy detection function It is also a judgment module, which takes the current tool invocation action as input. and historical call trajectory (Records all previously executed tool calls and their effects), and outputs a second judgment result indicating whether the current call is a redundant operation. A redundant operation refers to a behavior that has the same function as a previous call and does not result in a valid state change, such as repeatedly calling the same de-identification tool without finding new sensitive words, or repeatedly calling the SQL repair tool after the SQL has been successfully generated. The function output is usually a Boolean value (0 for redundancy, 1 for non-redundancy), but can also output continuous penalty values ​​based on the number of repetitions and the interval steps. For example, the agent queries the tool call sequence recorded in the historical trajectory. If the current tool is found to be exactly the same as a previous call, and the privacy de-identification mapping set M and intermediate results I in the state space have not changed since that call, it is judged as a redundant call, and the output value is 0; otherwise, it is 0.

[0080] Specifically, the function is based on historical call patterns. Determine if the current call is a redundant operation. The implementation is as follows: Get the type of the currently invoked tool and the invocation parameters (for example, a de-identification tool may have the parameter "mask name"); In historical call trajectory Search forward to see if a tool call of the same type and with the same parameters has occurred before; If they exist, further examine whether the privacy-de-identified mapping set M and intermediate result I in the state space have changed between that call and the current time step. If no changes have occurred (e.g., no new mappings added, no new SQL generated), then the current call is determined to be redundant, and output... =0; If no identical call exists in the history, or if a call did exist but the state changed during that time (indicating that the environment has changed since the last call and a subsequent call may have new effects), then it is determined to be non-redundant and outputs the result. =1.

[0081] To avoid overly strict judgments, a tolerance threshold can be set: for example, the same tool can be called a maximum of 2 times consecutively, and exceeding this is considered redundant.

[0082] Validity detection function It is also a judgment module, which takes the current tool invocation action as input. State before execution and the state after execution The system outputs a third-party judgment result indicating whether a valid state change has occurred. A valid state change refers to a change in the privacy-de-identifying mapping set M or the intermediate result set I. For example, if the de-identifying tool successfully replaces a sensitive word with a mask symbol and adds a mapping relationship, or if the SQL generation tool outputs a new SQL statement, these are all valid changes. If M and I are the same after the tool is executed as before, it is considered invalid. The output is usually 0 (invalid) or 1 (valid). This allows the agent to compare the state differences before and after executing the current tool. Specifically, it records the mapping set before execution. and intermediate results After executing the tool, the following results were obtained: and .if or If the condition is met, the output value is 1; otherwise, the output value is 0.

[0083] After obtaining the above three judgment results (each value is 1 or 0), they can be weighted and merged according to preset weights to obtain the first type of reward. A simple way to integrate them is by equal weighting:

[0084] Of course, different weights can also be set (for example, if more emphasis is placed on order compliance, its weight is set to 0.5, and the rest to 0.25). The merged result... The value range is usually between [0,1], but in some serious violations (such as calling SQL generation before de-identification), a negative penalty value (such as -0.5) can be directly applied. This scheme can use continuous rewards to provide detailed gradient information.

[0085] Through this dimensional evaluation and weighted fusion, the first type of reward can reflect the standardization of tool calls in a fine-grained manner, providing a clear direction for optimization of the strategy model: the model will be encouraged to select actions that are in the correct order, do not repeat, and can effectively advance the task, thereby ultimately learning to autonomously complete privacy-preserving NL2SQL tasks.

[0086] Regarding the calculation of the second type of reward, after the agent performs a de-identification operation (such as calling a privacy protection tool or repair tool), if all detected masked sensitive words have been processed (i.e., de-identification is complete), the calculation of the second type of reward is triggered immediately. Detecting "de-identification complete" can be achieved by judging the de-identification mapping set in the current state. This can be achieved by covering all field values ​​in the database or entities in the query that are considered sensitive; alternatively, a list of sensitive keywords can be pre-defined, and when each keyword in the list is... When there is a mapping, it is considered complete. The calculation process is as follows: Get the mask recall rate, which is the ratio of the number of sensitive words that are actually masked in the current de-identification results to the total number of sensitive words that should be masked. Call the pre-built adversarial inference model, take the current desensitization result as input, and output the number of mask tags that were not successfully recovered; The re-identification score is obtained based on the number of mask tags that were not successfully recovered. The re-identification score is the proportion of the number of mask tags that were not successfully recovered to the total number of mask tags. The mask recall rate and re-identification score are weighted and fused to obtain the second type of reward.

[0087] Masking Recall (MR) represents the degree of display coverage and can be referred to as a privacy and security reward. The computation is triggered during the complete desensitization phase of the model, before the desensitization results are sent to the cloud. The agent obtains the set of all sensitive words that need protection in the current natural language query Q. (This can be obtained through a pre-built sensitive word dictionary or named entity recognition model). Then, the set of words that have already been mapped is statistically analyzed within these word groups. The number N of overlays (i.e., those that have been replaced with mask symbols) masked The formula for masked recall is:

[0088] N is the total number of sensitive words that should be masked. masked This represents the number of sensitive words actually masked in the current de-identification results. The mask recall rate ranges from [0,1], with a higher value indicating more thorough de-identification. A value of 1 indicates that all sensitive words have been successfully masked.

[0089] The re-identification score represents the degree of resistance and can be referred to as the privacy inference resistance reward. The calculation is triggered synchronously with the masked recall rate mentioned above, aiming to evaluate whether the de-identification results can resist implicit inference attacks based on contextual semantic clues. The Re-identification Score (RI) is calculated by simulating an attack on the masking problem, that is, providing the model with a masked natural language problem and database pattern, requiring it to infer the words before the mask, and then calculating the percentage of masked tokens that cannot be successfully inferred.

[0090] Specifically, the agent invokes a pre-built adversarial inference model (which can be a slightly larger local language model or a rule-based re-identification engine). The input includes: a de-identified natural language query Q′ (where sensitive words have been replaced with placeholders, such as [MASK_1], [MASK_2]) and database schema information D (containing structural information such as the actual table names and column names). The adversarial inference model's task is to attempt to infer the original real word corresponding to each masked symbol.

[0091] For each mask tag j, if the predicted value given by the adversarial inference model differs from the true value, or if the model cannot give a definite prediction, then the resistance is considered successful. Let the total number of mask tags be... The number of markers that were not successfully resisted is The re-identification score is then defined as:

[0092] The value range is [0,1]. A value of 1 indicates that all mask tags cannot be recovered by the adversarial inference model, that is, it has perfect implicit resistance.

[0093] To improve computational efficiency, the adversarial inference model can be pre-trained offline with its parameters fixed during training, and it does not participate in updating the policy model. It is only used to provide re-identification scores as a reward signal.

[0094] Then, the mask recall rate and re-identification score can be combined to obtain the second type of reward. The fusion method can be weighted summation:

[0095] in, and To preset the weights, they can typically be set to 0.5 each, indicating that both are equally important. Alternatively, a product form can be used, i.e. This ensures that the reward is high only when both are high.

[0096] During training, the agent executes a tool call and triggers the first type of reward each time. And triggering a second type of reward upon completion of desensitization. These immediate rewards are temporarily stored. Ultimately, the total reward for a complete trajectory is a weighted aggregation of all inter-step rewards (Type 1 + Type 2). In this way, the policy model can obtain fine-grained feedback from each key decision node: disordered tool invocation will immediately reduce... Desensitization omissions or those easily inferred will immediately decrease This dense reward mechanism effectively solves the difficulty of credit allocation caused by relying solely on the sparse reward of the final execution result in traditional reinforcement learning, enabling small-parameter models to gradually learn standardized tool scheduling and strict privacy review.

[0097] In the above implementation, by simultaneously introducing masked recall and re-identification score, masked recall directly quantifies whether sensitive words are completely masked, effectively guiding the policy model to avoid explicit leakage risks caused by missed entities, thus solving the problem of low recall in existing static methods. Re-identification score, through an adversarial inference model simulating the attacker's perspective, evaluates whether masked words can be recovered from the context, thereby incentivizing the model to not only perform surface masking but also sever semantic association clues, enhancing its resistance to implicit inference attacks. After weighted fusion of the two, the second type of reward provides the policy model with comprehensive and fine-grained optimization signals, forcing the model to pursue both high coverage and high resistance to inference during training, avoiding the shortcomings of a single metric.

[0098] To improve the training accuracy of the model, the immediate reward may also include a third type of reward. The third type of reward is determined based on the correctness of the execution of the final generated SQL statement. In the above training process, for each training sample, the steps from reading the current state to determining the current immediate reward can be repeatedly executed, that is, steps S120-S150 can be repeatedly executed until the termination condition is reached, an execution trajectory can be obtained, and the third type of reward can be determined.

[0099] The reward system of this scheme may include four reward categories, such as the step-by-step reward mentioned above: Category 1 Rewards The second type of reward and And result-based rewards: the third type of reward . and Used for training in privacy protection review capabilities; Used for training in tool invocation capabilities; This ensures the overall goal orientation. The triggering timing, evaluation content, and training capability dimensions of each reward function are shown in the table below.

[0100]

[0101] Third category of rewards A computation is triggered at the end of the entire interaction trajectory to evaluate the correctness of the execution of the final generated SQL statement. This reward provides the policy model with an overall quality signal for task completion, ensuring that the model not only focuses on the standardization of intermediate steps (first-type reward) and the integrity of privacy protection (second-type reward), but also guarantees that the final output SQL can correctly answer the user's question.

[0102] SQL execution correctness refers to executing the generated SQL statement in a real or simulated database environment and comparing the execution result with the pre-labeled real labels (the execution result or expected output of standard SQL) in the training samples. The comparison results are usually divided into three categories: successful execution and completely consistent results, successful execution but inconsistent results, and execution failure (such as syntax errors, referencing non-existent tables or columns, etc.).

[0103] Regarding the termination condition, there are two possibilities: first, the agent selects the task termination action at a certain time step, actively ending the task; second, the number of consecutive steps reaches a preset threshold T. max If the trajectory still has not ended after 20 steps (e.g., 20 steps), then the trajectory will be forcibly terminated.

[0104] During the training phase, steps S120-S150 are repeated multiple times for each training sample to generate a complete execution trajectory. After the trajectory terminates, the process starts from the final state. intermediate results Extract the final generated SQL statement (if any). If there is no valid SQL statement due to timeout, error, or lack of SQL generation, the execution is considered to have failed. Then, the agent calls the pre-set SQL execution and comparison module to calculate the third type of reward.

[0105] Third category of rewards The value of is divided into three levels based on the SQL execution result, to guide the model to pursue outputs that are not only syntactically correct but also semantically matched. The specific calculation formula is as follows:

[0106] After obtaining all inter-step rewards and outcome rewards for an execution trajectory, the total immediate reward for that trajectory can be calculated using the following formula. :

[0107] in, This includes the first type of reward. The second type of reward and , This is the inter-step reward balancing factor, typically ranging from 0.3 to 0.5, used to adjust the weighting ratio between inter-step rewards and the final SQL execution result reward. Through this aggregation, the model can obtain fine-grained feedback at each step without the importance of the final result being overwhelmed by excessive accumulation of inter-step rewards.

[0108] Through the above implementation, the third type of reward, together with the first and second types of inter-step rewards, constitutes a hierarchical optimization signal: inter-step rewards guide the model to operate in a standardized manner and ensure privacy, while result rewards ensure the practicality of the final output, thereby comprehensively improving the performance of small parameter models in privacy-preserving NL2SQL tasks.

[0109] This reward design based on standardized evaluation metrics aligns the optimization direction during the training phase with the final evaluation criteria: and The improved scrutiny capabilities of the agent thus optimize mask recall (MR) and re-identification score (RI). This ensures that each tool invocation step can be executed normally and This design directly optimizes execution accuracy, ensuring that the policy improvements gained from reinforcement learning training are directly reflected in the final evaluation metrics, thus avoiding the bias between agent rewards and the true objective.

[0110] After obtaining the immediate reward using the above method, a group relative policy optimization algorithm can be used to optimize the policy model. Alignment training is performed. The swarm relative policy optimization algorithm effectively reduces reward variance by generating multiple trajectories for the same problem and performing relative scoring within the group. The training process of the swarm relative policy optimization algorithm is as follows: Figure 2 As shown. The specific steps are as follows: For each training sample, obtain multiple execution trajectories; Calculate the total immediate reward for each execution trajectory among multiple execution trajectories. The total immediate reward is obtained by weighted fusion of the first type of reward, the second type of reward, and the third type of reward. Calculate the mean and standard deviation of rewards based on the total immediate rewards for each execution reward; Calculate the relative advantage value of each execution reward based on the mean reward and the standard deviation of the reward; The objective function of the group relative policy optimization algorithm is used to update the parameters of the policy model based on the relative advantage value.

[0111] Among them, Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that does not rely on an independent value network. Its core idea is: for the same input problem, multiple complete execution trajectories are independently sampled from the current policy model to form a trajectory group; then, the relative advantage value of each trajectory is obtained by standardizing based on the total reward actually obtained by each trajectory within the group; finally, the policy parameters are updated using a pruning mechanism and a KL divergence penalty objective function. This method replaces the additionally trained value network required in traditional algorithms (such as Proximal Policy Optimization (PPO)) with intra-group comparisons, reducing training complexity and resource requirements.

[0112] During training, for each training sample in the training dataset (natural language query Q and database schema information D), the agent repeatedly executes the complete trajectory generation process G times (e.g., G=4). Each time, the policy model uses the current parameters θ, and due to random sampling (rather than greedy) in action selection, the same sample can generate multiple different trajectories. These trajectories may exhibit different tool call orders, review results, and final SQL quality. The set of G execution trajectories... These trajectories may exhibit different tool invocation behaviors, for example: trajectories Entities were missed during the desensitization process; trajectory Complete desensitization but redundant tool call traces exist; traces The entire generation, review, and repair loop has been completed. To control training overhead, this scheme allows setting a truncation threshold for the trajectory length. When the number of interaction rounds for a certain trajectory exceeds... At that time, forcibly terminate the trajectory and... Set to 0 to penalize excessively long, ineffective explorations.

[0113] Initially, an empty set is created to store the G trajectories to be generated. For the generation of each trajectory, steps S120 to S150 (see the aforementioned implementation methods for details) are called independently until the trajectory terminates, and all inter-step rewards in the trajectory (the first type of reward for each step, the second type of reward when desensitization is completed) and the third type of reward are recorded. Finally, each trajectory and its reward list are stored in the trajectory group.

[0114] For each trajectory in the trajectory group, extract all recorded inter-step rewards and outcome rewards (each trajectory corresponds to one outcome reward). The total reward is aggregated according to a preset formula. That is, the above The specific calculation formula is as described in the aforementioned embodiment.

[0115] The agent's total reward for all G trajectories in the current trajectory group Calculate the mean μ and standard deviation σ:

[0116]

[0117] Then, for each execution trajectory i, its relative advantage value Defined as:

[0118] This value standardizes the rewards within the group, making the group... The mean is 0, and the standard deviation is 1. Relative advantage value This method directly replaces the advantage function estimated by the value network in traditional reinforcement learning algorithms, used to evaluate the superiority of a trajectory relative to other trajectories in the same group. This advantage estimation method based on within-group statistics means that the evaluation of the reward signal no longer depends on an independent value network, but rather determines the superiority of each trajectory through a relative comparison between multiple sampled trajectories under the same problem. Even on difficult samples where absolute rewards are generally low, this method can still effectively distinguish relatively superior policy behaviors.

[0119] After obtaining each set of trajectories and their relative advantage values, the agent calculates the policy gradient using the following objective function and updates the policy model parameters θ along the gradient ascent direction. To prevent the model from deviating excessively from its pre-trained language capabilities when pursuing high rewards, a pruning mechanism is introduced to constrain the policy update magnitude and the current policy. Reference Strategy The KL divergence between the two policies is used as a penalty to prevent excessive policy shift. The final optimization objective function is defined as:

[0120] in, The parameters of the policy model are represented; q represents the current training sample. For the empirical distribution of the training data, This means taking the expectation over all training samples; It is the probability ratio of the current policy model to the old policy model for the same execution trajectory output, used to measure the degree of change in the probability assigned to the same output before and after the policy update; This is the pruning threshold, used to limit the magnitude of a single policy update, and is usually set to 0.2; The function is used to achieve the cropping effect when Exceeding In this case, the latter option is chosen, thus forming a trust region constraint; For the clipping function, when Exceeding When the range is defined, it is truncated to the boundary value; For the first The relative advantage value of each execution trajectory; The KL divergence penalty coefficient is used to control the current policy model. Compared with the reference strategy model The severity of the penalty for the degree of deviation between them, the greater the Values ​​tend to make the trained policy closer to the reference policy; smaller values... The value allows the strategy to explore a wider range, and is usually set to 0.01 to 0.1; The KL divergence between the current policy model and the reference policy model encourages the updated policy not to deviate too far from the initial reference policy (usually a model that has been fine-tuned under supervision) in order to maintain the basic ability of language generation. This represents the i-th execution trajectory. It is typically set as an initial policy model that has been fine-tuned under supervision before reinforcement learning training begins.

[0121] In actual calculations, for each trajectory in the trajectory group, the agent first calculates the importance sampling ratio. Then, the clipped objective term is calculated, and β multiplied by the KL divergence is subtracted. Finally, the objective values ​​of all trajectories are averaged to obtain the final objective for that batch (multiple trajectories of a single sample). The agent uses an automatic differentiation framework to calculate the gradient and performs a one-step parameter update (e.g., using the Adam optimizer).

[0122] For each training sample, the agent performs the above steps to update the parameters once. Then, it iterates through the entire training dataset multiple times. As training progresses, the policy model gradually tends to generate high-reward trajectories (i.e., trajectories with proper tool calls, complete privacy protection, and correct SQL execution) while avoiding low-reward trajectories. Because this optimization algorithm does not rely on a value network, and intra-group normalization effectively eliminates the influence of differences in the absolute value of rewards between different samples, the training process is more stable, making it particularly suitable for learning small-parameter models.

[0123] Training terminates when the average total reward (or execution accuracy) of the policy model on the validation set no longer improves, or when the preset maximum number of training steps is reached. The resulting trained policy model can then be deployed to a local agent for privacy-preserving NL2SQL tasks during the prediction phase.

[0124] In summary, during the specific training process of the group relative policy optimization algorithm, for each training sample, the current policy is synchronously sampled. The algorithm outputs execution trajectories and calculates their reward scores one by one. To accurately measure the quality of these outputs, the algorithm directly standardizes the rewards within each set of results, thereby obtaining a relative advantage estimate for each output. Based on this feedback from within-set comparisons, the algorithm finally applies the set objective function to adjust the policy parameters. Gradient ascent updates are performed to drive the agent to continuously optimize its decision-making strategy.

[0125] Through the above implementation process, the group relative policy optimization algorithm was used to achieve aligned training of the policy model, enabling it to have standardized tool calling capabilities and strict privacy review capabilities without relying on large cloud models and independent value networks.

[0126] The trained policy model can be deployed to a local agent for privacy-preserving NL2SQL tasks during the prediction phase. Please refer to [link / reference]. Figure 3 , Figure 3 The flowchart illustrates a privacy protection method provided in this application embodiment. This method is applied to the aforementioned intelligent agent, whose policy model is trained using the aforementioned method. For details, please refer to the implementation process of the aforementioned embodiment.

[0127] The method includes the following steps: Step S210: Receive natural language query and database pattern information.

[0128] Step S220: At each time step, read the current state of the Markov decision process.

[0129] When a user submits a query request through an application programming interface (API) or interactive interface, the local agent first receives two inputs: a natural language query Q (a string) and database schema information D (structured data, serializable as text or JSON). The agent then initializes the prediction state S0, which includes: Q: Original query (immutable); D: Database schema (immutable); M0: An initially empty set of privacy-de-identification maps used to record the correspondence between the original sensitive words and mask symbols; I0: An initially empty set of intermediate results, used to store the gradually generated de-identified queries, review judgments, and generated SQL.

[0130] At the same time, the agent sets the current step counter t=0 and the maximum step threshold T. max (e.g., T) max =12), to prevent getting stuck in an infinite loop or over-consuming resources.

[0131] At each time step t, the agent reads the current state S from the state space. tAt each subsequent time step, the state space stores the desensitized mapping and intermediate results obtained after the previous action was executed.

[0132] Step S230: Invoke the policy model, taking the current state as input, and output the probability distribution of the current action.

[0133] The agent will set the current state S t Serialize to an input format acceptable to the policy model (e.g., Q, D, M) t The current representation of I t (Concatenated into text or encoded vectors) and input into the trained policy model π θ The model performs forward computation and outputs the probability score for each action in the action space. For example, in a certain state, the model outputs the following probabilities: calling the de-identification tool 92%, reflection and review 5%, SQL generation 2%, termination 1%.

[0134] Step S240: Select the action with the highest probability from the action space based on the current action probability distribution and execute it.

[0135] The action space includes tool invocation actions, reflection and review actions, and task termination actions. The tool invocation action is used to invoke the tools in the toolset, and the reflection and review action is used to make the strategy model output a security judgment result on the current de-identification result.

[0136] Since the prediction phase no longer requires exploring randomness, the agent typically employs a greedy strategy: directly selecting the action with the highest probability. A greedy approach can also be approximated by using temperature sampling (with temperatures close to 0). As shown in the probability distribution example above, the agent chooses to "call the desensitization tool".

[0137] If a de-identification tool is invoked, the current query Q and database schema D are scanned to identify sensitive keywords (using a pre-defined sensitive keyword list or a lightweight NER model). These keywords are then replaced with irreversible symbols (such as [T_EMP], [V_SALES], [C_NAME], [C_SALARY]), and the mapping relationship is recorded to M. t and update intermediate result I t This is the anonymized query text.

[0138] Step S250: Repeat the above steps until the SQL statement is output or the preset maximum number of iterations is reached to obtain the final SQL statement.

[0139] After the agent completes the current step, it increments the step counter by 1 and then checks: If the current action is "task terminated", then exit the loop.

[0140] If a valid SQL statement has been generated after the current action is executed (the SQL generation tool has been called and symbolic SQL has been generated), the agent can also terminate early (optional strategy).

[0141] If the number of steps t ≥ T max If the loop terminates prematurely, it will be forcibly terminated to avoid an infinite loop.

[0142] Otherwise, repeat steps S220-S250 until the loop ends.

[0143] After the loop ends, the agent starts from the final state S. t Intermediate result I t Extract the final generated SQL statement from the mapping set M. If symbolic SQL exists, then use the mapping set M. t The symbolic representation is restored to its actual table name, column name, and value to obtain plaintext SQL that can be executed directly. If restoration is not required (e.g., executing symbolic SQL directly in an anonymized environment), the symbolic output is retained. If no valid SQL is generated due to timeout or error, an empty string or error message is returned.

[0144] Understandably, some of the implementation steps of the above prediction process can be referred to the relevant implementation steps in the training process. Their specific implementation content is similar, and for the sake of brevity, they will not be repeated here.

[0145] Using the aforementioned prediction method, the local intelligent agent can autonomously convert natural language queries into secure SQL without relying on large cloud models, while simultaneously meeting privacy compliance requirements to ensure data remains within its domain.

[0146] In the aforementioned prediction process, the local agent autonomously decides each action using a pre-trained policy model. To ensure privacy and security, if the currently selected action is a reflection and review action and the agent's security assessment of the current de-identification result is unsatisfactory, then in subsequent action selections, privacy protection and remediation tools are allowed to be called, while SQL processing tools are prohibited, until the security assessment result is satisfactory.

[0147] In the specific implementation, the agent can maintain an additional review status flag, `review_status`, in the state space. This flag can be set to "pass" or "fail". Initially, this flag is set to "pass" (indicating no security issues have been found). When a review action is selected at a certain time step, and the model outputs a security decision result of "fail", the agent updates the flag to "fail" and writes the diagnostic information from the decision result (e.g., "table name employees not masked") into the intermediate result I. t .

[0148] In subsequent time steps, when the agent selects actions, it no longer relies solely on the action probability distribution output by the policy model, but instead introduces an action filtering layer. This action filtering layer filters candidate actions in the action space based on the current review state flag. Permitted actions: Privacy protection and remediation tools (such as supplementary desensitization tools and masking repair tools), because these tools can perform additional desensitization operations on omissions or weaknesses, thereby improving the integrity of privacy protection.

[0149] Actions prohibited from being invoked: SQL processing tools (including SQL generation tools, SQL syntax checking tools, SQL execution tools, etc.), because the de-identification results have not yet reached the security standard, and the generation or processing of SQL statements is prohibited to avoid leakage of sensitive information.

[0150] Other actions (such as re-invoking the reflection review action, invoking other privacy protection tools, or terminating the task) are not restricted.

[0151] The action filtering layer is implemented as follows: after the policy model outputs the action probability distribution, the agent traverses the action space, sets the probability value of the currently prohibited action to zero, and then renormalizes the probabilities of the remaining actions (or directly selects the action with the highest probability among the allowed actions). In this way, the agent will be unable to select SQL processing tools, thus avoiding the output of SQL that may contain sensitive information in an insecure state.

[0152] The agent continues to execute permitted actions, typically invoking repair tools to supplement the masking results. After each repair tool execution, the agent should trigger a re-evaluation review action to reassess the security of the masking results. If the new review judgment becomes "passed," the agent updates the review status flag to "passed," and the action filtering layer resumes normal permission for all actions (including SQL processing tools). If it remains "failed," the SQL processing tools remain prohibited, and iterative repair continues until the review passes or the preset maximum number of steps is reached (at which point it is forcibly terminated and an error message is returned).

[0153] Through this action constraint mechanism, the prediction process can effectively ensure that SQL statements are only allowed to be generated after the de-identified results are confirmed as safe by the model itself (approved by the review), thereby eliminating the risk of privacy leakage caused by incomplete de-identification.

[0154] In conjunction with the above embodiments, please refer to Figure 4 , Figure 4This is a structural block diagram of a training device 300 for a policy model in an intelligent agent, provided in an embodiment of this application. The training device 300 can be a module, program segment, or code on an electronic device. It should be understood that the training device 300 corresponds to the method embodiment described above and is capable of executing the various steps involved in the method embodiment. The specific functions of the training device 300 can be found in the description above; detailed descriptions are omitted here to avoid repetition. The training device 300 is applied to an intelligent agent, which includes a policy model, a toolset, and a state space. The toolset includes privacy protection tools, privacy protection and remediation tools, and SQL processing tools. The state space is used to maintain the state of a Markov decision process, including the original natural language query, database schema information, the current privacy-de-identified mapping set, and intermediate execution results.

[0155] Optionally, the training device 300 for the policy model in the intelligent agent includes: The training set acquisition module 310 is used to acquire a training dataset, wherein each training sample in the training dataset includes the original natural language query and the corresponding database pattern information. The state reading module 320 is used to read the current state of the Markov decision process at each time step; Action probability output module 330 is used to call the strategy model, take the current state as input, and output the action probability distribution; Action execution module 340 is used to select and execute a target action from the action space according to the action probability distribution. The action space includes tool invocation action, reflection and review action and task termination action. The tool invocation action is used to invoke the tools in the toolset. The reflection and review action is used to make the strategy model output a security judgment result on the current de-identification result. The reward calculation module 350 is used to update the current state based on the action execution result and determine the current instant reward; The parameter update module 360 ​​is used to update the parameters of the strategy model based on the current instant reward.

[0156] Optionally, the current immediate reward includes a first type of reward and a second type of reward. The first type of reward is determined based on at least one dimension of the current tool call sequence: compliance, redundancy, and effectiveness. The second type of reward is determined based on the protection integrity of the sensitive information by the current de-identification result. The protection integrity is used to characterize the degree of coverage of displayed sensitive words and / or the degree of resistance to contextual implicit inference attacks.

[0157] Optionally, the reward calculation module 350 is used to call a preset logical compliance detection function to determine whether the current tool call conforms to the preset execution order of privacy protection tasks and obtain a first judgment result; call a preset redundancy detection function to determine whether the current tool call is a redundant operation based on historical call trajectories and obtain a second judgment result; call a preset validity detection function to determine whether the state has undergone a valid change after the current tool call is executed and obtain a third judgment result; and weight and fuse the first judgment result, the second judgment result, and the third judgment result to obtain the first type of reward.

[0158] Optionally, the reward calculation module 350 is used to obtain the mask recall rate, which is the ratio of the number of sensitive words actually masked in the current de-identification result to the total number of sensitive words that should be masked; call a preset adversarial inference model, take the current de-identification result as input, and output the number of mask tags that were not successfully recovered; obtain a re-identification score based on the number of mask tags that were not successfully recovered, which is the proportion of the number of mask tags that were not successfully recovered to the total number of mask tags; and perform a weighted fusion of the mask recall rate and the re-identification score to obtain the second type of reward.

[0159] Optionally, the current instant reward includes a third type of reward, which is determined based on the correctness of the execution of the final generated SQL statement. The reward calculation module 350 is used to repeatedly execute the steps from reading the current state to determining the current instant reward for each training sample until the termination condition is met, obtain an execution trajectory, and determine the third type of reward.

[0160] Optionally, the parameter update module 360 ​​is configured to: acquire multiple execution trajectories for each training sample; calculate the total immediate reward for each execution trajectory, wherein the total immediate reward is obtained by weighted fusion of the first type of reward, the second type of reward, and the third type of reward; calculate the mean and standard deviation of the reward based on the total immediate reward of each execution reward; calculate the relative advantage value of each execution reward based on the mean and standard deviation of the reward; and update the parameters of the policy model based on the relative advantage value using the objective function of the group relative policy optimization algorithm.

[0161] Optionally, the objective function is expressed as:

[0162] in, The parameters of the policy model are represented by , and q represents the current training sample. For the empirical distribution of the training data, This means taking the expectation over all training samples; This represents the probability ratio between the current strategy model and the old strategy model for the same execution trajectory. This is the clipping threshold; The function is used to achieve the cropping effect; For the clipping function, when Exceeding When the range is defined, it is truncated to the boundary value; For the first The relative advantage value of each execution trajectory; The KL divergence penalty coefficient is used to control the current policy model. Compared with the reference strategy model The severity of the penalty for the degree of deviation between them; The KL divergence between the current policy model and the reference policy model; This represents the i-th execution trajectory.

[0163] In conjunction with the above embodiments, please refer to Figure 5 , Figure 5 This is a structural block diagram of a privacy protection device 400 provided in an embodiment of this application. The privacy protection device 400 can be a module, program segment, or code on an electronic device. It should be understood that the privacy protection device 400 corresponds to the above method embodiment and is capable of executing the various steps involved in the method embodiment. The specific functions of the privacy protection device 400 can be found in the description above; detailed descriptions are omitted here to avoid repetition. The privacy protection device 400 is applied to an intelligent agent, which includes a policy model, a toolset, and a state space. The toolset includes privacy protection tools, privacy protection repair tools, and SQL processing tools. The state space is used to maintain the state of the Markov decision process, and the state includes the original natural language query, database schema information, the current privacy-de-identified mapping set, and intermediate execution results.

[0164] Optionally, the privacy protection device 400 includes: Information acquisition module 410 is used to receive natural language queries and database pattern information; The state acquisition module 420 is used to read the current state of the Markov decision process at each time step; The probability output module 430 is used to call the strategy model, take the current state as input, and output the probability distribution of the current action; The action selection module 440 is used to select the action with the highest probability from the action space according to the current action probability distribution and execute it. The action space includes tool invocation action, reflection and review action and task termination action. The tool invocation action is used to invoke the tools in the toolset. The reflection and review action is used to make the strategy model output a security judgment result on the current de-identification result. The output module 450 is used to repeat the above steps until the SQL statement is output or the preset maximum number of iterations is reached to obtain the final SQL statement.

[0165] Optionally, if the currently selected action is a reflection and review action and the agent's security judgment result for the current de-identification result is not passed, the action selection module 440 is used to allow the invocation of privacy protection and repair tools and prohibit the invocation of SQL processing tools when selecting subsequent actions, until the security judgment result is passed.

[0166] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0167] Please refer to Figure 6 , Figure 6 This is a schematic diagram of an electronic device for executing a training method or privacy protection method for a policy model in an intelligent agent, provided in an embodiment of this application. The electronic device may include: at least one processor 510, such as a CPU; at least one communication interface 520; at least one memory 530; and at least one communication bus 540. The communication bus 540 is used to establish communication between these components. In this embodiment, the communication interface 520 is used for signaling or data communication with other node devices. The memory 530 may be a high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 530 may also be at least one storage device located remotely from the aforementioned processor. The memory 530 stores computer-readable instructions, which, when executed by the processor 510, cause the electronic device to perform the aforementioned method process.

[0168] Understandable. Figure 6 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown. Figure 6 The components shown can be implemented using hardware, software, or a combination thereof.

[0169] This application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the method process executed by the electronic device in the above method embodiments.

[0170] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as including: Obtain the training dataset, wherein each training sample in the training dataset includes the original natural language query and the corresponding database pattern information; At each time step, read the current state of the Markov decision process; The policy model is invoked, taking the current state as input, and the action probability distribution is output. The target action is selected from the action space and executed according to the action probability distribution. The action space includes tool invocation action, reflection and review action and task termination action. The tool invocation action is used to invoke the tools in the toolset. The reflection and review action is used to make the strategy model output a security judgment result on the current de-identification result. Update the current state based on the action execution result and determine the current immediate reward; The parameters of the strategy model are updated based on the current instant reward.

[0171] In summary, this application provides a training method, privacy protection method, device, storage medium, and program product for a policy model in an intelligent agent. This method models the privacy protection task of the intelligent agent as a Markov decision process and uniformly incorporates tool invocation and reflection review actions into the action space. Combined with an immediate reward mechanism, it trains the policy model, effectively solving the problems in existing technologies where models are prone to temporal disorder and getting stuck in infinite loops during tool scheduling, as well as the omission of explicit sensitive terms and insufficient resistance to implicit inference attacks in privacy review. This allows the model to autonomously learn standardized tool invocation sequences and strict review strategies without relying on an independent value network. The resulting local intelligent agent can autonomously complete end-to-end generation from natural language queries to secure SQL during the prediction phase, maintaining high masked recall and strong implicit inference resistance while preserving the semantic integrity and execution accuracy of SQL statements, achieving an effective balance between privacy protection and task utility.

[0172] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0173] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0175] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0176] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for training a policy model in an intelligent agent, characterized in that, The intelligent agent includes a policy model, a toolset, and a state space. The agent makes decisions and optimizes privacy-preserving tasks through a Markov decision process. The toolset includes privacy-preserving tools, privacy-preserving remediation tools, and SQL processing tools. The state space is used to maintain the state of the Markov decision process, which includes the original natural language query, database schema information, the current privacy-de-identification mapping set, and intermediate execution results. The method includes: Obtain the training dataset, wherein each training sample in the training dataset includes the original natural language query and the corresponding database pattern information; At each time step, read the current state of the Markov decision process; The policy model is invoked, taking the current state as input, and the action probability distribution is output. The target action is selected from the action space and executed according to the action probability distribution. The action space includes tool invocation action, reflection and review action and task termination action. The tool invocation action is used to invoke the tools in the toolset. The reflection and review action is used to make the strategy model output a security judgment result on the current de-identification result. Update the current state based on the action execution result and determine the current immediate reward; The parameters of the strategy model are updated based on the current instant reward.

2. The method according to claim 1, characterized in that, The current immediate reward includes a first type of reward and a second type of reward. The first type of reward is determined based on at least one dimension of the current tool call sequence: compliance, redundancy, and effectiveness. The second type of reward is determined based on the protection integrity of sensitive information by the current de-identification result. The protection integrity is used to characterize the degree of coverage of displayed sensitive words and / or the degree of resistance to contextual implicit inference attacks.

3. The method according to claim 2, characterized in that, The first type of reward is determined in the following manner: Call the pre-defined logical compliance detection function to determine whether the current tool call conforms to the preset execution order of privacy protection tasks, and obtain the first judgment result; Call the preset redundancy detection function, determine whether the current tool call is a redundant operation based on the historical call trajectory, and obtain a second judgment result; Call the pre-defined validity check function to determine whether the state has changed effectively after the current tool call is executed, and obtain the third judgment result; The first judgment result, the second judgment result, and the third judgment result are weighted and fused to obtain the first type of reward.

4. The method according to claim 2, characterized in that, The second type of reward is determined in the following manner: Obtain the mask recall rate, which is the ratio of the number of sensitive words actually masked in the current de-identification result to the total number of sensitive words that should be masked; Invoke the preset adversarial inference model, take the current desensitization result as input, and output the number of mask tags that were not successfully recovered; A re-identification score is obtained based on the number of mask tags that were not successfully recovered, wherein the re-identification score is the proportion of the number of mask tags that were not successfully recovered to the total number of mask tags; The mask recall rate and the re-identification score are weighted and fused to obtain the second type of reward.

5. The method according to claim 2, characterized in that, The current instant reward includes a third type of reward, which is determined based on the correctness of the execution of the final generated SQL statement. The method further includes: For each training sample, the steps from reading the current state to determining the current immediate reward are repeated until the termination condition is met, resulting in an execution trajectory and the determination of the third type of reward.

6. The method according to claim 5, characterized in that, The parameters for updating the strategy model based on the current instant reward include: For each training sample, obtain multiple execution trajectories; Calculate the total instant reward for each of the multiple execution trajectories, wherein the total instant reward is obtained by weighted fusion of the first type of reward, the second type of reward, and the third type of reward; Calculate the mean and standard deviation of rewards based on the total immediate rewards for each execution reward; Calculate the relative advantage value of each execution reward based on the mean reward and the standard deviation of the reward; The objective function of the group relative strategy optimization algorithm is used to update the parameters of the strategy model based on the relative advantage value.

7. The method according to claim 6, characterized in that, The objective function is expressed as: in, The parameters of the policy model are represented by , and q represents the current training sample. For the empirical distribution of the training data, This means taking the expectation over all training samples; This represents the probability ratio between the current strategy model and the old strategy model for the same execution trajectory. This is the clipping threshold; The function is used to achieve the cropping effect; For the clipping function, when Exceeding When the range is defined, it is truncated to the boundary value; For the first The relative advantage value of each execution trajectory; The KL divergence penalty coefficient is used to control the current policy model. Compared with the reference strategy model The severity of punishment for the degree of deviation between them; The KL divergence between the current policy model and the reference policy model; This represents the i-th execution trajectory.

8. A privacy protection method, characterized in that, This method is applied to an intelligent agent, which includes a policy model, a toolset, and a state space. The agent makes decisions and optimizes privacy-preserving tasks using a Markov decision process. The toolset includes privacy-preserving tools, privacy-preserving remediation tools, and SQL processing tools. The state space maintains the state of the Markov decision process, including the original natural language query, database schema information, the current privacy-de-identification mapping set, and intermediate execution results. The method includes: Receive natural language queries and database schema information; At each time step, read the current state of the Markov decision process; The policy model is invoked, taking the current state as input, and the probability distribution of the current action is output. Based on the current action probability distribution, the action with the highest probability is selected from the action space and executed. The action space includes tool invocation action, reflection and review action, and task termination action. The tool invocation action is used to invoke the tools in the toolset. The reflection and review action is used to make the strategy model output a security judgment result on the current de-identification result. Repeat the above steps until the SQL statement is output or the preset maximum number of iterations is reached to obtain the final SQL statement.

9. The method according to claim 8, characterized in that, If the currently selected action is a reflection and review action and the agent's security assessment result for the current desensitization result is "not passed", the method further includes: When selecting subsequent actions, privacy protection and repair tools should be allowed to be called, while SQL processing tools should be prohibited, until the security assessment result is passed.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the method as described in any one of claims 1-9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1-9.

12. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-9.

Images