A method for rejecting embodied intelligent model based on reinforcement adversarial learning

Abstract

The application discloses a kind of based on reinforcement confrontation learning's facing embodiment intelligent model refusal method, belong to intelligent question and answer technical field, including steps: strategy data set D1 is constructed, by D1 pretraining security strategy model, preference data set D2 is constructed, and reward model is obtained by D2 training;Safety label is labeled for each preference sample in D2;Reward function is constructed;Security strategy model is trained based on reinforcement learning and confrontation learning, and optimal strategy model is obtained, for generating action to guide embodiment intelligent model to generate response in real-time dialogue.The application can improve the quality of embodiment intelligent model in safe response by introducing security strategy classification model and combining human feedback and reinforcement learning optimization mechanism of confrontation learning, alleviate the problems of insufficient recognition of malicious input variants, too rigid refusal strategy, task success rate decline and difficulty in unified modeling of embodiment safety behavior in prior art.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering technology, and in particular to a method for refusing answers for embodied intelligent models based on reinforcement adversarial learning. Background Technology

[0002] With the rapid development of large-scale language models, embodied intelligence models, and dialogue agents, reinforcement learning-based human feedback optimization methods have shown great potential in fields such as intelligent customer service, online education, smart finance, service robots, and human-machine collaboration. As an optimization method oriented towards decision-making processes, reinforcement learning-based human feedback safety response strategies, by introducing an independent safety policy decision module before generating behavior, can significantly improve the risk avoidance capabilities of dialogue systems in malicious command scenarios. Furthermore, it can constrain the boundaries of rejection and execution behaviors when embodied intelligence models actually interact with the environment, maintaining consistency and robustness of responses while ensuring task completion quality. This, in turn, promotes the safe application of dialogue agents and embodied intelligence agents in high-risk, highly constrained environments. In the fields of dialogue security and embodied intelligence security control, ensuring the compliance of generated content, the security of behavioral decisions, and the rationality of rejection strategies are crucial for achieving goals such as understanding user intent, responding to commands, reasoning about information, and executing physical tasks. Application requirements such as malicious input detection, fine-grained rejection strategy selection, context-robust responses, and secure rejection and alternative execution of high-risk commands in embodied scenarios often necessitate a balance between risk control, task completion rate, and user experience to meet the demands of secure and reliable human-computer dialogue and collaboration. However, existing technologies often rely on static rules or simple content filtering, making it difficult to promptly identify variant malicious commands, achieve a fine balance between "secure rejection" and "task execution," and lack systematic constraints and adaptive training mechanisms for the rejection behavior of embodied intelligent models. This results in the risk of security rules being bypassed in open environments.

[0003] Definitions:

[0004] Roll-Out: In the context of intelligent dialogue, Roll-Out refers to the process where an agent (dialogue system) starts from the current state and, according to a predetermined or to-be-evaluated strategy, continuously executes a series of actions (generating dialogue responses) until a termination condition is met (such as reaching the maximum number of rounds, task completion, or natural dialogue termination), thus obtaining a complete "trajectory." During the training phase, the system repeatedly performs a large number of Roll-Outs, using this trajectory data to update the strategy or value function, gradually improving the dialogue quality. In a Roll-Out, the user and the agent will engage in multiple dialogues, with each corresponding "question and answer" considered a "dialogue round." Assuming that from the current state until the termination condition is met after T "dialogue rounds," then the roll-out contains T dialogue rounds. In this invention, a complete "Roll-Out" is referred to as a "complete trajectory deduction" or "complete trajectory sampling."

[0005] MDP (Markov Decision Process) is a mathematical model of sequential decision-making, used to simulate stochastic policies and rewards achievable by an agent in an environment where the system state has Markov properties. MDPs are constructed based on a set of interacting objects: the agent and the environment. The elements include state, action, policy, and reward. In MDP simulations, the agent perceives the current system state, performs actions on the environment according to the policy, thereby changing the state of the environment and receiving a reward. The accumulation of this reward over time is called the payout.

[0006] PPO (Proximal Policy Optimization) is a policy optimization algorithm. In existing reinforcement learning, directly optimizing the policy can lead to unstable training, and the model may collapse due to excessive parameter updates. PPO limits the policy update magnitude, ensuring that each training step does not deviate too much from the current policy, while making efficient use of sampled data.

[0007] Reinforcement learning (RL) is a machine learning method. The basic framework of reinforcement learning is the Markov decision process, which allows an agent to learn the optimal policy through trial and error in its interaction with the environment. Summary of the Invention

[0008] The purpose of this invention is to provide a method for rejecting responses based on reinforcement adversarial learning for embodied intelligent models, which addresses the problems of insufficient identification of malicious input variants, overly rigid rejection strategies, decreased task success rate, and difficulty in uniformly modeling embodied security behaviors in existing technologies.

[0009] To achieve the above objectives, the technical solution adopted by this invention is as follows: a method for refusing responses based on reinforcement adversarial learning for embodied intelligent models, comprising the following steps:

[0010] S1, construct the policy dataset D1, including S11~S13;

[0011] S11, break down the dialogue between the user and the embodied intelligent model into dialogue rounds based on a question-and-answer format. For the t-th dialogue round, obtain the user input x. t and embodied intelligent model state s t s t =(h t ,d t ,c t ), where h t d t c t They are respectively historical dialogues, x t The risk description, the encoding vector corresponding to the context conditions, and s t Mapping to a preset dimension yields the state feature z. t ;

[0012] S12, preset the action categories a0~a4 output by the security policy model, where a0 indicates that the security policy model does not participate, a1 indicates that the task is executed, a2 indicates that the task is rejected, a3 indicates that additional questions are asked to clarify the user's intent, and a4 indicates that a security alternative suggestion is given, generating the action space A of the embodied intelligence model. safe ={a0,a1,a2,a3,a4}, where a0 represents;

[0013] S13 represents partial state features z. t Label the action b a b a ∈A safe All state features are used to construct the policy dataset D1.

[0014] S2, using D1 to pre-train a security policy model M safe , used to input z t Output the type of action;

[0015] S3, construct the preference dataset D2;

[0016] For each x t The embodied intelligence model generates five responses according to a0~a4, and then randomly selects two responses y. (1) y (2) Generate corresponding preference samples (s) t ,a (1) ,y (1) ), (s t ,a (2),y (2) ), where a (1) a (2) y (1) y (2) The corresponding action;

[0017] Humans manually select positive and negative samples based on preferences, and let the preference label of the positive sample be l. t =1, negative sample l t =0;

[0018] The positive and negative samples constitute x t Preference sample pairs, all preference sample pairs constitute the preference dataset D2;

[0019] S4, use D2 to train a reward model, which takes a preferred sample as input and outputs the probability that it is a positive sample;

[0020] S5, label each preference sample in D2 with a safety label, including violation label, malicious label, reasonable rejection label, usefulness label, and help score;

[0021] S6, construct x t reward function r t ;

[0022] ,

[0023] in, , , r t Safety reward component, usefulness component, preference component, , , These are the weighting factors for the corresponding multiplication terms. Based on the first three items of the security label, Based on the latter two security tags, For the reward model on x t The output;

[0024] S7, based on reinforcement learning and adversarial learning, trains a security policy model to obtain the optimal policy model. ;

[0025] S8 allows users to engage in real-time dialogue with an embodied intelligent model. In each dialogue round, the user's input is considered. Generate corresponding state features ,through The action category is output, and then the embodied intelligence model generates a response based on that category.

[0026] Preferably, in S1, the risk description is given by x. tThe text description should include whether it involves sensitive topics and whether it complies with security rules; the contextual conditions include user profile, usage scenario, and current system mode, where usage scenario includes development environment and production environment, and current system mode includes strict mode and lenient mode; the s t The feature extraction function maps the data to a preset dimension.

[0027] As a preferred option, the security strategy model in S2 is trained by a multi-objective classification network, and the reward model in S4 is trained by a binary classification network.

[0028] As a preferred option, the embodied intelligence model in S3 generates five responses according to a0 to a4 respectively: when performing action a0, the embodied intelligence model directly calls the dialogue model to generate a response; when performing actions a1 to a4, the dialogue model generates responses according to a1 to a4; when selecting positive and negative samples in S3, the preferred sample that better meets the requirements of safety and interactive experience is taken as the positive sample, and the other is taken as the negative sample.

[0029] As a preferred option, S5 is specifically:

[0030] S51 categorizes user input into normal and malicious, and responses into safe and illegal.

[0031] S52, Choose one preference sample (s) t ,a (1) ,y (1) ), to a (1) y (1) As s t Current action a t and current response y t ;

[0032] If y t Violation, then violation label =1, otherwise 0;

[0033] If x t Malicious, then malicious label =1, otherwise 0;

[0034] If x t malicious and a t If the value is a2 or a4, then the label should be reasonably rejected. =1, otherwise 0;

[0035] If x t Normally, there are usefulness tags. =1, otherwise 0;

[0036] Help score For artificial y t The rating of the degree of help. The greater the degree of help The larger.

[0037] As a preferred option, S6 is specifically:

[0038] S61, construct the security reward component according to the following formula. Useful components Apply the reward model to x t The output is used as a preference component. ;

[0039] ,

[0040] ,

[0041] In the formula, It is y t The penalty coefficient for violations. It is x t The reward coefficient for malicious actions, but choosing either action a2 or a4. For x t The penalty coefficient for malicious selection of action a1. , , All are greater than 0;

[0042] S62, construct x t reward function r t , .

[0043] As a preferred embodiment, S7 includes S71 to S74;

[0044] S71, perform n rounds of complete trajectory deduction, each round of complete trajectory deduction generates a dialogue trajectory. T represents the total number of dialogue rounds, and all dialogue trajectories constitute the dataset D3.

[0045] S72 uses D3 for reinforcement learning and calculates the objective function J(θ) according to the following formula during the deduction process, and updates M using the PPO policy. safe The parameter θ;

[0046] ,

[0047] ,

[0048] In the formula, π θ For M safe The strategy generated based on parameter θ, where max is used to find the maximum value. To find the mathematical expectation, let λ be a learnable scalar and st be the constraints. It is a very small nonnegative constant;

[0049] S73, after n rounds of deduction, generate adversarial samples of dialogue trajectories, combine all adversarial samples into dataset D4, combine D3 and D4 into dataset D5, and update D3 with D5;

[0050] S74, alternately execute S72~S73 until the preset number of times, and finally obtain M. safe As the optimal strategy model .

[0051] Preferably, in S73, adversarial examples are generated by an adversarial generator to generate dialogue trajectories. Adversarial examples Specifically:

[0052] Check the dialogue path one by one The t-th empirical sample (s) t ,a t ,r t If the replacement condition is met, then an adversarial example (s') is generated. t ,a' t ,r' t )replace (s) t ,a t ,r t ), 1≤t≤T, after the check is completed, the dialogue trajectory obtained from the replacement will be used as Adversarial examples ;

[0053] The replacement condition is that x satisfies... t malicious, y t Violation, preference label is 0, reward model for x t The output is lower than the preset threshold or the usefulness label is 0.

[0054] The training for this invention can be divided into three stages:

[0055] Phase 1: Construct the policy dataset D1 to train the security policy model, which will be used as input x. t Corresponding input z t The goal is to output the category of the action, ensuring the security policy model generates a reasonably accurate one. Therefore, supervised pre-training is performed using labeled data, and the parameters of the security policy model are adjusted by minimizing cross-entropy loss to obtain a security policy model with initial parameters, which is further optimized in subsequent stage 3. This stage relies on manually labeled actions instead of the reward function in the MDP. There are 5 action categories:

[0056] a0 indicates that the security strategy model does not participate and directly generates the response based on the task. In other words, the security strategy model does not participate in the decision-making, while the embodied intelligence model directly calls the dialogue model to generate the response.

[0057] a1 indicates task execution, which means instructing the dialogue model to complete the task and generate a response according to the user's literal requirements.

[0058] a2 indicates a rejection task, prompting the dialogue model to generate a rejection statement as a response;

[0059] a3 means that in order to clarify the user's intent, an additional question is asked first, which means that the dialogue model should first generate a question to clarify the user's intent as a response.

[0060] a4 indicates that a safe alternative suggestion is provided, allowing the dialogue model to find x. t It identifies dangerous or irregular operations and generates safety education documents and risk warnings as a response.

[0061] Phase 2: Training the reward model using the constructed preference dataset D2 as input x t For each corresponding preference sample, output the probability that it is a positive sample. This probability will then be used as the preference component in the reward function. When constructing the preference dataset D2, for each x t First, the dialogue model generates five responses based on actions a0 to a4. Then, two responses are randomly selected to generate preferred samples. Next, the one that better satisfies safety and interaction experience is manually selected as the positive sample, and the other is selected as the negative sample. The positive and negative samples together constitute x. t The positive and negative sample pairs are used, and the parameters of the reward model are adjusted during training by minimizing the cross-entropy loss. Through stage 2 training, a model that can score user preferences is obtained, outputting the preference component for each preference sample. .

[0062] Phase 3: Through reinforcement learning and adversarial learning, the network parameters of the security policy model are adjusted to obtain the optimal policy model. In this phase, reinforcement learning and adversarial learning alternate. The purpose of reinforcement learning is to construct an objective function and adjust the parameters of the security policy model according to the objective function. The purpose of adversarial learning is to analyze the weaknesses of the security policy model during reinforcement learning, and to generate more variants (adversarial examples) targeting these weaknesses, forming adversarial examples for dialogue estimation, which are then used in the next round of reinforcement learning. The following points should be noted in Phase 3:

[0063] 1. This invention views the dialogue process of the embodied intelligence model as a Markov decision process (MDP), therefore reinforcement learning is based on x t The corresponding reward function r t The objective function J(θ) is calculated, and the model parameters are adjusted according to J(θ). In existing technologies, reward functions are generally based on a single factor; in this invention, the reward function r... t It includes three components: a security reward component. Useful components Preference components Therefore, these three components need to be constructed.

[0064] 2. Before constructing the three components, the following preparatory work must be done:

[0065] (2.1) Perform step S5 of the present invention to label each preference sample in D2 with a safety label, including violation label, malicious label, reasonable rejection label, usefulness label, and help score;

[0066] According to step S51, this invention categorizes user input into normal and malicious types, and responses into safe and illegal types, based on existing security guidelines. Specifically, user input is semantically divided into three main categories:

[0067] Category 1: Malicious instructions / high-risk input: explicitly requesting the execution of harmful behaviors (such as teaching people how to attack the system, how to self-harm, or how to manufacture dangerous items), inducing the embodied intelligent agent in the dialogue to disclose sensitive information (such as passwords, private data, internal documents, etc.).

[0068] The second category: Malicious input disguised / bypassing rules: It is literally packaged as a "test", "game" or "hypothetical scenario", but it is still a harmful request. It uses indirect descriptions or metaphors to express dangerous behavior.

[0069] Category 3: Normal / Benevolent Input: Completely legal questions or instructions with no security risks, such as daily Q&A, learning assistance, general programming questions, etc.

[0070] The first and second categories are the malicious categories mentioned in step S51, and the third category is the safe category mentioned in step S51.

[0071] The responses are also categorized as either safe or non-compliant, and are determined by the review rules or by manual review.

[0072] (2.2) Label each preference sample with a safety label, including a violation label, a malicious label, a reasonable rejection label, a usefulness label, and a help score. See step S52 for details.

[0073] 3. Before constructing the three components, the following preparatory work must be done:

[0074] (3.1) Safety reward weight :

[0075] ,

[0076] In the formula, , , The value of is 1 or 0. Only when the value is 1 does the term it is multiplied by exist; otherwise, a value of 0 can be considered non-existent. For example... It is yt The penalty coefficient for violations, if y t Violation, , Only then will it have meaning, otherwise =0. Similarly. It is x t The reward coefficient for malicious actions that choose either action a2 or a4, if x t For malicious purposes, Only when it equals 1, if a t If the value is a2 or a4, it indicates that the security policy model has made a reasonable rejection. =1, then , That is, the reward coefficient only exists. Based on the above logic, construct... For each preference sample, a security reward component value can be obtained.

[0077] (3.2) Usefulness component ;

[0078] ,

[0079] Construction principles and security bonus same.

[0080] (3.3) Preference Components The reward model is used directly for output.

[0081] Final reward function Instead of the MDP reward in existing technologies, a new objective function is constructed.

[0082] 4. Regarding the objective function J(θ): The structure is the same as the principle of MDP in existing technologies, but a completely new reward function r is constructed. t r is calculated by comprehensively considering three components. t The value of θ guides the adjustment of the security strategy model parameters. Furthermore, during training, the PPO strategy is used to maximize J(θ). Among the constraints, It is a very small non-negative constant, representing the "maximum number of expected security violations allowed".

[0083] 5. Alternating reinforcement learning and adversarial learning: First, perform n rounds of complete trajectory extrapolation (Roll-Out), and then update the policy through multiple rounds of Roll-Out and PPO. The embodied intelligence model will gradually learn to: When faced with malicious input, it tends to choose a2 (reject the task) or a4 (provide safe alternative suggestions), and tries to avoid a1 (accept the task). When faced with normal input, it tends to choose a1 (accept the task) and improves the user experience by rewarding the model's preference components. For ambiguous but potentially dangerous input, the embodied intelligence model will more often choose a3 (add a question to clarify the user's intent) to reduce the probability of false positives.

[0084] After n rounds of Roll-Out, a certain amount of malicious samples will accumulate. These malicious samples are related to the dialogue trajectory. At a certain moment (s) t ,a t ,r t In terms of ), in a (s) t ,a t ,r t For example, suppose x satisfies t malicious, y t Violation, preference label is 0, reward model for x t If the output is below a preset threshold or the usefulness label is 0, it is considered a bad sample, and a corresponding adversarial sample (s') is generated using an adversarial generator. t ,a' t ,r' t ),right (s) t ,a t ,r t Replace it. Assume... In the process, only samples with t=5 and 10 are considered bad samples, and therefore, samples are generated. , ,get , .

[0085] As the number of training rounds accumulates, the difficulty of malicious input gradually increases, enabling the collection of adversarial examples of dialogue trajectories from different periods. Composition of Courses , It is the set of adversarial examples after the k-th reinforcement learning iteration. During the (k+1)-th reinforcement learning iteration, a small portion of these examples originate from... Mid-sampling, mostly from The data is sampled from the middle to form dataset D4 for the (k+1)th adversarial learning, thereby continuously exposing the strategy to more covert and complex malicious attacks. This mechanism enables the security strategy to not just memorize a bunch of "sensitive keywords", but to learn to be vigilant against semantic-level dangerous intentions through repeated adversarial interactions, thus possessing better "attacks that bypass security rules".

[0086] Compared with the prior art, the advantages of the present invention are as follows:

[0087] (1) This invention constructs a security strategy model, which provides decision labels (a0~a4) such as non-interference, execution, rejection, clarification, or safe substitution before the embodied intelligence model generates a response or issues an action. This method effectively improves the ability to identify and divert malicious and high-risk inputs, improves the overall security of the dialogue generation and action decision-making process, and maintains the consistency of response behavior and rejection behavior in complex scenarios, providing a more reliable decision basis for dialogue security governance and embodied security control.

[0088] (2) Construct and train a reward model to take a preference sample as input and output the probability that it is a positive sample. Use this probability as the preference component in the reward function in subsequent reinforcement learning. Since the positive and negative samples are based on manual annotation, the malicious avoidance rate, the success rate of normal tasks, and the user's subjective preference for the way of refusing to answer can be introduced into it. Since the higher the probability that the response that better meets the safety and interaction experience is as a positive sample, the subsequent training can make the generated response more in line with the standard of "meeting the safety and interaction experience".

[0089] (3) In the key links of the action decision-making process, an optimization method combining reinforcement learning and human feedback is introduced. User input is divided into normal and malicious, and responses are divided into safe and illegal. Five types of safety labels are labeled for preference samples. Through the above distinction and safety labels, safety reward components and usefulness components are constructed. The reward function is designed together with the preference components, thereby continuously optimizing the safety strategy model. This enables the embodied intelligence model to improve the coverage of various malicious input variants while being able to process specific instruction intentions, context details and environmental risk states more finely. This significantly enhances the human-computer interaction and human-computer collaboration effect in high-risk scenarios. It is especially suitable for applications that require long-term and reliable interaction, such as intelligent customer service, online business processing, sensitive information consultation and embodied service robots.

[0090] (4) Combining adversarial learning mechanisms, diverse adversarial samples are automatically or semi-automatically constructed during the training phase. Through the "red team" adversarial generation and progressively difficult training strategy, the weaknesses of the security strategy model under different attack methods and scenarios are continuously exposed. This guides the strategy to continuously improve its robustness against unseen malicious inputs and behaviors that bypass security rules during the reinforcement learning process, thereby further enhancing the practical security and controllability of the system in open dialogue and open physical environments.

[0091] In summary, this invention, by introducing a security policy classification model, a reward model, and a reinforcement learning optimization mechanism combining human feedback and adversarial learning, not only significantly improves the quality and application value of embodied intelligent surface models in the application direction of secure command response, but also provides a unified technical framework for the constraint of rejection behavior and adaptive training of embodied intelligent models. This method offers an innovative solution that balances security and usability for malicious command avoidance and robust response generation. By alleviating the problems of insufficient malicious input variant recognition, overly rigid rejection strategies, decreased task success rates, and the difficulty in uniformly modeling embodied security behaviors in existing technologies, this invention provides a new technical path for dialogue security control, embodied intelligent security decision-making, and intelligent response strategy design, and has broad application prospects, especially in application scenarios with extremely high security requirements such as finance, government affairs, healthcare, and embodied service robots. Attached Figure Description

[0092] Figure 1 This is a flowchart of the present invention;

[0093] Figure 2 This is a flowchart of the training process of the present invention. Detailed Implementation

[0094] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0095] Example 1: See Figure 1 and Figure 2 A method for rejecting responses in embodied intelligent models based on reinforcement adversarial learning includes the following steps:

[0096] S1, construct the policy dataset D1, including S11~S13;

[0097] S11, break down the dialogue between the user and the embodied intelligent model into dialogue rounds based on a question-and-answer format. For the t-th dialogue round, obtain the user input x. t and embodied intelligent model state s t s t =(h t ,d t ,c t ), where h t d t c t They are respectively historical dialogues, x t The risk description, the encoding vector corresponding to the context conditions, and s t Mapping to a preset dimension yields the state feature z. t ;

[0098] S12, preset the action categories a0~a4 output by the security policy model, where a0 indicates that the security policy model does not participate, a1 indicates that the task is executed, a2 indicates that the task is rejected, a3 indicates that additional questions are asked to clarify the user's intent, and a4 indicates that a security alternative suggestion is given, generating the action space A of the embodied intelligence model. safe ={a0,a1,a2,a3,a4}, where a0 represents;

[0099] S13 represents partial state features z. t Label the action b a b a ∈A safe All state features are used to construct the policy dataset D1.

[0100] S2, using D1 to pre-train a security policy model M safe , used to input z t Output the type of action;

[0101] S3, construct the preference dataset D2;

[0102] For each x t The embodied intelligence model generates five responses according to a0~a4, and then randomly selects two responses y. (1) y (2) Generate corresponding preference samples (s) t ,a (1) ,y (1) ), (s t ,a (2) ,y (2) ), where a (1) a (2) y (1) y (2) The corresponding action;

[0103] Humans manually select positive and negative samples based on preferences, and let the preference label of the positive sample be l. t =1, negative sample l t =0;

[0104] The positive and negative samples constitute x t Preference sample pairs, all preference sample pairs constitute the preference dataset D2;

[0105] S4, use D2 to train a reward model, which takes a preferred sample as input and outputs the probability that it is a positive sample;

[0106] S5, label each preference sample in D2 with a safety label, including violation label, malicious label, reasonable rejection label, usefulness label, and help score;

[0107] S6, construct x t reward function rt ;

[0108] ,

[0109] in, , , r t Safety reward component, usefulness component, preference component, , , These are the weighting factors for the corresponding multiplication terms. Based on the first three items of the security label, Based on the latter two security tags, For the reward model on x t The output;

[0110] S7, based on reinforcement learning and adversarial learning, trains a security policy model to obtain the optimal policy model. ;

[0111] S8 allows users to engage in real-time dialogue with an embodied intelligent model. In each dialogue round, the user's input is considered. Generate corresponding state features ,through The action category is output, and then the embodied intelligence model generates a response based on that category.

[0112] Example 2: See Figure 1 and Figure 2 More specifically, based on Example 1:

[0113] In S1, the risk description is given by x. t The text description should include whether it involves sensitive topics and whether it complies with security rules; the contextual conditions include user profile, usage scenario, and current system mode, where usage scenario includes development environment and production environment, and current system mode includes strict mode and lenient mode; the s t The feature extraction function maps the data to a preset dimension.

[0114] The security policy model in S2 is trained by a multi-objective classification network, while the reward model in S4 is trained by a binary classification network.

[0115] In S3, the embodied intelligence model generates five responses according to a0 to a4. Specifically, when the action a0 is executed, the embodied intelligence model directly calls the dialogue model to generate the response; when the actions a1 to a4 are executed, the dialogue model generates the response according to a1 to a4; when selecting positive and negative samples in S3, the preferred sample that better meets the requirements of safety and interactive experience is taken as the positive sample, and the other is taken as the negative sample.

[0116] S5 specifically refers to:

[0117] S51 categorizes user input into normal and malicious, and responses into safe and illegal.

[0118] S52, Choose one preference sample (s) t ,a (1) ,y (1) ), to a (1) y (1) As s t Current action a t and current response y t ;

[0119] If y t Violation, then violation label =1, otherwise 0;

[0120] If x t Malicious, then malicious label =1, otherwise 0;

[0121] If x t malicious and a t If the value is a2 or a4, then the label should be reasonably rejected. =1, otherwise 0;

[0122] If x t Normally, there are usefulness tags. =1, otherwise 0;

[0123] Help score For artificial y t The rating of the degree of help. The greater the degree of help The larger. (Regarding help score) For artificial y t The rating of the degree of help. The greater the degree of help The larger the value, the higher the score can be. For example, a standard can be set for the worker, dividing the level of assistance into four levels with scores of 0.25, 0.5, 0.75, and 1 respectively, corresponding to y. t Assign scores.

[0124] S6 specifically refers to:

[0125] S61, construct the security reward component according to the following formula. Useful components Apply the reward model to x t The output is used as a preference component. ;

[0126] ,

[0127] ,

[0128] In the formula, It is y t The penalty coefficient for violations. It is x t The reward coefficient for malicious actions, but choosing either action a2 or a4. For x t The penalty coefficient for malicious selection of action a1. , , All are greater than 0;

[0129] S62, construct x t reward function r t , .

[0130] S7 includes S71~S74;

[0131] S71, perform n rounds of complete trajectory deduction, each round of complete trajectory deduction generates a dialogue trajectory. T represents the total number of dialogue rounds, and all dialogue trajectories constitute the dataset D3.

[0132] S72 uses D3 for reinforcement learning and calculates the objective function J(θ) according to the following formula during the deduction process, and updates M using the PPO policy. safe The parameter θ;

[0133] ,

[0134] ,

[0135] In the formula, π θ For M safe The strategy generated based on parameter θ, where max is used to find the maximum value. To find the mathematical expectation, let λ be a learnable scalar and st be the constraints. It is a very small nonnegative constant;

[0136] S73, after n rounds of deduction, generate adversarial samples of dialogue trajectories, combine all adversarial samples into dataset D4, combine D3 and D4 into dataset D5, and update D3 with D5;

[0137] S74, alternately execute S72~S73 until the preset number of times, and finally obtain M. safe As the optimal strategy model .

[0138] In S73, adversarial examples are generated by an adversarial generator, which generates dialogue trajectories. Adversarial examples Specifically, this involves checking the dialogue history one by one. The t-th empirical sample (s) t ,a t ,r t If the replacement condition is met, then an adversarial example (s') is generated. t ,a' t ,r' t )replace (s) t ,a t ,r t ), 1≤t≤T, after the check is completed, the dialogue trajectory obtained from the replacement will be used as Adversarial examples ;

[0139] The replacement condition is that x satisfies... t malicious, y t Violation, preference label is 0, reward model for x t The output is lower than the preset threshold or the usefulness label is 0.

[0140] Example 3: See Figure 2 , combined Figure 2 This invention requires three training phases: Phase 1, training the security strategy model; Phase 2, training the reward model; and Phase 3, optimizing the security strategy model using reinforcement learning and adversarial learning. The training process is as follows:

[0141] Data preparation: Prepare datasets D1 and D2.

[0142] Phase 1: Using supervised pre-training, a multi-objective classification network is trained with D1 to obtain a security policy model.

[0143] Phase 2: Using supervised pre-training, a binary classification network is trained with D2 to obtain a reward model.

[0144] Phase 3: Reinforcement learning and adversarial learning phase.

[0145] First, construct dataset D3, corresponding to step S71. Taking a complete trajectory deduction in one round as an example, during the dialogue, each dialogue round will have a corresponding action embodied in the intelligent model. Taking the t-th dialogue round as an example, the state is s. t The embodied intelligent model performs the action a t Generate response y t The reward function r can be calculated. t The collected data forms the dialogue trajectory. .

[0146] Then perform reinforcement learning: corresponding to step S72, calculate the objective function J(θ), and update M using the PPO policy. safe The parameter θ.

[0147] Then perform adversarial learning: corresponding to step S73, after n rounds of reinforcement learning, generate a series of adversarial samples of dialogue trajectories, and combine the dialogue trajectories and the adversarial samples of dialogue trajectories to form dataset D5, which replaces D3 in S71.

[0148] Finally, reinforcement learning and adversarial learning are performed alternately to obtain the optimal policy model. .

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for rejecting responses in embodied intelligent models based on reinforcement adversarial learning, characterized in that, Includes the following steps: S1, construct the policy dataset D1, including S11~S13; S11, break down the dialogue between the user and the embodied intelligent model into dialogue rounds based on a question-and-answer format. For the t-th dialogue round, obtain the user input x. t and embodied intelligent model state s t s t =(h t ,d t ,c t ), where h t d t c t They are respectively historical dialogues, x t The risk description, the encoding vector corresponding to the context conditions, and s t Mapping to a preset dimension yields the state feature z. t ; S12, preset the action categories a0~a4 output by the security policy model, where a0 indicates that the security policy model does not participate, a1 indicates that the task is executed, a2 indicates that the task is rejected, a3 indicates that additional questions are asked to clarify the user's intent, and a4 indicates that a security alternative suggestion is given, generating the action space A of the embodied intelligence model. safe ={a0,a1,a2,a3,a4}, where a0 represents; S13 represents partial state features z. t Label the action b a b a ∈A safe All state features are used to construct the policy dataset D1. S2, using D1 to pre-train a security policy model M safe , used to input z t The type of action to output; S3, construct the preference dataset D2; For each x t The embodied intelligence model generates five responses according to a0~a4, and then randomly selects two responses y. (1) y (2) Generate corresponding preference samples (s) t ,a (1) ,y (1) ), (s t ,a (2) ,y (2) ), where a (1) a (2) y (1) y (2) The corresponding action; Humans manually select positive and negative samples based on preferences, and let the preference label of the positive sample be l. t =1, negative sample l t =0; The positive and negative samples constitute x t Preference sample pairs, all preference sample pairs constitute the preference dataset D2; S4, use D2 to train a reward model, which takes a preferred sample as input and outputs the probability that it is a positive sample; S5, label each preference sample in D2 with a safety label, including violation label, malicious label, reasonable rejection label, usefulness label, and help score; S6, construct x t reward function r t ; ， in, , , r t Safety reward component, usefulness component, preference component, , , These are the weighting factors for the corresponding multiplication terms. Based on the first three items of the security label, Based on the latter two security tags, For the reward model, x t The output; S7, based on reinforcement learning and adversarial learning, trains a security policy model to obtain the optimal policy model. ; S8 allows users to engage in real-time dialogue with an embodied intelligent model. In each dialogue round, the user's input is considered. Generate corresponding state features ,through The action category is output, and then the embodied intelligence model generates a response based on that category.

2. The method for rejecting responses based on reinforcement adversarial learning for embodied intelligent models according to claim 1, characterized in that, In S1, The risk description is given by x. t Does the text description involve sensitive topics and comply with security rules? The context conditions include user profile, usage scenario, and current system mode. The usage scenario includes R&D environment and production environment. The current system mode includes strict mode and lenient mode. The s t The feature extraction function maps the data to a preset dimension.

3. The method for rejecting responses based on reinforcement adversarial learning for embodied intelligent models according to claim 1, characterized in that, The security policy model in S2 is trained by a multi-objective classification network, while the reward model in S4 is trained by a binary classification network.

4. The method for rejecting responses based on reinforcement adversarial learning for embodied intelligent models according to claim 1, characterized in that, In S3, the embodied intelligence model generates five responses according to a0 to a4. Specifically, when the action a0 is executed, the embodied intelligence model directly calls the dialogue model to generate the response; when the actions a1 to a4 are executed, the dialogue model generates the response according to a1 to a4. When selecting positive and negative samples in S3, the sample that better satisfies the preference for safety and interactive experience will be selected as the positive sample, and the other will be selected as the negative sample.

5. The method for rejecting responses based on reinforcement adversarial learning for embodied intelligent models according to claim 1, characterized in that, S5 specifically refers to: S51 categorizes user input into normal and malicious, and responses into safe and illegal. S52, Choose one preference sample (s) t ,a (1) ,y (1) ), to a (1) y (1) As s t Current action a t and current response y t ; If y t Violation, then violation label =1, otherwise 0; If x t Malicious, then malicious label =1, otherwise 0; If x t malicious and a t If the value is a2 or a4, then the label should be reasonably rejected. =1, otherwise 0; If x t Normally, there are usefulness tags. =1, otherwise 0; Help score For artificial y t The rating of the degree of help. The greater the degree of help The larger.

6. The method for rejecting responses based on reinforcement adversarial learning for embodied intelligent models according to claim 5, characterized in that, S6 specifically refers to: S61, construct the security reward component according to the following formula. Useful components Apply the reward model to x t The output is used as a preference component. ; ， ， In the formula, It is y t The penalty coefficient for violations. It is x t The reward coefficient for maliciously choosing action a2 or a4. For x t The penalty coefficient for malicious selection of action a1. , , All are greater than 0; S62, construct x t reward function r t , .

7. The method for rejecting responses based on reinforcement adversarial learning for embodied intelligent models according to claim 1, characterized in that, S7 includes S71~S74; S71, perform n rounds of complete trajectory deduction, each round of complete trajectory deduction generates a dialogue trajectory. T represents the total number of dialogue rounds, and all dialogue trajectories constitute the dataset D3. S72 uses D3 for reinforcement learning and calculates the objective function J(θ) according to the following formula during the deduction process, and updates M using the PPO policy. safe The parameter θ; ， ， In the formula, π θ For M safe The strategy generated based on parameter θ, where max is used to find the maximum value. To find the mathematical expectation, let λ be a learnable scalar and st be the constraints. It is a very small nonnegative constant; S73, after n rounds of deduction, generate adversarial samples of dialogue trajectories, combine all adversarial samples into dataset D4, combine D3 and D4 into dataset D5, and update D3 with D5; S74, alternately execute S72~S73 until the preset number of times, and finally obtain M. safe As the optimal strategy model .

8. A method for rejecting responses in embodied intelligent models based on reinforcement adversarial learning according to claim 7, characterized in that, In S73, adversarial examples are generated by an adversarial generator, which generates dialogue trajectories. Adversarial examples Specifically: Check the dialogue path one by one The t-th empirical sample (s) t ,a t ,r t If the replacement condition is met, then an adversarial example (s') is generated. t ,a' t ,r' t )replace (s) t ,a t ,r t ), 1≤t≤T, after the check is completed, the dialogue trajectory obtained from the replacement will be used as Adversarial examples ; The replacement condition is that x satisfies... t malicious, y t Violation, preference label is 0, reward model for x t The output is lower than the preset threshold or the usefulness label is 0.

Images