A defensive AIGC adversarial agent and its training method

By combining reinforcement learning, decoupling learning, and adversarial learning modules of the defensive AIGC adversarial agent, the problems of generating reasonable counterfactual actions and recognizing context in AI autonomous debate are solved, and more flexible debate strategies and context recognition are achieved.

CN118586430BActive Publication Date: 2024-11-15HANGZHOU INTERNATIONAL INNOVATION INSTITUTE OF BEIHANG UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411059119.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-15
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

Existing AI-powered autonomous debate technology has gaps in online information research, making it difficult to generate reasonable counterfactual actions and identify contexts with different semantic meanings.

Method used

A defensive AIGC adversarial agent is adopted, including a reinforcement learning module, a decoupling learning module, a buffer pool, and an adversarial learning module. By combining the observation policy network, the counterfactual trajectory generation module, the decoupling learning module, and the adversarial learning module, more reasonable counterfactual actions are generated and the context is identified.

Benefits of technology

In the counterfactual action generation task, generating more reasonable counterfactual actions enables the agent to develop more flexible debate strategies and recognize different semantic contexts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118586430B_ABST
    Figure CN118586430B_ABST
Patent Text Reader

Abstract

A defensive AIGC adversarial agent and a training method thereof belong to the field of artificial intelligence technology. In the agent, a reinforcement learning module obtains a trajectory through an observation strategy network; a decoupling learning module generates a counterfactual trajectory according to the trajectory, and performs decoupling learning to obtain a reward function, calculates a reward according to the reward function, and generates parameters of the counterfactual strategy network according to the reward; in the adversarial learning module, a controller controls the agent to generate multiple reverse debate actions according to the counterfactual strategy network, controls the observation strategy network to generate multiple forward debate actions, and a discriminator judges the authenticity of forward debate actions and reverse debate actions according to the transfer states of multiple reverse debate actions and the transfer states of multiple forward debate actions, and puts the true debate action, state, next state and reward into a buffer pool as a four-tuple. The present invention can recognize the context, so that the agent can generate a more flexible debate strategy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a defensive AIGC confrontation intelligent agent and a training method thereof, and belongs to the technical field of artificial intelligence. Background Art

[0002] Current research on AI autonomous debate mainly focuses on experimental games such as chess and Go, or high-intelligence game activities in human society such as intelligent agent collaboration.

[0003] There is still a gap in introducing relevant technologies in the field of AI autonomous debate into network information research. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the object of the present invention is to provide a defensive AIGC adversarial intelligent agent and a training method thereof, which can generate more reasonable counterfactual actions in counterfactual action generation tasks, so that the intelligent agent can produce more flexible debate strategies and recognize contexts with different semantics.

[0005] To achieve the above-mentioned purpose, the present invention provides a defensive AIGC adversarial agent, which is characterized by comprising a reinforcement learning module, a decoupling learning module, a buffer pool and an adversarial learning module, wherein:

[0006] The reinforcement learning module includes the observation strategy network SCM- , obtain a trajectory through the observation strategy network, where a t is the forward debate action of the observation strategy network at time t, s t is the forward debate state of the observation strategy network at time t, θ is the parameter of the neural network of the observation strategy network, t=1, 2, …, n;

[0007] The inverse reinforcement learning module includes a counterfactual trajectory generation module, a decoupling learning module and a counterfactual strategy network, wherein the counterfactual trajectory generation module generates a counterfactual trajectory according to the trajectory; the decoupling learning module generates a reward function according to the counterfactual trajectory , is the parameter of the reward function; the reward is calculated according to the reward function , according to the reward Generate counterfactual policy network SCM- The parameters of the neural network , is the reverse debate action of the counterfactual strategy at time t, is the reverse debate state of the counterfactual strategy at time t;

[0008] The adversarial learning module includes a controller and a discriminator. The controller controls the counterfactual strategy network SCM- Generate multiple reverse debate actions and control the observation strategy network SCM- Generate multiple forward debate actions. The discriminator judges the authenticity of the forward debate actions and the reverse debate actions according to the transition states of the multiple reverse debate actions and the transition states of the multiple forward debate actions, and puts the true debate action, debate state, transition state of the debate action and reward into the buffer pool as a four-tuple.

[0009] To achieve the above-mentioned purpose of the invention, the present invention also provides a training method for the above-mentioned defensive AIGC adversarial agent, which comprises the following steps:

[0010] S01: Extract a quaternion array from the buffer pool, denoted as ( ,in, is the state of the policy network at time point j, is the observation strategy network The debate action at time point j; is the state of the observation strategy network at time point j+1; is the reward obtained by the observation strategy network at time point j;

[0011] S02: Generate a random number Ψ using standard normal distribution, and Gain value through communication , where is the parameter of the value network;

[0012] S03: Select the optimal action at the next time j+1 by the relevant calculator according to the following formula: , where Yes Status The context vector in , Represents related operations; It is all actions;

[0013] S04: Utilize the target value network according to the following formula Predict the target value at time point j+1: , are the parameters of the target value network;

[0014] S05: Calculate the time variance value by using the time variance calculator according to the following formula: , and use the bit error calculator to calculate the error according to the following formula , is the adjustment coefficient, between 0 and 1;

[0015] S06: Update the parameters of the value network through the following formula for :

[0016] ;

[0017] Update the parameters of the target value network by for :

[0018] ;

[0019] S07: Let the observation strategy network make predictions: ,

[0020] Update the neural network parameters in the observation strategy network using the following formula for :

[0021] , where is the learning rate, For The gradient function of is the gradient function of a;

[0022] S08: Update the target observation strategy network by weighted average The parameters of the neural network in for :

[0023] .

[0024] Compared with the prior art, the defensive AIGC adversarial agent and its training method provided by the present invention have the following beneficial effects: in the counterfactual action generation task, more reasonable counterfactual actions are generated, so that the agent can produce more flexible debate strategies and recognize contexts with different semantics. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a block diagram of the composition of the defensive AIGC adversarial agent provided by the first embodiment of the present invention.

[0026] Figure 2 It is a block diagram of the composition of the RNN neural network provided by the first embodiment of the present invention.

[0027] Figure 3 It is a block diagram of the composition of the RNN neural network provided by the second embodiment of the present invention.

[0028] Figure 4 It is a block diagram of the composition of the RNN neural network provided by the third embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] First embodiment

[0031] Figure 1 FIG. 1 is a block diagram of the composition of the defensive AIGC confrontation agent provided by the first embodiment of the present invention. Figure 1 As shown, the defensive AIGC adversarial agent provided by the first embodiment of the present invention includes a reinforcement learning module, a decoupling learning module, a buffer pool and an adversarial learning module, wherein the reinforcement learning module includes an observation strategy network SCM- , obtain a trajectory through the observation strategy network, where a t is the forward debate action of the observation strategy network at time t, s t is the forward debate state of the observation strategy network at time t, θ is the parameter of the neural network of the observation strategy network, t=1, 2, …, n;

[0032] The decoupled learning module includes a counterfactual trajectory generation module, a decoupled learning module and a counterfactual strategy network, wherein the counterfactual trajectory generation module generates a counterfactual trajectory according to the trajectory; the decoupled learning module generates a reward function according to the counterfactual trajectory , is the parameter of the reward function; the reward is calculated according to the reward function , according to the reward Generate counterfactual policy network SCM- The parameters of the neural network , is the reverse debate action at time t under the counterfactual strategy network, is the reverse debate state of the counterfactual strategy network at time t;

[0033] The adversarial learning module includes a controller and a discriminator. The controller controls the counterfactual strategy network SCM- Generate multiple reverse debate actions and control the observation strategy network SCM- Generate multiple forward debate actions. The discriminator judges the authenticity of the forward debate action and the reverse debate action according to the transition states of the multiple reverse debate actions and the transition states of the multiple forward debate actions, and puts the true debate action, debate state, next debate state and reward into the buffer pool as a four-tuple.

[0034] In the first embodiment,

[0035] is an approximate regression; is the learning rate, g is the discount factor, For The gradient function of .

[0036] In the first embodiment, the actions in the counterfactual trajectory include: attacking the argument, attacking the demonstration, attacking the evidence, etc.

[0037] The present invention can achieve the following beneficial effects through the above technical solution: according to human strategic behavior experience, formulate the debate strategy of the intelligent agent, that is, the counterfactual strategy network SCM- . According to the implicit intention signal and the historical sequence of the agent's action selection, counterfactual actions are generated, thereby expanding the agent's action selection space and avoiding the echo chamber effect caused by overly repetitive strategies.

[0038] In the first embodiment, the state s includes L word vectors x input to the input end of the neural network for representing the weak points of the debate. l and the context vector u representing the L word vectors that represent the weak points of the debate p , l=1,…,L; p=1,…,P; L and M are both integers greater than or equal to 1.

[0039] In the first embodiment, each word vector x l The context vector u p same.

[0040] In the first embodiment, the action and context vector u that generate state s p Related.

[0041] The first embodiment also provides a method for training a defensive AIGC adversarial agent, which comprises the following steps:

[0042] S01: Extract a quaternion array from the buffer pool, denoted as ( ,in, is the state of the policy network at time point j, is the observation strategy network The debate action at time point j; is the state of the observation strategy network at time point j+1; is the reward obtained by the observation strategy network at time point j;

[0043] S02: Generate the first random number Ψ using the standard normal distribution, and Gain value through communication , where is the parameter of the value network;

[0044] S03: Select the optimal action at the next time j+1 by the relevant calculator according to the following formula: , where Yes Status The context vector in , Represents related operations; It is all actions;

[0045] S04: Utilize the target value network according to the following formula Predict the target value at time point j+1: , are the parameters of the target value network;

[0046] S05: Calculate the time variance value by using the time variance calculator according to the following formula: , and use the bit error calculator to calculate the error according to the following formula , is the adjustment coefficient, between 0 and 1;

[0047] S06: Update the parameters of the value network through the following formula for :

[0048] ;

[0049] Update the parameters of the target value network by for :

[0050] ;

[0051] S07: Let the observation strategy network make predictions: ,

[0052] Update the neural network parameters in the observation strategy network using the following formula for :

[0053] , where is the learning rate, For The gradient function of is the gradient function of a;

[0054] S08: Update the target observation strategy network by weighted average The parameters of the neural network in for :

[0055] .

[0056] The present invention can make the observation strategy network SCM- The parameters of the neural network are optimized in the same way that the counterfactual policy network SCM- The parameters of the neural network are optimized. Therefore, in the counterfactual action generation task, more reasonable counterfactual actions are generated, which enables the intelligent agent to generate more flexible debate strategies and recognize different semantic contexts.

[0057] In the first embodiment, the neural network is a RNN neural network.

[0058] Figure 2 is a block diagram of the RNN neural network provided by the first embodiment of the present invention, such as Figure 2 As shown, the RNN neural network provided in the first embodiment includes L input terminals and one output terminal, each of which represents L word vectors x of the weak points of the debate. l , each word vector is inserted into the vector u representing the context p , the relationship between the output and input is:

[0059]

[0060] In the formula, is the output of the lth hidden layer of the RNN neural network, l=1,…,L; is the output of the L-1th hidden layer.

[0061] Training the RNN neural network includes the following steps:

[0062] S2-01: Extract a quaternion array from the buffer pool, denoted as ( ,make , , , ;

[0063] S2-02: Generate the first random number Ψ using the standard normal distribution, and Gain value through communication , where is the parameter of the value network;

[0064] S2-03: Select the optimal action at the next time j+1 through the correlation calculator according to the following formula: , where Yes Status The context vector in , Represents related operations; It is all the action of argument;

[0065] S2-04: Use the target value network according to the following formula Predict the target value at time point j+1: , are the parameters of the target value network;

[0066] S2-05: Calculate the time variance value using the time variance calculator according to the following formula:

[0067] , and use the bit error calculator to calculate the error according to the following formula , is the adjustment coefficient, between 0 and 1;

[0068] S2-06: Update the parameters of the value network through the following formula for :

[0069] ,

[0070] In the formula, is the adjustment factor, For The gradient function of

[0071] Update the parameters of the target value network by for :

[0072] ;

[0073] S2-07: Let the RNN neural network make predictions: ;

[0074] The parameters of the RNN neural network are calculated by the following formula and To optimize:

[0075] ,

[0076] ,

[0077] In the formula, and is the learning rate, is the gradient function of U; is the gradient function of W; For The gradient function of ;

[0078] S2-08: Update the parameters of the target RNN neural network by weighted average:

[0079] ;

[0080] .

[0081] In the first embodiment, .

[0082] The first embodiment can adjust the parameters of the RNN neural network through the above technical solution, thereby generating a debate model that can recognize context.

[0083] Second embodiment

[0084] The second embodiment of the present invention only describes the contents that are different from the first embodiment, and the same contents are not repeated.

[0085] Figure 3 is a block diagram of the RNN neural network provided by the second embodiment of the present invention, such as Figure 3 As shown, the RNN neural network provided by the second embodiment of the present invention includes L input terminals and one output terminal, and each input terminal represents L word vectors x of the weak points of the debate. l , insert the context vector u at the end of the sentence p , where the relationship between the output terminal and the L input terminals is:

[0086]

[0087] In the formula, is the output of the lth hidden layer of the RNN neural network, l=1,…,L; is the output of the L-1th hidden layer of the RNN neural network; It is the output of the L-2th hidden layer of the RNN neural network.

[0088] Training the RNN neural network provided in the second embodiment includes the following steps:

[0089] S3-01: Extract a quaternion array from the buffer pool, denoted as ( ,make , , , ;

[0090] S3-02: Generate the first random number Ψ using the standard normal distribution, and Gain value through communication , where is the parameter of the value network;

[0091] S3-03: Select the optimal action at the next time j+1 by using the correlation calculator according to the following formula: , where Yes Status The context vector in , Represents related operations; It is all the action of argument;

[0092] S3-04: Use the target value network according to the following formula Make a prediction to get the target value at time point j+1: , are the parameters of the target value network;

[0093] S3-05: Calculate the time variance value using the time variance calculator according to the following formula: , and use the bit error calculator to calculate the error according to the following formula , is the adjustment coefficient, between 0 and 1;

[0094] S3-06: Update the parameters of the value network through the following formula for :

[0095] ,

[0096] In the formula, is the adjustment factor, For The gradient function of

[0097] Update the parameters of the target value network by for :

[0098] ;

[0099] S3-07: Make RNN neural network make predictions:

[0100] The parameters of the RNN neural network are calculated by the following formula , , To optimize:

[0101] ,

[0102] ,

[0103]

[0104] In the formula, , and are the gradient functions of U, W, and V respectively, , is the learning rate; For The gradient function of

[0105] S3-08: Update the parameters of the target RNN neural network by weighted average:

[0106] ;

[0107] ;

[0108] .

[0109] In the present invention, .

[0110] The second embodiment can train the RNN neural network into a debate model that can recognize context through the above technical solution.

[0111] Third embodiment

[0112] The multiple embodiments of the present invention only describe the contents different from the first embodiment, and the same contents are not repeated.

[0113] Figure 4 is a block diagram of a RNN neural network provided by a third embodiment of the present invention, such as Figure 4 As shown, the RNN neural network provided by the present invention includes L input terminals and L output terminals, and the first to Bth (B is less than or equal to L) input terminals respectively represent B word vectors x of the weak points of the debate l , insert the context vector u into the Bth input p , the relationship between the Bth output terminal and the input terminal is:

[0114] ,

[0115] In the formula, is the output of the lth hidden layer of the RNN neural network, l=1,…,B; is the output of the B-2th hidden layer of the RNN neural network; is the output of the B-1th hidden layer of the RNN neural network.

[0116] The training of the RNN neural network in the third embodiment is similar to that in the second embodiment, and will not be repeated here. Only the last two steps are given:

[0117] S07: Make RNN neural network make predictions:

[0118]

[0119] The parameters of the RNN neural network are calculated by the following formula , , Optimize, the optimization process includes the following steps:

[0120] ,

[0121] ,

[0122] , where , is the learning rate; , and They are the gradient functions of U, W, and V respectively; For The gradient function

[0123] S08: Update the target RNN neural network optimization parameters by weighted average:

[0124] ;

[0125] ;

[0126] .

[0127] In the third embodiment, .

[0128] The third embodiment can adjust the parameters of the RNN neural network through the above technical solution, thereby generating a debate model that can recognize context.

[0129] Fourth embodiment

[0130] A fourth embodiment of the present invention further provides a program product, which includes a computer program code. The computer program code can be called by a processor to execute the methods in the first to third embodiments.

[0131] The beneficial effects of the method provided by the fourth embodiment are the same as those of the first to third embodiments, and will not be described again here.

[0132] Fifth embodiment

[0133] A fifth embodiment of the present invention provides a computer storage medium storing a computer program code, wherein the computer program code can be called by a processor to execute the methods in the first to third embodiments. The storage medium includes a memory. The processor includes a CPU

[0134] The beneficial effects of the computer storage medium provided by the fourth embodiment are the same as those of the first to third embodiments, and will not be described again here.

[0135] In the present invention, in addition to a single-layer unidirectional RNN neural network, the neural network can also be a bidirectional RNN neural network, a stacked RNN neural network, an LSTM neural network, an attention mechanism neural network, an autonomous intention mechanism neural network and a Transformer neural network, or a combination of one or more of these neural networks.

[0136] The present invention generates more reasonable counterfactual actions in the counterfactual action generation task through decoupled learning, so that the intelligent agent can produce more flexible debate strategies.

[0137] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. The meaning of "several" is one or more, unless otherwise clearly and specifically defined.

[0138] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A system based on defensive AIGC against intelligent agents, characterized in that: It includes reinforcement learning module, inverse reinforcement learning module, buffer pool and adversarial learning module, among which, The reinforcement learning module includes an observation policy network , obtain a trajectory through the observation strategy network, where a t is the forward debate action of the observation strategy network at time t, s t is the forward debate state of the observation strategy network at time t, θ is the parameter of the neural network of the observation strategy network, t=1, 2, …, n; The inverse reinforcement learning module includes a counterfactual trajectory generation module, a decoupling learning module and a counterfactual strategy network, wherein the counterfactual trajectory generation module generates a counterfactual trajectory according to the trajectory; the decoupling learning module generates a reward function according to the counterfactual trajectory , is the parameter of the reward function; the reward is calculated according to the reward function , according to the reward Generating Counterfactual Policy Networks The parameters of the neural network , is the reverse debate action of the counterfactual strategy at time t, is the reverse debate state of the counterfactual strategy at time t; The adversarial learning module includes a controller and a discriminator. The controller controls the counterfactual strategy network. Generate multiple reverse debate actions and control the observation policy network Generate multiple forward debate actions, and the discriminator judges the authenticity of the forward debate action and the reverse debate action according to the transfer states of multiple reverse debate actions and the transfer states of multiple forward debate actions, and puts the true debate action, debate state, reward and the transfer state of the debate action into the buffer pool as a four-tuple; the debate state includes L word vectors x input to the input end of the neural network to represent the weak points of the debate l and the context vector u representing the L word vectors that represent the weak points of the debate p , l=1,…,L; p=1,…,P; L and P are both integers greater than or equal to 1; the reverse argumentative actions in the counterfactual trajectory include: attacking the argument, attacking the reasoning and attacking the evidence.

2. The system based on defensive AIGC against intelligent agents according to claim 1, characterized in that: is an approximate regression; is the learning rate, g is the discount factor, For The gradient function of .

3. The system based on defensive AIGC against intelligent agents according to claim 2, characterized in that: The neural networks in the observation strategy network and the counterfactual strategy network are one or more of the following neural networks: RNN neural network, bidirectional RNN neural network, stacked RNN neural network, LSTM neural network, attention mechanism neural network, autonomous intention mechanism neural network and Transformer neural network.

4. The system based on defensive AIGC against intelligent agents according to claim 3, characterized in that: Each word vector x l The context vector u p same.

5. The system based on defensive AIGC against intelligent agents according to claim 4, characterized in that: The context vector u representing the L word vectors that represent the weak points of the debate p Inserted after each word vector or after the sentence consisting of word vectors.

6. The system based on defensive AIGC against intelligent agents according to claim 5, characterized in that: Debate action and context vector u in debate state p Related.

7. A training method for training an agent in a system based on defensive AIGC adversarial agent according to any one of claims 1 to 6, comprising the following steps: S01: Extract a quaternion array from the buffer pool, denoted as ( ,in, is the state of the policy network at time point j, is the observation strategy network The debate action at time point j; is the state of the observation strategy network at time point j+1; is the reward obtained by the observation strategy network at time point j; S02: Generate a random number Ψ using standard normal distribution, and Gain value through communication , where is the parameter of the value network; S03: Select the optimal action at the next time j+1 by the relevant calculator according to the following formula: , where Yes Status The context vector in , Represents related operations; It is all actions; S04: Utilize the target value network according to the following formula Predict the target value at time point j+1: , are the parameters of the target value network; S05: Calculate the time variance value by using the time variance calculator according to the following formula: , and use the bit error calculator to calculate the error according to the following formula , is the adjustment coefficient, between 0 and 1; S06: Update the parameters of the value network through the following formula for : , In the formula, is the adjustment factor, For The gradient function of Update the parameters of the target value network by for : ; S07: Let the observation strategy network make predictions: ; Update the neural network parameters in the observation strategy network using the following formula for : , where is the learning rate, For The gradient function of is the gradient function of a; S08: Update the target observation strategy network by weighted average The parameters of the neural network in for : 。