A data security defense system based on multi-agent reinforcement learning

Abstract

The application relates to the field of reinforcement learning and network security technology, in particular to a data security defense system based on multi-agent reinforcement learning, which comprises a data poisoning attack filtering module, a cooperative defense module and an interactive reward confidence module; the data poisoning attack filtering module is arranged at a sensor end and filters state disturbance attacks on state data through a pre-trained poisoning attack filtering network; the cooperative defense module is arranged at a controller end of the system, performs consistency analysis on the filtered state data, and adjusts a communication cooperation strategy; the interactive reward confidence module is arranged at an actuator end of the system and outputs the confidence degree of interactive rewards among the multi-agents. Through data acquisition, system interaction and algorithm learning in the multi-agent reinforcement learning training stage, the data poisoning attack filtering module, the system cooperative defense module and the interactive reward confidence module are used to avoid or alleviate the data poisoning attack problem in the multi-agent system reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a data security defense system based on multi-agent reinforcement learning. Background Technology

[0002] In recent years, the cooperative control of multi-agent systems has been widely applied in civilian and defense fields. While networked multi-agent systems bring convenience, they also pose a potential risk of malicious attacks.

[0003] Currently, multi-agent systems (MAS) differ from general systems in their larger scale, more complex network structures and communication protocols, and the interactions between agents within the system mean that malicious attacks on certain communication links or critical equipment can severely impact the secure operation of the entire system. Compared to traditional information system security, which primarily focuses on protecting the security, confidentiality, and integrity of the information itself, MAS security must also consider the impact of malicious attacks in both cyberspace and physical space on the system's performance. This presents a significant challenge to MAS security research. Therefore, the theoretical and technological research on MAS security has become a crucial research topic urgently needing resolution in the development and application of security control.

[0004] Currently, multi-agent reinforcement learning security defense models have few defenses against data poisoning attacks during the training phase, and lack a poisoning defense method that involves the joint participation of multiple agent reinforcement learning security subjects and objects. Summary of the Invention

[0005] This invention provides a data security defense system based on multi-agent reinforcement learning to address the lack of defense measures against data poisoning attacks during the training phase in existing technologies. By designing corresponding data poisoning attack filtering modules, system collaborative defense modules, and interactive reward confidence modules from multiple perspectives such as data acquisition, system interaction, and algorithm learning during the training phase of multi-agent reinforcement learning, the system can avoid or mitigate data poisoning attack problems that may occur in multi-agent reinforcement learning systems.

[0006] This invention provides a data security defense system based on multi-agent reinforcement learning, comprising a data poisoning attack filtering module, a collaborative defense module, and an interactive reward confidence module connected in sequence.

[0007] The data poisoning attack filtering module is located at the sensor end of the system. The receiving end of the data poisoning attack filtering module is connected to several neighboring agents, receives the state data perceived by each neighboring agent, and filters the state perturbation attacks generated in the state data through a poisoning attack filtering network pre-trained based on the adversarial training method.

[0008] The collaborative defense module is located on the controller of the system. The collaborative defense module receives filtered state data, performs consistency analysis on the filtered state data, and adjusts the communication and cooperation strategies of multiple neighboring intelligent agents.

[0009] The interaction reward confidence module is set at the executor end of the system. It obtains the value function assigned to each agent during the multi-agent reinforcement learning training process, adds a confidence control function to the reward of each agent to address the confidence poisoning attack that exists between agents during the training process, and outputs the confidence degree of the interaction reward between multiple agents.

[0010] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, wherein the data poisoning attack filtering module includes an adversarial training perturbation sample generation module and a poisoned data filtering network training module.

[0011] The adversarial training perturbation sample generation module is used to pre-set several adversarial training perturbation samples containing noise factors before the multi-agent reinforcement learning training process.

[0012] The poisoning data filtering network model is trained by designing the filtering network structure and setting the filtering network parameters based on the characteristics of the adversarial training samples, and training the poisoning data filtering network based on the training samples.

[0013] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, wherein the training method of the poisoning attack filtering network pre-trained based on adversarial training includes:

[0014] Before the training process of multi-agent reinforcement learning, several adversarial training perturbation samples δ containing noise factors are preset. t ;

[0015] Based on adversarial training samples δ t Training the poison attack filtering network;

[0016] During multi-agent reinforcement learning training, a poisoning attack filtering network is connected to the sensor terminals of the multi-agents to filter the environmental state data perceived by the sensors.

[0017] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, wherein the collaborative defense module includes a security consistency analysis submodule, a topology switching control submodule, a channel switching control submodule, and a supervisory control submodule;

[0018] The security consistency analysis submodule detects and analyzes the current state data of the multi-agents according to the security consistency analysis algorithm, and provides adjustment strategies for the communication topology switching and communication channel switching of the multi-agents.

[0019] The topology switching control submodule is used to provide a multi-agent communication topology switching method under poisoning attack;

[0020] The channel switching control submodule is used for optimizing multi-agent communication requirements, triggering channel switching, and matching channel selection under poisoning attacks.

[0021] The supervisory control submodule is used to monitor the abnormal behavior of multi-agents under poisoning attacks, run a supervisory verifier to determine whether the behavior of multi-agents violates expected constraints, and compensate for the multi-agent policy behavior according to the compensator.

[0022] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, wherein the channel switching control submodule is used to optimize communication strategies and select communication channels under poisoning attacks;

[0023] The communication strategy optimization includes: establishing optimization task objectives; multiple agents using learning algorithms to determine the time to send information, the type of information to send, the content of the information to send, and the target agent to receive the information based on the current state information;

[0024] The communication channel selection includes: obtaining the communication channel attenuation rate under different attack states based on the channel model, obtaining the equivalent attenuation rate corresponding to each channel, and selecting the channel with the lowest equivalent attenuation rate.

[0025] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, wherein the confidence control function takes maximizing the global reward value as the final optimization objective for generating the reward value of a single multi-agent. When the trend of the interaction reward is opposite to the trend of the global reward, the confidence of the interaction reward is reduced, and vice versa.

[0026] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, wherein the confidence control function generation step includes:

[0027] Before reinforcement learning begins, the impact between agents and on the global picture is evaluated through simulated prediction, and an initial value allocation scheme is generated based on this.

[0028]

[0029] At the start of reinforcement learning, the overall value Q of multiple agents is used as a basis. tol And the value Q of each agent j Impact on output value allocation weights:

[0030]

[0031] The confidence control function generates a confidence level γ for reward allocation weights based on the changes in weights. When the trend of change in interactive reward is opposite to the direction of change in global reward, the confidence level of interactive reward is decreased; conversely, the confidence level of interactive reward is increased. That is, the confidence level of reward allocation weights in the k-th reinforcement learning iteration is:

[0032]

[0033] Where, ω n The value function decomposition network assigns reward weights to agent n; Q tol This represents the overall reward for multi-agent reinforcement learning in response to feedback from the environment.

[0034] On the other hand, a data security defense method based on the multi-agent reinforcement learning data security defense system includes:

[0035] The data poisoning attack filtering module filters out state perturbation attacks generated by multiple neighboring agents.

[0036] The collaborative defense module receives filtered state data, performs consistency analysis on the filtered state data, and adjusts the communication and cooperation strategies of multiple neighboring agents.

[0037] The interaction reward confidence module obtains the value function assigned to each agent during the multi-agent reinforcement learning training process. To address the confidence poisoning attack that exists between agents during the training process, a confidence control function is added to the reward of each agent, and the confidence degree of the interaction reward between the multi-agents is output.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the data security defense method described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data security defense method described above.

[0040] The present invention provides a data security defense system based on multi-agent reinforcement learning, which has at least the following technical effects, including:

[0041] (1) To address the vulnerability of multi-agent reinforcement learning training data to state perturbation attacks, a poisoning attack filtering network is trained at the sensor end based on adversarial training to eliminate adversarial perturbations and filter out the perturbations that are difficult to detect caused by state perturbation attacks.

[0042] (2) To address potential communication attacks and malicious node attacks, a multi-agent system collaborative interaction security defense module is designed to improve the security performance of the multi-agent system in collaborative control and communication mechanisms.

[0043] (3) Add a confidence control function to the reward of each agent so that the agent can continuously understand the contribution of its own actions and the confidence level of the contribution during the learning process;

[0044] (4) To address the various security vulnerabilities that may arise in the training phase of a multi-agent reinforcement learning system, the complex defense tasks under the global security objectives can be efficiently completed through the connection and collaboration of security participants of different types and focuses in the multi-agent composition. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the data security defense system based on multi-agent reinforcement learning provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the poisoning attack filtering network provided in an embodiment of the present invention;

[0048] Figure 3 This is a flowchart illustrating the security consistency analysis submodule provided in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the channel switching control submodule provided in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the interactive reward confidence module provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.

[0053] It should be noted that the terms "first" and "second" used in this invention merely distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permissible. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those described or illustrated herein.

[0054] In one embodiment, such as Figure 1 As shown, the present invention provides a data security defense system based on multi-agent reinforcement learning, comprising a data poisoning attack filtering module, a collaborative defense module, and an interactive reward confidence module connected in sequence.

[0055] The data poisoning attack filtering module is located at the sensor end of the system. The receiving end of the data poisoning attack filtering module is connected to several neighboring agents, receives the state data perceived by each neighboring agent, and filters the state perturbation attacks generated in the state data through a poisoning attack filtering network pre-trained based on the adversarial training method.

[0056] The collaborative defense module is located on the controller of the system. The collaborative defense module receives filtered state data, performs consistency analysis on the filtered state data, and adjusts the communication and cooperation strategies of multiple neighboring intelligent agents.

[0057] The interaction reward confidence module is set at the executor end of the system. It obtains the value function assigned to each agent during the multi-agent reinforcement learning training process, adds a confidence control function to the reward of each agent to address the confidence poisoning attack that exists between agents during the training process, and outputs the confidence degree of the interaction reward between multiple agents.

[0058] It should be noted that the application scenario of the data security defense system based on multi-agent reinforcement learning provided by this invention is to solve the security risks of multi-agent reinforcement learning systems during the training phase. Specifically, this manifests as follows: within a multi-agent system, agents communicate and cooperate with each other to transmit data through a network and coordinate their tasks. In this collaborative communication process, the open environment of the multi-agent system brings various security risks. For example, various communication behaviors during agent training can lead to different forms of communication message poisoning attacks. As agents continuously explore the value of their states, the actions of other agents can affect the cooperative state perceived by the agent. Therefore, malicious neighboring agents can use invalid or even anti-cooperative actions to poison the state data perceived by the agent.

[0059] like Figure 2 As shown, a poisoned data filtering module is added to the sensor of the multi-agent system. The poisoned data filtering network filters the perceived state data for poisoning perturbation. The data poisoning attack filtering module includes an adversarial training perturbation sample generation module and a poisoned data filtering network training module.

[0060] The adversarial training perturbation sample generation module is used to pre-set several adversarial training perturbation samples containing noise factors before the multi-agent reinforcement learning training process.

[0061] The poisoning data filtering network model is trained by designing the filtering network structure and setting the filtering network parameters based on the characteristics of the adversarial training samples, and training the poisoning data filtering network based on the training samples.

[0062] Specifically, according to the data security defense system based on multi-agent reinforcement learning provided by the present invention, during the training process of multi-agent reinforcement learning, when the reinforcement learning model is subjected to a poisoning attack, the state perception process of the individual agent sensors in the system may be affected by the environmental state data poisoning attack, causing the agent to observe not the real environmental state, but a small residual δ_t perturbation state containing noise factors, denoted as s_t^′=s_t+δ_t. The agent's exploration learning and action value update under the carefully designed perturbation state will be distorted or interfered with to a certain extent, resulting in poor performance of the trained action value function in the non-perturbation execution stage. To address this state perturbation attack δ_t that is difficult to detect during the training stage, a poisoning attack filtering network is designed at the sensor end based on an adversarial training method to eliminate adversarial perturbations, filtering out the small perturbations that are difficult to detect caused by the state perturbation attack. The training method of the poisoning attack filtering network pre-trained based on the adversarial training method includes:

[0063] Before the training process of multi-agent reinforcement learning, several adversarial training perturbation samples δ containing noise factors are preset. t ;

[0064] Based on adversarial training samples δ t Training the poison attack filtering network;

[0065] During multi-agent reinforcement learning training, a poisoning attack filtering network is connected to the sensor terminals of the multi-agents to filter the environmental state data perceived by the sensors.

[0066] By using attack filtering methods in adversarial training, the impact of state data poisoning attacks on individual agents during policy learning and training can be reduced, the agent's perception of environmental states can be improved, the action selection accuracy of the agent's reinforcement learning value function can be increased, and the robustness of reinforcement learning models to added sample perturbation types and random noise can be enhanced.

[0067] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, wherein the collaborative defense module includes a security consistency analysis submodule, a topology switching control submodule, a channel switching control submodule, and a supervisory control submodule;

[0068] Among them, such as Figure 3 As shown, the security consistency analysis submodule detects and analyzes the current state data of the multi-agent based on the security consistency analysis algorithm, and provides adjustment strategies for the communication topology switching and communication channel switching of the multi-agent;

[0069] The topology switching control submodule is used to provide a multi-agent communication topology switching method under poisoning attack;

[0070] The channel switching control submodule is used for optimizing multi-agent communication requirements, triggering channel switching, and matching channel selection under poisoning attacks.

[0071] The supervisory control submodule is used to monitor the abnormal behavior of multi-agents under poisoning attacks, run a supervisory verifier to determine whether the behavior of multi-agents violates expected constraints, and compensate for the multi-agent policy behavior according to the compensator.

[0072] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, such as... Figure 4 As shown, the channel switching control submodule is used to optimize communication strategies and select communication channels under poisoning attacks;

[0073] The communication strategy optimization includes: establishing optimization task objectives; multiple agents using learning algorithms to determine the time to send information, the type of information to send, the content of the information to send, and the target agent to receive the information based on the current state information;

[0074] Specifically, in the event of a malicious attack on a multi-agent system, the design of its response strategy is implemented using a security consistency analysis algorithm.

[0075] By detecting and analyzing the current system state, a consistent and effective adjustment strategy is provided for communication topology switching and communication channel switching in multi-agent systems. Security consensus analysis algorithms can be divided into two categories according to their defense type: detection-based security consensus algorithms and tolerance-based security consensus algorithms. Specifically:

[0076] Attack detection-based security consensus algorithms sort and detect nodes in an agent system, eliminating malicious nodes or agents. Then, they re-count the remaining safe nodes and, based on a control protocol, utilize this set of safe nodes to defend against malicious attacks, maintaining the multi-agent system's state within a safe range and ultimately achieving secure consensus. An observer-based distributed control strategy is designed, forming a feedback control gain matrix based on observations of agent adjacency elements. This allows followers to track the leader even with limited communication links, achieving leader-follower secure consensus control under certain conditions.

[0077] Tolerant-type secure consensus algorithms do not require the detection of malicious nodes. When malicious attacks are highly concealed or the detection cost is high, multi-agent systems can tolerate the presence of malicious nodes in the system. By adding extra nodes or incorporating constraints into the secure consensus protocol, the connectivity of the multi-agent system network can be improved. By eliminating extreme nodes, the state of the remaining participating nodes is not excessively affected by extreme values ​​when updating their own states, ensuring that they remain within a safe range.

[0078] The communication channel selection includes: obtaining the communication channel attenuation rate under different attack states based on the channel model, obtaining the equivalent attenuation rate corresponding to each channel, and selecting the channel with the lowest equivalent attenuation rate.

[0079] This invention provides a data security defense system based on multi-agent reinforcement learning. Figure 5 The diagram illustrates the interaction reward confidence module provided in this embodiment of the invention. The nonlinear network in the diagram is a value function decomposition network commonly used in multi-agent reinforcement learning algorithms, and the value function assigned to each agent is based on this nonlinear network. Due to the existence of data poisoning attacks, the interaction evaluations based on other agents in the network are unconfident; the reward allocation weights of the agents in the nonlinear network... They describe the current action a respectively. i Overall value Q tol Other intelligent agent value Q jThe impact of this is that when a malicious agent rewards low-value actions with high interaction rewards, the agent's value function will be affected or even controlled as the policy is continuously updated.

[0080] Therefore, in order to generate individual evaluations that do not affect policy learning, maximizing the global reward value is taken as the final optimization objective for generating individual reward values. At the same time, a confidence evaluation that can reflect the degree of influence of the agent's own actions on other agents is introduced.

[0081] The confidence control function takes maximizing the global reward value as the final optimization objective for generating the reward value of a single multi-agent. When the trend of the interaction reward is opposite to the trend of the global reward, the confidence of the interaction reward is reduced, and vice versa.

[0082] According to the present invention, a data security defense system based on multi-agent reinforcement learning is provided, wherein the confidence control function generation step includes:

[0083] Before reinforcement learning begins, the impact between agents and on the global picture is evaluated through simulated prediction, and an initial value allocation scheme is generated based on this.

[0084]

[0085] At the start of reinforcement learning, the overall value Q of multiple agents is used as a basis. tol And the value Q of each agent j Impact on output value allocation weights:

[0086]

[0087] The confidence control function generates a confidence level γ for reward allocation weights based on the changes in weights. When the trend of change in interactive reward is opposite to the direction of change in global reward, the confidence level of interactive reward is decreased; conversely, the confidence level of interactive reward is increased. That is, the confidence level of reward allocation weights in the k-th reinforcement learning iteration is:

[0088]

[0089] Where, ω n The value function decomposition network assigns reward weights to agent n; Q tol This represents the overall reward received by multi-agent reinforcement learning in response to feedback from the environment.

[0090] In this way, other agents can be rewarded when they have a positive impact, enabling the multi-agent system to continuously clarify the contribution of agent actions and the confidence level of those contributions during the learning process.

[0091] On the other hand, a data security defense method based on the multi-agent reinforcement learning data security defense system includes:

[0092] The data poisoning attack filtering module filters out state perturbation attacks generated by multiple neighboring agents.

[0093] The collaborative defense module receives filtered state data, performs consistency analysis on the filtered state data, and adjusts the communication and cooperation strategies of multiple neighboring agents.

[0094] The interaction reward confidence module obtains the value function assigned to each agent during the multi-agent reinforcement learning training process. To address the confidence poisoning attack that exists between agents during the training process, a confidence control function is added to the reward of each agent, and the confidence degree of the interaction reward between the multi-agents is output.

[0095] The present invention also provides an electronic device, which may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other through the communication bus. The processor can invoke logical instructions in the memory to execute the steps of the data security defense method based on the multi-agent reinforcement learning data security defense system provided above.

[0096] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the steps of the data security defense method based on the multi-agent reinforcement learning data security defense system provided by the above method.

[0098] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data security defense method based on the multi-agent reinforcement learning data security defense system provided by the above method.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data security defense system based on multi-agent reinforcement learning, characterized in that, This includes a data poisoning attack filtering module, a collaborative defense module, and an interactive reward confidence module, which are connected in sequence. The data poisoning attack filtering module is located at the sensor end of the system. The receiving end of the data poisoning attack filtering module is connected to several neighboring agents, receives the state data perceived by each neighboring agent, and filters the state perturbation attacks generated in the state data through a poisoning attack filtering network pre-trained based on the adversarial training method. The collaborative defense module is located on the controller of the system. The collaborative defense module receives filtered state data, performs consistency analysis on the filtered state data, and adjusts the communication and cooperation strategies of multiple neighboring intelligent agents. The interaction reward confidence module is set at the executor end of the system. It obtains the value function assigned to each agent during the multi-agent reinforcement learning training process. In response to the confidence poisoning attack that exists between agents during the training process, it adds a confidence control function to the reward of each agent and outputs the confidence degree of the interaction reward between the multi-agents. The steps for generating the confidence control function include: Before reinforcement learning begins, the impact of agents on each other and on the global picture is evaluated through simulated prediction to generate an initial value allocation scheme. ; At the start of reinforcement learning, the overall value of multiple agents is considered. and the value of each intelligent agent Impact on output value allocation weights: ; The confidence control function generates confidence levels by assigning weights to rewards based on changes in weights. When the trend of change in interactive reward is opposite to the direction of change in global reward, the confidence level of interactive reward is decreased; conversely, the confidence level of interactive reward is increased. The confidence level of the reward allocation weights in the next reinforcement learning iteration is: ; in, The representation value function decomposes the network into agents. The allocated reward weights; This represents the overall reward for multi-agent reinforcement learning in response to feedback from the environment.

2. The data security defense system based on multi-agent reinforcement learning according to claim 1, characterized in that, The data poisoning attack filtering module includes an adversarial training perturbation sample generation module and a poisoned data filtering network training module. The adversarial training perturbation sample generation module is used to pre-set several adversarial training perturbation samples containing noise factors before the multi-agent reinforcement learning training process. The poisoning data filtering network model is trained by designing the filtering network structure and setting the filtering network parameters based on the characteristics of the adversarial training samples, and training the poisoning data filtering network based on the training samples.

3. A data security defense system based on multi-agent reinforcement learning according to claim 2, characterized in that, The training method for the poisoning attack filtering network pre-trained based on adversarial training includes: Before the training process of multi-agent reinforcement learning, several adversarial training perturbation samples containing noise are pre-set. ; Based on adversarial training samples Training the poison attack filtering network; During multi-agent reinforcement learning training, a poisoning attack filtering network is connected to the sensor terminals of the multi-agents to filter the environmental state data perceived by the sensors.

4. A data security defense system based on multi-agent reinforcement learning according to claim 1, characterized in that, The collaborative defense module includes a security consistency analysis submodule, a topology switching control submodule, a channel switching control submodule, and a monitoring control submodule. The security consistency analysis submodule detects and analyzes the current state data of the multi-agents according to the security consistency analysis algorithm, and provides adjustment strategies for the communication topology switching and communication channel switching of the multi-agents. The topology switching control submodule is used to provide a multi-agent communication topology switching method under poisoning attack; The channel switching control submodule is used for optimizing multi-agent communication requirements, triggering channel switching, and matching channel selection under poisoning attacks. The supervisory control submodule is used to monitor the abnormal behavior of multi-agents under poisoning attacks, run a supervisory verifier to determine whether the behavior of multi-agents violates expected constraints, and compensate for the multi-agent policy behavior according to the compensator.

5. A data security defense system based on multi-agent reinforcement learning according to claim 4, characterized in that, The channel switching control submodule is used to optimize communication strategies and select communication channels under poisoning attacks; The communication strategy optimization includes: establishing optimization task objectives; multiple agents using learning algorithms to determine the time to send information, the type of information to send, the content of the information to send, and the target agent to receive the information based on the current state information; The communication channel selection includes: obtaining the communication channel attenuation rate under different attack states based on the channel model, obtaining the equivalent attenuation rate corresponding to each channel, and selecting the channel with the lowest equivalent attenuation rate.

6. A data security defense system based on multi-agent reinforcement learning according to claim 1, characterized in that, The confidence control function takes maximizing the global reward value as the final optimization objective for generating the reward value of a single multi-agent. When the trend of the interaction reward is opposite to the trend of the global reward, the confidence of the interaction reward is reduced, and vice versa.

7. A data security defense method based on the multi-agent reinforcement learning data security defense system according to any one of claims 1-6, characterized in that, include: The data poisoning attack filtering module filters out state perturbation attacks generated by multiple neighboring agents. The collaborative defense module receives filtered state data, performs consistency analysis on the filtered state data, and adjusts the communication and cooperation strategies of multiple neighboring agents. The interaction reward confidence module obtains the value function assigned to each agent during the multi-agent reinforcement learning training process. To address the confidence poisoning attack that exists between agents during the training process, a confidence control function is added to the reward of each agent, and the confidence degree of the interaction reward between the multi-agents is output.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data security defense method as described in claim 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the data security defense method as described in claim 7.

Images