A multi-agent federal reinforcement learning poisoning attack defense method based on neuron-level multi-dimensional feature reconstruction

Abstract

The application discloses a kind of multi-agent federal reinforcement learning poisoning attack defense method based on neuron level multi-dimensional feature reconstruction, it is related to distributed machine learning and system security technical field.For the problem that existing global defense mechanism faces high concealment sparse poisoning attack and exists dimension disaster and low detection efficiency, the application sinks abnormal review granularity to neuron level low-dimensional feature space.The method first extracts amplitude distortion, direction consistency and distribution distortion and the like multi-dimensional statistical features according to linear layer to the local model uploaded by each agent and reduces dimension and splices;Second, the feature vector is reconstructed using feature auto-encoder in an unsupervised manner, and the node security risk is evaluated by mean square reconstruction error;Finally, based on the historical normal error sliding window, the interception threshold is dynamically calculated, the benign nodes are screened to perform adaptive fault-tolerant safety aggregation, and the safety features are used to fine-tune the auto-encoder online.The application can accurately intercept unknown concealed attacks without prior attack knowledge, effectively adapt to the concept drift of multi-agent strategy evolution, and significantly improve the robustness of the system.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security and distributed machine learning, specifically involving a poisoning attack defense method based on neuron-level multidimensional feature reconstruction in a multi-agent federated reinforcement learning scenario. Background Technology

[0002] In recent years, with the continuous breakthroughs in artificial intelligence technology and the accelerating pace of industrialization, multi-agent systems have played an increasingly important role in complex collaborative tasks such as drone swarm collaboration and joint scheduling of autonomous vehicle fleets. Among them, multi-agent federated reinforcement learning combines the privacy-preserving characteristics of federated learning with the decision-making capabilities of reinforcement learning: agents are trained in local environments and only upload model updates to a central server for aggregation. This breaks down data silos, ensures privacy, and reduces communication overhead, making it a key technological path for the implementation of large-scale distributed intelligent applications.

[0003] However, this decentralized collaborative architecture and open interaction environment also introduce unprecedented security challenges. During the distributed collaboration phase of federated training, the system faces a severe internal Byzantine threat. Because the central server struggles to verify the authenticity of each agent's local private environment and interaction data, a small number of captured malicious nodes or hijacked edge nodes can upload carefully constructed poisoned model parameters to suppress normal network gradients, thereby disrupting the convergence of the global policy.

[0004] To defend against such model poisoning attacks, traditional Byzantine fault-tolerant defense mechanisms, such as Krum, TrimmedMean, and Median, are mostly based on spatial distribution metrics for updating model parameters via multi-node uploads. These defenses compare the global Euclidean distance of the update vectors from each node or... Norms, at the macroscopic parameter level, isolate anomalous poisoning models that produce significant spatial shifts. However, in practical applications, such detection algorithms based on global model distances or independent coordinate axis statistics completely ignore the underlying structural couplings and feature relationships between neural network layers, exhibiting significant limitations.

[0005] These limitations have further spurred the development of more targeted and destructive, highly covert sparse attack methods, such as critical path poisoning. Advanced attackers have abandoned the blind perturbation of all parameters, instead using activation values ​​and gradients for sensitivity checks and specifically selecting highly influential critical neurons. These attacks, through highly sparse, specific structural modifications, significantly reduce the degree of anomalous deflection in the global macroscopic parameter space, successfully circumventing traditional methods with minimal modification cost. Norms and other methods are used for distance-based inspection of the global model's variation. Furthermore, in the complex dynamic learning environments of multi-agent systems, different benign agents often upload models with significant exploration noise and high statistical variance due to dynamic environmental changes and distributed trial and error. Faced with covert poisoning attacks that combine highly sparse structures with minute parameter perturbations, relying solely on anomaly inspection in the macroscopic parameter space suffers from a severe "curse of dimensionality": the defense system is easily obscured by massive amounts of normal exploration noise, making it difficult to accurately screen for malicious anomalies hidden in high-dimensional updates. It may also excessively truncate high-value normal node gradients during reinforcement learning, leading to system paralysis. Therefore, there is an urgent need for a defense method that can overcome the dimensionality curse bottleneck of global distance defense, adapt to complex concept drift and exploration noise, and possess microscopic inspection accuracy to intercept unknown, highly covert poisoning attacks. Summary of the Invention

[0006] To address the "curse of dimensionality" and poor detection capabilities in existing multi-agent federated reinforcement learning systems for defending against attacks on hidden nodes, this invention proposes a poisoning attack defense method based on neuron-level multidimensional feature reconstruction in multi-agent federated reinforcement learning. This invention aims to overcome the "curse of dimensionality" bottleneck of traditional global defenses by fitting the feature space of normal policy updates in an unsupervised manner, achieving accurate interception and robust aggregation of attacks on unknown hidden nodes without prior attack knowledge.

[0007] This invention is achieved through the following technical solution: A multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction, comprising the following steps:

[0008] Step 1: In the first In round-fed communication, the central server receives data from participants in the system's federated aggregation. Local models uploaded by individual agents In conjunction with the global benchmark model issued in the previous round, Fine-grained feature extraction at the neuron level is performed on the local models submitted by each agent. This is done for each linear layer in the policy network. Independent extraction of intelligent agents The amplitude distortion features, orientation consistency features, and distribution distortion features updated in this layer are concatenated to form the local statistical feature vector of this layer. ;

[0009] intelligent agents All Local statistical feature vectors of linear layers By concatenating the network topology, the high-dimensional, extremely sparse model parameters are updated and mapped into a low-dimensional, high-information-density complete statistical feature vector. .

[0010] Step 2: Utilize a pre-built and initialized feature autoencoder For statistical eigenvectors Unsupervised feature reconstruction is performed; the feature autoencoder includes an encoder. and decoder The encoder converts the feature vector The features are mapped to a low-dimensional latent variable space to remove redundant noise, and then reconstructed using a decoder. Calculate reconstructed features Compared with the original input features Mean square reconstruction error between Using this as an intelligent agent In the Safety risk assessment indicators for wheels.

[0011] Step 3: Design a dynamic threshold update mechanism based on a historical error sliding window to address the "concept drift" phenomenon caused by the dynamic evolution of policy distribution during multi-agent exploration. The central server maintains a threshold update mechanism of length [missing information] in memory. Pure historical error sliding window Calculate the empirical mean within the sliding window. with sample standard deviation Based on the Raida criterion and combined with the established sensitivity control coefficient Calculate the current number Dynamic interception threshold of the wheel .

[0012] Step 4: Connect each agent Reconstruction error With dynamic interception threshold Compare and perform anomaly detection and screening. If If the model submitted by the agent is found to have been subjected to a covert poisoning attack, it will be removed from the current aggregation set; if If it is safe, it will be included in the safe set for the current round. middle.

[0013] Step 5: The central server extracts the sets deemed safe. Perform adaptive fault-tolerant safe aggregation and complete the online evolution update of the autoencoder. If the safe set is not empty ( Then, the updated global model is obtained by arithmetically averaging the local models within the safe set. Simultaneously, the reconstruction error of the safety set is pushed into the historical error sliding window. In the middle, and use a subset of safe features to perform local gradient descent fine-tuning on the feature autoencoder; if all nodes in this round are judged as abnormal, resulting in an empty safe set ( If the parameter update operation fails, the fault-tolerant rollback mechanism will be triggered, directly refusing to perform any parameter update operation and forcing the use of the previous round's global baseline model. As a distribution model.

[0014] Furthermore, in step 1 above, for the policy network... Includes A linear layer with n parameters has a parameter update matrix of... The specific methods for extracting three-dimensional deep statistical features involved are as follows: (1) Amplitude distortion features: extract the flattened matrix norm and norm (2) Directional consistency feature: Calculate the cosine similarity between each independent neuron (parameter matrix row vector) and the corresponding neuron of the global benchmark.

[0015] Statistically analyze and extract the proportion of nodes with negative cosine similarity. and the average cosine similarity of all neurons in that layer. (3) Distribution distortion characteristics: Flatten the parameter update matrix of this layer into a one-dimensional vector, and calculate the sample finite variance of all parameter update elements. Third-order standardized skewness, which characterizes the degree of asymmetry in the data distribution. By concatenating the above statistics, we obtain the local six-dimensional feature vector of this layer: .

[0016] Furthermore, in step 2 above, the loss function of the autoencoder is defined as minimizing the mean squared reconstruction error of the normal features: The formula for calculating the mean square reconstruction error is as follows:

[0017]

[0018] When the system encounters a poisoned model with local parameter tampering, its feature vectors deviate from the normal policy update and optimization trajectory implicitly learned by the autoencoder. As a result, the decoder cannot effectively reconstruct the model using the existing normal pattern, thus generating a huge mean square reconstruction error.

[0019] Furthermore, in step 3 above, the current number is calculated. Dynamic interception threshold of the wheel The formula is as follows:

[0020]

[0021] in, and The computational data comes from the set of reconstruction errors of all historical safe nodes within the sliding window, which allows the decision boundary to evolve smoothly with the changes in the benign policy exploration of the multi-agent system.

[0022] Compared with the prior art, the present invention has the following beneficial effects and advantages:

[0023] 1. Breaking through the bottleneck of dimensional disaster and achieving precise positioning of sparse perturbations: The granularity of anomaly review is reduced to the microscopic neuron-level low-dimensional feature space. By systematically extracting the amplitude distortion features, direction consistency features, and distribution distortion features of parameter updates, the small malicious perturbations targeting local nodes are amplified, thereby effectively detecting and intercepting highly covert model poisoning attacks.

[0024] 2. Introducing unsupervised autoencoder detection significantly improves scenario adaptability: It eliminates the dependence on prior labels or positive / negative data types of attack samples. Without knowing the attacker's specific poisoning strategy, it can accurately identify and intercept attacks on unknown nodes simply by learning the reconstruction error distribution pattern of normal updated data in an unsupervised manner.

[0025] 3. Design an adaptive dynamic threshold mechanism to effectively address data distribution shift: By maintaining a clean historical error sliding window and combining it with an online fine-tuning mechanism, the anomaly detection boundary can adaptively evolve with the distribution shift during the benign policy evolution of the multi-agent system. This maintains a tight security line during the model policy exploration and later training stages, effectively balancing the system's defense security with the global model's convergence performance.

[0026] 4. Constructing a security barrier for multi-agent collaborative training to achieve robust training in complex environments: Addressing the model poisoning threat faced in multi-agent environments, a complete defense algorithm from fine-grained feature extraction to adaptive fault-tolerant aggregation is proposed. This method can accurately remove the poisoned model from malicious nodes, effectively resisting the disruption of global collaborative strategies by malicious agents. It not only ensures the training security of multi-agent clusters in a distributed architecture but also guarantees the normal evolution of benign agent strategies within the system, ultimately achieving robust convergence of the multi-agent system in complex adversarial environments. Attached Figure Description

[0027] Figure 1 This is a diagram of the neuron-level fine-grained feature extraction architecture in an embodiment of the present invention.

[0028] Figure 2 This is a diagram of the overall framework architecture of the federated defense aggregation based on autoencoders and dynamic thresholds in an embodiment of the present invention. Detailed Implementation

[0029] The following section, in conjunction with the specification and the operating mechanism of the multi-agent federated reinforcement learning system, provides a more detailed description of the implementation steps of the multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction proposed in this patent.

[0030] Example: This method de-emphasizes the anomaly review granularity of the defense mechanism from the macroscopic model level to the microscopic neuron-level low-dimensional feature space. Its main execution flow consists of four stages: system initialization and security warm-up, fine-grained feature extraction, anomaly detection based on autoencoders and dynamic thresholds, and adaptive fault-tolerant security aggregation and online fine-tuning. The specific implementation steps are as follows:

[0031] Phase 1: System Initialization and Safety Warm-up Phase In the initial phase of federated reinforcement learning, a short safety warm-up period is established. All agents conduct preliminary exploration and interactive training based on their respective local environments. The central server collects these normal local model updates and extracts the statistical feature vectors of benign updates according to the method in Step 1. Then, these benign features are used to train a pre-established unsupervised autoencoder. This allows the encoder to filter redundant exploration noise and extract distribution patterns of normal strategies, while the decoder can reconstruct features from the latent space. At the end of the warm-up period, the mean square reconstruction error of the normal features will be calculated and filled into an initial length of [missing information]. History sliding window This lays the foundation for subsequent online testing.

[0032] Phase Two: Fine-grained Multidimensional Statistical Feature Extraction (corresponding to Steps 1 and 2) In the actual operation of the... In round-fed communications, perform the following operations:

[0033] Step 1.1: Amplitude distortion feature extraction. The central server receives data from the agents participating in the aggregation. Submitted local model And compared with the global benchmark model issued in the previous round. Perform a layer-by-layer decoupling review. This applies to the first layer of the policy network. Linear layers (parameter matrix) ), computational intelligent agents The parameter update matrix of this layer , its first The update vector for the row is Subsequently, the extraction will Flattened into a one-dimensional vector norm With the infinite norm It is used to capture local parameter spikes caused by malicious tampering.

[0034] Step 1.2: Directional Consistency Feature Extraction. Calculate the cosine similarity between the weight row vectors and the global baseline at the neuron level: And in the hierarchical statistics, the proportion of negative cosine similarity nodes in the abnormal flipping ratio is calculated. and the average cosine similarity of all neurons in that layer. This feature is used to examine hidden deviations in neuronal updates caused by counter-poisoning.

[0035] Step 1.3: Distribution Distortion Feature Extraction and Multilayer Concatenation. Extract the update matrix. sample element mean Finite sample variance With third-order standardized skewness This is to capture the naturally symmetric distribution properties of parameters disrupted by localized poisoning. Subsequently, all features are arranged according to... Combined into local feature vectors of this layer Finally, according to the network topology, The local features of each layer are sequentially spliced ​​together to form the agent in the [missing information]. A complete statistical feature vector with low dimensionality and high information density .

[0036] Phase 3: Anomaly Detection Based on Autoencoder and Dynamic Threshold (corresponding to Steps 3 and 4) After entering the detection phase, the following review process is executed:

[0037] Step 2.1: Autoencoder Feature Reconstruction. Reconstruct the complete statistical feature vectors of each agent. Input to self-encoder In the process, the reconstructed features are restored through the decoder. And calculate the mean square reconstruction error index. Models that have been secretly poisoned will extract features that deviate from their historical benign optimization trajectory, resulting in reconstruction errors far exceeding the normal baseline.

[0038] Step 2.2: Dynamic Threshold Determination. To address the "concept drift" exploration noise caused by the evolution of multi-agent systems in complex environments, the central server adjusts the historical sliding window. The accumulated safety reconfiguration error set is used to calculate the current round's empirical mean. with sample standard deviation Then, an adaptive dynamic interception threshold is set according to the Raida criterion. If the reconstruction error of a certain intelligent agent If it is determined to be malicious poisoning, it will be directly removed from the current aggregation queue; if If the node is not found, it will be retained, and its corresponding model will be added to the set of benign nodes. middle.

[0039] Phase Four: Adaptive Fault-Tolerant Safety Aggregation and Online Fine-Tuning (corresponding to Step 5) Based on the results of the above safety screening, adaptive protection fusion is performed:

[0040] When the system is in a secure environment or partially attacked (number of pre-survivable secure nodes after filtering) The central server uses a benign set that meets the admission criteria. The model update issued by the corresponding node performs mean aggregation to calculate the new global model. Simultaneously, the mean squared reconstruction error of these surviving benign nodes in the current round is extracted and used to update the historical sliding window. The sequence content is used, and their feature vectors are used for local gradient fine-tuning updates of the current autoencoder to ensure that the boundary of the evaluation model can smoothly follow the legitimate evolutionary drift of the policy itself.

[0041] When the system encounters extremely severe collaborative poisoning or is at the Byzantine tolerance limit, filtering is blocked, resulting in (the remaining number of surviving safe nodes). The central server will directly reject any abnormal or suspicious update merging, forcibly activate the fault tolerance protection mechanism, and abandon this round of network-wide parameter updates. It will directly use the secure global model inherited from the previous round. Copy it to the sending end to serve as the latest global policy state. Replaying ensures that the overall task of the multi-agent system will not collapse under extreme conditions.

[0042] The above preferred embodiments should be regarded as illustrative examples of the embodiments of this application. Any technical deductions, substitutions, improvements, etc. that are similar to or based on the embodiments of this application, such as changing the feature splicing order or adjusting the length of the sliding window under different federated learning frameworks, should fall within the protection scope of this invention.

Claims

1. A multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction, characterized in that, Includes the following steps: Step 1, in the In round-fed communication, the central server obtains the data of participants in the federated aggregation. Local models uploaded by individual agents In conjunction with the global benchmark model issued in the previous round, A layer-by-layer comparison is performed, and fine-grained feature extraction at the neuron level is carried out on the local models uploaded by each agent. Based on this, a dimensionality reduction mapping is combined to obtain a low-dimensional statistical feature vector. ; Step 2: Convert the low-dimensional statistical feature vector Input a pre-built feature autoencoder Unsupervised feature reconstruction verification is performed to obtain reconstructed features and calculate the mean squared reconstruction error between the reconstructed features and the original input features. , as a security risk assessment indicator for the intelligent agent; Step 3: Maintain a clean historical error sliding window in system memory. The current node is calculated based on the set of reconstruction errors of all historical safe nodes recorded within the sliding window. Dynamic interception threshold of the wheel ; Step 4: Connect each intelligent agent Reconstruction error With the dynamic interception threshold Perform a comparison, if If so, the local model submitted by the agent is determined to have suffered a highly covert poisoning attack and is therefore removed; if If it is a safe and benign node, its model update is retained, and its reconstruction error is pushed into the historical error sliding window. In, it is also included in the safety set of the current round. ; Step 5: Extract the benign set that is deemed safe. The local model in the model performs adaptive fault-tolerant safety aggregation to generate the updated global safety model for this round. The model evolution is achieved by using this safe subset of features to perform local gradient descent fine-tuning on the feature autoencoder.

2. The multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction according to claim 1, characterized in that, In step 1, each component of the policy network... Fine-grained feature extraction is performed using independent linear layers with individual parameters, including calculating and extracting deep statistical indicator features in the following three dimensions: Amplitude distortion characteristics: computational agent In the Layer parameter update matrix Extract the one-dimensional vector after flattening the model parameter update matrix. Norm and Norm; Directional consistency feature: Calculate the cosine similarity between the row vectors of the neuron weight matrix and the corresponding neurons of the global benchmark at the neuron level. This allows us to extract the proportion of negative cosine similarity nodes. and the average cosine similarity of all neurons in that layer ; Distribution distortion characteristics: Flatten the parameter update matrix into a one-dimensional vector and calculate the sample mean of the updated elements. Then, the finite sample variance of the updated element is calculated. and the third-order normalized skewness characterizing the degree of asymmetry ; All statistical features are concatenated according to the network topology to form a high-information-density, low-dimensional statistical feature vector representing the parameters of the agent's representative model without loss of dimensionality reduction. .

3. The multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction according to claim 1, characterized in that, The feature autoencoder in step 2 includes an encoder. and decoder The two parts have a loss function defined as minimizing the mean squared reconstruction error of the normal features, and the corresponding reconstruction error formula is as follows: 。 4. The multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction according to claim 1, characterized in that, The dynamic interception threshold calculated based on the Raida criterion in step 3. The formula for setting it is: in, The empirical mean of the historical reconstruction errors of all records within the sliding window is calculated. This represents the sample standard deviation of the aforementioned historical reconstruction error. The sensitivity control coefficient represents the degree of stringency in the interception of the control system.

5. The multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction according to claim 1, characterized in that, The adaptive fault-tolerant security aggregation in step 5 specifically includes a circuit breaker mechanism to cope with extremely severe communication poisoning conditions: when the number of benign nodes is determined to be greater than zero, i.e., the security set is not empty ( The latest global model will be obtained by averaging the arithmetic values ​​of the local models of the normal ends contained in this safe subset: ; When all upload nodes in this round are judged as abnormal and removed due to a large-scale coordinated poisoning or extreme Byzantine attack (i.e., the safe set is empty): When this happens, the central server directly blocks and rejects the parameter aggregation and update being performed, and forces the use and distribution of the previous round's global baseline model state across the entire network. .

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, the computer instructions implement the multi-agent federated reinforcement learning poisoning attack defense method based on neuron-level multidimensional feature reconstruction as described in any one of claims 1-5.

Images