Method and system for dynamic adjustment of strategy based on multi-agent reinforcement learning

Abstract

The embodiment of the present disclosure discloses a strategy dynamic adjustment method and system based on multi-agent reinforcement learning. The method comprises: deploying agents on a plurality of network nodes in a zero-trust network access system, respectively, and configuring a local policy model for the agents; constructing a hybrid reward function for each network node; generating an access control strategy based on the local policy model, obtaining a reward value based on the hybrid reward function after execution, and updating the local policy model based on the reward value; uploading the parameters of the updated local policy model to an aggregation server at a preset period; grouping and federatedly aggregating the uploaded parameters in the aggregation server based on the role types of the network nodes, generating a global model corresponding to each role type and issuing the global model, and updating the local policy model of each network node. The method can combine independent decision-making of each network node with cross-node collaborative learning, and realize real-time dynamic self-adaptive adjustment of the access control strategy with changes in the network environment.

Description

Technical Field

[0001] This disclosure relates to the field of cybersecurity technology, and in particular to a method and system for dynamic policy adjustment based on multi-agent reinforcement learning. Background Technology

[0002] Zero Trust Network Access (ZTNA) is a new security architecture that abandons the traditional perimeter-based intranet trust assumptions and requires strict authentication and authorization for every access request. With the deepening of enterprise digital transformation, business systems are becoming increasingly decentralized, and remote work and multi-cloud access are becoming the norm. ZTNA is gradually becoming the mainstream solution for secure enterprise access. Access control policies are a core component of the ZTNA system. They determine under what conditions users and devices can access which application resources, and are the foundation for the effective operation of the entire security system.

[0003] In the ZTNA architecture, when a user initiates an access request, network nodes (such as ZTNA security gateways, firewalls, or API gateways) match and adjudicate the request according to preset access control policies, such as Role-Based Access Control (RBAC) or Attribute-Based Access Control (ABAC) policy sets. These policies are essentially static sets of rules, whose formulation relies on the administrator's prior knowledge of the network environment. Once deployed, they operate with fixed logic and are difficult to adaptively adjust their decision-making logic to real-time dynamic changes in the network environment. Summary of the Invention

[0004] In view of this, embodiments of the present disclosure provide a method and system for dynamic policy adjustment based on multi-agent reinforcement learning, which enables the combination of independent decision-making by each network node and cross-node collaborative learning, thereby realizing real-time dynamic adaptive adjustment of access control policies as the network environment changes.

[0005] In a first aspect, embodiments of this disclosure provide a method for dynamic policy adjustment based on multi-agent reinforcement learning, employing the following technical solution: Agents are deployed on multiple network nodes in a zero-trust network access system, and local policy models are configured for the agents. Construct a hybrid reward function for each network node; The local environment state of each network node is input into the corresponding agent, and an access control policy is generated based on the configured local policy model. After executing the access control policy, a reward value is obtained based on the hybrid reward function, and the local policy model is updated based on the reward value; The parameters of the updated local policy models of each agent are uploaded to the aggregation server at a preset period. Based on the role type of each network node, the uploaded parameters are grouped and federated in the aggregation server to generate a global model corresponding to each role type, and the global model is then distributed to each network node of the corresponding role type. Based on the global model issued, update the local policy model of each network node.

[0006] Optionally, constructing a hybrid reward function for each network node includes: Based on the total number of access requests and the total number of policy violation events for each network node, a global collaborative reward item is constructed for each network node; Configure a competition preference vector for each network node, and construct a local competition reward term for each network node based on the competition preference vector; The global collaboration reward and the local competition reward are weighted and summed to obtain the hybrid reward function.

[0007] Optionally, the construction of a global collaborative reward item for each network node based on the total number of access requests and the total number of policy violation events for each network node includes: The global policy violation rate is obtained based on the total number of access requests and the total number of policy violation events for each network node. Obtain the causal contribution of each network node to the global policy violation rate, and obtain the individual reward for each network node based on the causal contribution. Based on the average policy violation rate of all network nodes of the same role type, obtain the group reward for each role type; Based on the global policy violation rate, obtain the team reward shared by all network nodes; The global collaboration reward item is obtained by weighted summing of the individual reward, the group reward, and the team reward.

[0008] Optionally, obtaining the causal contribution of each network node to the global policy violation rate, and obtaining the individual reward for each network node based on the causal contribution, includes: Perform counterfactual reasoning for each network node to estimate the counterfactual global violation rate of each network node not enforcing the current access control policy; Based on the difference between the global policy violation rate and the counterfactual global violation rate, the causal contribution of each network node to the global policy violation rate is determined, and the individual reward is obtained based on the causal contribution.

[0009] Optionally, configuring the competition preference vector for each network node, and constructing a local competition reward term for each network node based on the competition preference vector, includes: Initialize a competition preference vector for each network node, wherein the competition preference vector contains K components, each component representing the strength of the network node's preference for a class of resources; Collect the actual consumption values ​​of various resources after each network node executes the current access control policy; The local competitive reward item for each network node is obtained by weighting the competitive preference vector of each network node with the actual consumption value of the corresponding resource.

[0010] Optionally, the policy dynamic adjustment method based on multi-agent reinforcement learning further includes: Obtain the similarity of the competitive preference vectors among network nodes within the same role type; The competition preference vector of each network node is adjusted based on the similarity.

[0011] Optionally, the step of grouping and federating the uploaded parameters in the aggregation server based on the role type of each network node to generate a global model corresponding to each role type, and then distributing the global model to each network node of the corresponding role type, includes: Obtain the role type identifier of each network node, and based on the role type identifier, divide the parameters of the local policy model uploaded by each network node into multiple parameter groups, wherein the parameters of network nodes with the same role type are divided into the same parameter group. The local strategy model parameters in each parameter group are federated and aggregated to generate a global model corresponding to each role type. The global model corresponding to each role type is distributed to each network node of the corresponding role type.

[0012] Optionally, the step of federating and aggregating the local strategy model parameters within each parameter group to generate a global model corresponding to each role type includes: The local policy model parameters within each parameter group are weighted and averaged, with the weight of each network node's local policy model parameter being positively correlated with the number of samples in the network node's local area.

[0013] Optionally, updating the local policy model of each network node based on the distributed global model includes: After each network node receives the global model, it tests the fit of the global model on the local validation dataset. When the fit exceeds a preset fit threshold, the local strategy model is updated using the global model.

[0014] Secondly, this disclosure also provides a policy dynamic adjustment system based on multi-agent reinforcement learning, employing the following technical solution: The deployment module is used to deploy agents on multiple network nodes in a zero-trust network access system and configure local policy models for the agents. The building block is used to construct a hybrid reward function for each network node. The generation module is used to input the local environment state of each network node into the corresponding intelligent agent and generate access control policies based on the configured local policy model. The acquisition module is used to obtain a reward value based on the hybrid reward function after executing the access control policy, and update the local policy model based on the reward value; The upload module is used to upload the updated parameters of the local policy model of each agent to the aggregation server at a preset period. The aggregation module is used to perform grouped federated aggregation of uploaded parameters in the aggregation server based on the role type of each network node, generate a global model corresponding to each role type, and distribute the global model to each network node of the corresponding role type. The update module is used to update the local policy model of each network node based on the global model that has been issued.

[0015] Thirdly, this disclosure also provides a computer device, which adopts the following technical solution: The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform any of the above-described policy dynamic adjustment methods based on multi-agent reinforcement learning.

[0016] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to execute any of the above-described multi-agent reinforcement learning-based policy dynamic adjustment methods.

[0017] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0018] The multi-agent reinforcement learning-based policy dynamic adjustment method disclosed in this embodiment deploys agents and configures local policy models on each network node. After executing access control policies, the local policy models are updated based on a hybrid reward function, enabling them to continuously evolve with changes in the network environment. This allows for real-time responses to dynamic changes in network status, eliminating the decision-making lag problem of static policies in dynamic environments and achieving a shift from pre-configuration to adaptive decision-making, thus improving adaptability in dynamic network environments. By periodically uploading the local policy model parameters of each network node to an aggregation server, and then grouping and aggregating them according to the role type of each network node before distributing them, network nodes with different roles can share collaborative experience while maintaining the independence of their respective role characteristics. Each network node updates its local policy model using the distributed global model, thereby achieving cross-node collaborative learning without the need for sharing original data.

[0019] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating the policy dynamic adjustment method based on multi-agent reinforcement learning provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating the hybrid reward function construction method provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating the method for obtaining global collaboration reward items provided in this embodiment of the disclosure; Figure 4 A flowchart illustrating the method for obtaining local competitive reward items provided in an embodiment of this disclosure; Figure 5 A flowchart illustrating the grouped federation aggregation method provided in this embodiment of the disclosure; Figure 6 A schematic diagram of the principle of a policy dynamic adjustment system based on multi-agent reinforcement learning provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. Detailed Implementation

[0022] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0023] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0025] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0026] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0027] Reference Figure 1 This disclosure provides a method for dynamic policy adjustment based on multi-agent reinforcement learning, comprising the following steps: S1: Deploy agents on multiple network nodes in the zero-trust network access system and configure local policy models for the agents; S2: Construct a hybrid reward function for each network node; S3: Input the local environment state of each network node into the corresponding agent, and generate access control policies based on the configured local policy model; S4: After executing the access control policy, obtain the reward value based on the hybrid reward function, and update the local policy model based on the reward value; S5: Upload the updated local policy model parameters of each agent to the aggregation server at a preset cycle; S6: Based on the role type of each network node, the uploaded parameters are grouped and federated in the aggregation server to generate a global model corresponding to each role type, and the global model is distributed to each network node of the corresponding role type. S7: Based on the distributed global model, update the local policy model of each network node.

[0028] The multi-agent reinforcement learning-based dynamic policy adjustment method disclosed herein enables network nodes in a zero-trust network access system to dynamically generate and automatically optimize access control policies based on real-time environmental states, replacing the traditional pre-configured static rule set. Each network node deploys an agent and configures a local policy model. After executing the access control policy, the node obtains a reward value based on a hybrid reward function and updates its local policy model. This allows the local policy model to continuously evolve with changes in the network environment, thereby responding in real-time to dynamic changes in network states. This eliminates the decision-making lag problem of static policies in dynamic environments, achieving a shift from pre-configuration to adaptive decision-making and improving adaptability in dynamic network environments.

[0029] By periodically uploading the local policy model parameters of each network node to the aggregation server, and then grouping and aggregating them according to the role type of each network node before distributing them, network nodes with different roles can share collaborative experience while maintaining the independence of their respective role characteristics. Each network node uses the distributed global model to update its local policy model, thereby achieving cross-node collaborative learning without the need for sharing original data.

[0030] The above-mentioned aggregation update process is combined with the hybrid reward function. The hybrid reward function provides a unified optimization direction for the policy learning of each network node, while group aggregation refines and shares the learning results of each node in this direction. The global model after aggregation and update is then fed back to each node to participate in the next round of policy learning, so that the access control policy of the entire system can continuously evolve synchronously with the changes in the network environment, further enhancing the system's adaptive decision-making ability in dynamic environments.

[0031] In S1, network nodes include at least one of the following in a Zero Trust Network Access System (ZTNA): security gateway, firewall, authentication server, API gateway, and endpoint security agent. The type of network node determines the role of the agent. Agents are deployed on each network node, and a decentralized multi-agent deep deterministic policy gradient algorithm is used to build a collaborative framework among them. During the training phase, agents share a global reward signal to achieve collaborative learning. During the execution phase, each agent makes independent decisions based solely on its local environment state, without relying on a central node or real-time communication between nodes.

[0032] Each agent is configured with a local policy model, which is a deep neural network built on an Actor network. This Actor network takes the local environment state of the network node where the agent resides as input. The local environment state includes local queue depth, current CPU utilization, current memory utilization, the count of locally detected abnormal events, and network latency (RTT). The output of the Actor network is a continuous-valued access control action, which includes a trust threshold and a scan depth parameter. The trust threshold is a continuous score between 0 and 1 used to determine the trust level of the access request, and the scan depth parameter is a continuous value between 0 and 1 used to control the depth level of detection of traffic packets.

[0033] By employing continuous value action outputs, the agent can adaptively fine-tune the access control strategy according to subtle changes in the environmental state, achieving more granular dynamic access control compared to traditional discrete action methods.

[0034] In S2, refer to Figure 2 The flowchart illustrating the hybrid reward function construction method shows that "constructing a hybrid reward function for each network node" includes the following steps: S21: Based on the total number of access requests and the total number of policy violation events for each network node, construct a global collaborative reward item for each network node; S22: Configure the competition preference vector for each network node, and construct the local competition reward term for each network node based on the competition preference vector; S23: The global collaboration reward and the local competition reward are weighted and summed to obtain the hybrid reward function.

[0035] In S21, the global collaboration reward item is used to reflect the degree of reduction in the overall system policy violation rate from the perspective of the network node itself, referring to... Figure 3 The flowchart illustrating the method for obtaining global collaboration rewards shows that "constructing global collaboration rewards for each network node based on the total number of access requests and the total number of policy violation events for each network node" includes the following steps: S211: Obtain the global policy violation rate based on the total number of access requests and the total number of policy violation events for each network node; S212: Obtain the causal contribution of each network node to the global policy violation rate, and obtain the individual reward for each network node based on the causal contribution. S213: Based on the average policy violation rate of all network nodes of the same role type, obtain the group reward for each role type; S214: Obtain team rewards shared by all network nodes based on the global policy violation rate; S215: Weighted sum of individual rewards, group rewards, and team rewards to obtain the global collaboration reward item.

[0036] In S211, the total number of policy violation events of each network node is summed up, and the total number of access requests of each network node is summed up. The summed total number of policy violation events is divided by the summed total number of access requests to obtain the global policy violation rate.

[0037] In S212, counterfactual reasoning is performed on each network node to estimate the counterfactual global violation rate of each network node not implementing the current access control policy; based on the difference between the global policy violation rate and the counterfactual global violation rate, the causal contribution of each network node to the global policy violation rate is determined, and an individual reward is obtained based on the causal contribution.

[0038] Regarding counterfactual reasoning, for any given network node, if it currently implements a certain access control policy (e.g., the gateway node's trust threshold is increased from 0.5 to 0.7), the system needs to calculate the current actual global policy violation rate. To evaluate the contribution of this node's decision to the overall system security level, a counterfactual question needs to be answered: If this node had not implemented the policy at the current moment, but instead maintained the default policy or the policy from the previous moment, while the policies of other network nodes remained unchanged, what would the system's global policy violation rate be?

[0039] This counterfactual estimation is achieved through a pre-trained environment model (i.e., the state-value function in the Critic network). The action of the node in the current system state is replaced with its default policy action to generate a counterfactual system state. This counterfactual system state is then input into the Critic network, outputting a predicted value for the global policy violation rate. This process does not alter historical data and has no impact on the actual system; all calculations are performed offline.

[0040] The difference between the counterfactual global policy violation rate and the actual global policy violation rate represents the causal contribution of the node's current decision to the global policy violation rate. A positive difference indicates that the node's current decision lowers the estimated global policy violation rate after implementing the current policy, meaning the node's decision helps reduce the global policy violation rate, resulting in a positive contribution. A negative difference indicates that the node's current decision actually increases the global policy violation rate, resulting in a negative contribution. A difference close to zero indicates that the node's decision has no significant impact on the global policy violation rate. The larger the absolute value of the difference, the greater the impact of the node's current decision on the global policy violation rate, indicating a higher causal contribution.

[0041] The negative of a network node's causal contribution is taken as the individual reward. When the causal contribution is positive (the node's decision helps reduce the violation rate), the individual reward is negative, indicating that the node receives a positive incentive; when the causal contribution is negative (the node's decision leads to an increase in the violation rate), the individual reward is positive, indicating that the node receives a negative penalty. By minimizing individual rewards, each network node is incentivized to optimize its own strategy to reduce the overall system violation rate.

[0042] The core objective of individual reward design is to accurately attribute the macro-system indicator of global policy violation rate to each independent decision of each network node, thereby enabling each node to receive immediate and clear behavioral feedback.

[0043] This method comprises two key components: counterfactual reasoning and credit allocation. At the credit allocation level, it employs a comparison of actual and counterfactual factors, rather than simply using the global policy violation rate as the reward for each node. This is because the global policy violation rate is the result of the combined decisions of all nodes. Directly allocating it to all nodes would obscure the actual contributions of each node, leading to ambiguous credit allocation and allowing high-performing and low-performing nodes to receive the same reward, thus weakening the incentive for nodes to optimize their own strategies. By comparing the estimated violation rate when not implementing the current strategy with the actual violation rate after implementing the current strategy, each node's reward accurately reflects the marginal contribution of its own decision. Positive contributions result in positive incentives, while negative contributions result in penalties. This ensures that each node is responsible for its own decisions, effectively avoiding the free-rider problem in multi-node systems—where some nodes share global rewards without contributing to the global goal. When nodes realize that they only receive positive incentives when they actually contribute, they will naturally continue to optimize their strategies to reduce the overall system violation rate.

[0044] In S213, all network nodes are grouped by role type, and the average violation rate of all network nodes within each role type is calculated. The negative value of the average violation rate is used as the group reward for each network node within the corresponding role type. This method allows network nodes of the same role type to form a small group collaborative supervision relationship, prompting similar nodes to learn from each other and optimize together. For example, all gateway nodes share the same group reward signal. If a gateway node's decision causes the average violation rate within the group to increase, the rewards of other gateway nodes will also be affected, thereby driving the natural formation of a mutual supervision and collaborative improvement relationship among gateway nodes.

[0045] In S214, the negative value of the global policy violation rate is used as the team reward, so that all network nodes share the same macro security goal. This ensures that each node does not deviate from the fundamental direction of overall system security when pursuing its own local optimization, and prevents nodes from damaging the overall system security due to excessive focus on individual or group indicators.

[0046] In S215, the weighting coefficients used when weighting and summing individual rewards, group rewards, and team rewards should have the highest weight for individual rewards (e.g., 0.5), followed by group rewards (e.g., 0.3), and the lowest weight for team rewards (e.g., 0.2). This ensures that network nodes prioritize the immediate impact of their own decisions (individual rewards), then the collaborative performance of similar nodes (group rewards), while maintaining responsiveness to the global objective (team rewards). The sum of the three weighting coefficients is 1. Furthermore, the weighting coefficients can be flexibly adjusted according to the actual deployment scenario. When system security requirements are high, the team reward weight can be appropriately increased to make each network node focus more on the overall security level; when the system is in a stable operating state, the individual reward weight can be appropriately increased to make each network node focus more on the optimization efficiency of its own decisions.

[0047] In S22, the local competition reward item is used to reflect the resource consumption level of the network node itself, referring to... Figure 4 The flowchart illustrating the method for obtaining local competitive reward items shows the following steps: "Configuring the competitive preference vector for each network node and constructing the local competitive reward item for each network node based on the competitive preference vector": S221: Initialize a competition preference vector for each network node, where the competition preference vector contains K components, each component representing the strength of the network node's preference for a class of resources; S222: Collect the actual consumption values ​​of various resources after each network node executes the current access control policy; S223: Weight the competitive preference vector of each network node with the actual consumption value of the corresponding resource to obtain the local competitive reward of each network node.

[0048] In S221, for each network node in the zero-trust network access system, during its agent initialization phase, a competition preference vector is configured for the network node. This vector is a K-dimensional vector, where K represents the total number of resource types of interest in the zero-trust network access system. Each component of the vector corresponds to a type of resource, and its value is used to characterize the node's sensitivity or preference intensity for this type of resource. The larger the value, the more the node cares about the consumption of this type of resource and will be more inclined to reduce the consumption of this type of resource when making decisions. The smaller the value, the less sensitive the node is to the consumption of this type of resource and allows for relatively more consumption of this type of resource in decision-making.

[0049] The initialization of the competition preference vector is differentiated based on the role type of the network node. For example, gateway nodes, which undertake deep traffic inspection tasks and consume significant CPU computing resources, can be initialized with a high preference for CPU resources (close to 1) and a low preference for network bandwidth resources (close to 0). Authentication servers, which undertake user authentication tasks and are sensitive to response latency, can be initialized with a high preference for network bandwidth resources (close to 1) and a low preference for CPU computing resources (close to 0). This differentiated initialization lays the foundation for the subsequent differentiation of competition strategies among nodes.

[0050] In S222, during each iteration, the agent of each network node generates and executes an access control policy based on the current local environment state, and immediately collects the actual consumption of various resources after the policy is executed. The collected resource types correspond one-to-one with the K components in the competition preference vector, including but not limited to CPU utilization, network bandwidth utilization, and memory utilization. The collected resource consumption values ​​need to be normalized, mapping the original consumption values ​​to the range of 0 to 1 to eliminate differences in the units and numerical ranges of different resource types, allowing for comparison and weighted calculation of resource consumption values ​​on the same scale. The collection action is triggered after each decision cycle, ensuring a clear temporal correspondence between resource consumption data and the current policy decision.

[0051] In S223, each component of the competitive preference vector of the same network node is multiplied by the actual consumption value of the corresponding type of resource, and the multiplications are summed to obtain the weighted sum of the resource consumption of the network node. The negative value of this weighted sum is then taken as the local competitive reward term of the network node. This calculation method ensures that the higher the preference intensity of a network node for a certain type of resource (i.e., the larger the value of the corresponding component in the competitive preference vector), the lower the value of the local competitive reward term of the network node when the actual consumption value of that type of resource is high (the greater the penalty), thus driving the node to reduce actions that consume a high amount of that type of resource in subsequent decisions. Conversely, the lower the preference intensity of a network node for a certain type of resource (i.e., the smaller the value of the corresponding component in the competitive preference vector), the less affected the local competitive reward term of the network node is when the actual consumption value of that type of resource is high, and the network node is allowed to consume relatively more on that type of resource.

[0052] The formula for calculating the local competitive reward is as follows: In the formula, Indicates the first Local competitive reward items for individual network nodes; Indicates the total number of resource types; Indicates the first The network node pairs with the first The intensity of preference for resource types; Indicates the first The actual consumption value of the resource class.

[0053] The core of the above calculation method lies in differentiated competition. Network nodes with different roles have varying sensitivities to various resources. The gateway node's core function is deep packet inspection, making it highly sensitive to CPU consumption. The authentication server is responsible for authentication, making it highly sensitive to network latency and bandwidth consumption. The terminal agent is responsible for behavior monitoring, making it highly sensitive to memory consumption. If a uniform resource consumption penalty standard is applied to all network nodes, nodes with different roles will be unfairly penalized when performing their respective functions, leading to a decline in overall system performance. By introducing a competition preference vector, each node can carefully manage its resources in its sensitive dimensions and use them moderately in non-sensitive dimensions, thereby achieving differentiated competition. Each network node competes on its preferred resource dimensions, avoiding vicious competition among all nodes on the same resource dimension.

[0054] Furthermore, the similarity of the competitive preference vectors among network nodes within the same role type is obtained, and the competitive preference vectors of each network node are adjusted based on the similarity to reduce the similarity between the competitive preference vectors of different network nodes within the same role type.

[0055] Specifically, when the similarity between the competitive preference vector of any network node and the competitive preference vectors of other network nodes within the same role type exceeds a preset similarity threshold, a penalty adjustment is applied to the competitive preference vector of that network node. This penalty adjustment causes the competitive preference vector of that network node to be updated in a direction away from the competitive preference vectors of other network nodes within the same role type. Specifically, when a behavior does not meet expectations, a negative reward signal or update gradient is applied to reverse the adjustment, causing the model to tend to reduce that behavior in the future. In other words, a penalty term is added during training, and the value of the penalty term is positively correlated with similarity, thereby reducing the similarity between the competitive preference vector of that network node and the competitive preference vectors of other network nodes within the same role type.

[0056] In S23, the expression for the hybrid reward function is as follows: In the formula, Indicates the first A hybrid reward function for each network node; Indicates the first Global collaboration rewards for each network node; This represents the weighting coefficient.

[0057] Furthermore, the weighting coefficients of the global collaboration reward item and the local competition reward item are dynamically adjusted based on the current global strategy violation rate. When the global strategy violation rate is greater than a preset violation rate threshold, the weighting coefficient of the global collaboration reward item is increased; when the global strategy violation rate is less than the preset violation rate threshold, the weighting coefficient of the local competition reward item is increased. Specifically, Dynamically adjust as follows: In the formula, Represents the Sigmoid function; Indicates the overall policy violation rate; This indicates the preset violation rate threshold. When... Greater than hour, Increase and strengthen the weight of global collaboration reward items; when Less than hour, Decrease Correspondingly, the weight of local competitive reward items is strengthened.

[0058] In S3, during any iteration, the agents of each network node collect the current local environment state through the local monitoring module. This collected state is then input into a local policy model built on an Actor network for forward propagation calculation. The Actor network outputs a continuous action containing a trust threshold and a scan depth parameter, which serves as the access control policy for the current iteration. The agent then determines the trust level of the access request according to the trust threshold. If the trust score of the access request is lower than the threshold, the request is rejected; otherwise, it is allowed. Simultaneously, security checks at the corresponding depth level are performed on the allowed traffic packets according to the scan depth parameter. After all actions are completed, the agent returns the access result to the processed request. This process is entirely completed locally, without needing to communicate with other nodes or wait for instructions from a central node.

[0059] In S4, after executing the access control policy, data such as the global policy violation rate and counterfactual global violation rate of the current iteration are collected. The reward value is calculated according to the hybrid reward function constructed in step S2. The state, action, reward value, and next state are stored as experience in the replay pool. The local policy model parameters are updated through sampling and the DDPG algorithm. This design fully internalizes the concepts of competition and cooperation into the reward function. The cooperative goal is to transmit a common direction of reducing the overall system violation rate through global cooperative reward items. The competitive goal is to drive each node to carefully allocate its preferred resources through local competitive reward items. Each node can independently update its policy, spontaneously forming a balance during execution without requiring real-time communication between nodes and without increasing the overhead of zero-trust network access systems.

[0060] In S5, after completing the local policy model update in step S4, each network node uploads its own local policy model parameters to the aggregation server at preset intervals, while retaining local data and not transmitting it externally.

[0061] In S6, refer to Figure 5 The flowchart illustrating the grouped federated aggregation method shows that "based on the role type of each network node, the uploaded parameters are grouped and federated in the aggregation server to generate a global model corresponding to each role type, and the global model is then distributed to each network node of the corresponding role type." This includes the following steps: S61: Obtain the role type identifier of each network node, and based on the role type identifier, divide the parameters of the local policy model uploaded by each network node into multiple parameter groups, wherein the parameters of network nodes with the same role type are divided into the same parameter group. S62: Perform federated aggregation of the local strategy model parameters in each parameter group to generate a global model corresponding to each role type; S63: Distribute the global model corresponding to each role type to each network node of the corresponding role type.

[0062] In step S61, the aggregation server receives the local policy model parameters and role type identifiers of each network node uploaded by the network nodes. These role type identifiers were pre-configured and stored in the aggregation server during the deployment of agents on each network node in step S1. The aggregation server groups the uploaded parameters according to the role type identifiers, assigning parameters from network nodes with the same role type identifier to the same parameter group. This results in multiple parameter groups that correspond one-to-one with each role type, including but not limited to gateway parameter groups, authentication parameter groups, terminal parameter groups, and API parameter groups. Network nodes within the same parameter group belong to the same role type, while different parameter groups correspond to different role types.

[0063] In S62, a weighted average is calculated for the local policy model parameters within each parameter group. The weight of each network node's local policy model parameter is positively correlated with the number of local samples at that network node. Specifically, the aggregation server obtains the number of local training samples for each network node within the parameter group. The proportion of each network node's local training samples to the total number of samples in that parameter group is used as the aggregation weight for each network node. A weighted average is then calculated for the local policy model parameters of each network node to obtain the global model corresponding to that role type.

[0064] For network nodes with a larger number of local training samples, their local policy models are exposed to a richer distribution of data during training, resulting in higher confidence levels for the model parameters. Therefore, they are given higher weights during aggregation, making the generated global model more focused on experienced nodes and improving the generalization ability of the global model.

[0065] In S63, after the aggregation server completes the federated aggregation of each parameter group, it distributes the generated global models corresponding to each role type to the network nodes of the corresponding role type. Each network node only receives the global model corresponding to its own role type and does not receive global models of other role types, so as to maintain the independence and specialization of the strategy models of each role type.

[0066] In S7, after each network node receives the global model, it tests the fitness of the global model on the local validation dataset. When the fitness exceeds the preset fitness threshold, the local strategy model is updated using the global model.

[0067] The local validation dataset is a subset of historical data stored locally that was not used in training. Network nodes evaluate the global model on this local validation dataset, calculating its accuracy or F1 score as its fitness. When the fitness exceeds a preset threshold, it indicates a high degree of match between the global model and the node's local data distribution, and it will not interfere with the competitive strategies already learned by the local policy model. In this case, the global model is used to update the local policy model. When the fitness does not exceed the preset threshold, it indicates a significant difference between the global model and the node's local data distribution. Forcing its adoption would disrupt the dynamic environment already adapted to by the local policy model. In this case, the global model is rejected, and the current local policy model remains unchanged, awaiting the next aggregation cycle.

[0068] In addition, the aggregation server in this scheme does not participate in any real-time decision-making. It is only used to group and aggregate the local policy model parameters uploaded by each network node. Each network node can independently complete policy reasoning and execution without relying on the aggregation server or real-time communication between nodes during the execution phase. This architecture makes the decision-making process of the entire zero-trust network access system essentially decentralized.

[0069] Reference Figure 6 This disclosure provides a policy dynamic adjustment system based on multi-agent reinforcement learning, comprising: Deployment module 101 is used to deploy agents on multiple network nodes in a zero-trust network access system and configure local policy models for the agents. Module 102 is used to construct a hybrid reward function for each network node. The generation module 103 is used to input the local environment state of each network node into the corresponding intelligent agent and generate an access control policy based on the configured local policy model. The acquisition module 104 is used to obtain a reward value based on a hybrid reward function after executing the access control policy, and update the local policy model based on the reward value; Upload module 105 is used to upload the updated parameters of the local policy model of each agent to the aggregation server at a preset period. The aggregation module 106 is used to perform grouped federated aggregation of uploaded parameters in the aggregation server based on the role type of each network node, generate a global model corresponding to each role type, and distribute the global model to each network node of the corresponding role type. Update module 107 is used to update the local policy model of each network node based on the distributed global model.

[0070] The various variations and specific examples of the policy dynamic adjustment method based on multi-agent reinforcement learning provided above are also applicable to the policy dynamic adjustment system based on multi-agent reinforcement learning provided in this disclosure. Through the foregoing detailed description of the policy dynamic adjustment method based on multi-agent reinforcement learning, those skilled in the art can clearly understand the implementation method of the policy dynamic adjustment system based on multi-agent reinforcement learning. For the sake of brevity, it will not be described in detail here.

[0071] A computer device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0072] The processor may be a central processing unit (CPU) or other processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the computer device to perform all or part of the steps of the multi-agent reinforcement learning-based policy dynamic adjustment method described in the foregoing embodiments of this disclosure.

[0073] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0074] like Figure 7 This is a schematic diagram of a computer device provided for an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the computer device in the embodiments of the present disclosure. Figure 7 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0075] like Figure 7As shown, a computer device may include a processor (such as a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from storage devices into random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0076] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow the computer device to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 7 A computer apparatus with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.

[0077] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the policy dynamic adjustment method based on multi-agent reinforcement learning according to embodiments of this disclosure are performed.

[0078] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0079] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the policy dynamic adjustment method based on multi-agent reinforcement learning described in the foregoing embodiments of the present disclosure are performed.

[0080] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0081] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0082] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0083] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0084] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0085] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0086] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0087] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0088] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for dynamic policy adjustment based on multi-agent reinforcement learning, characterized in that, include: Agents are deployed on multiple network nodes in a zero-trust network access system, and local policy models are configured for the agents. Construct a hybrid reward function for each network node; The local environment state of each network node is input into the corresponding agent, and an access control policy is generated based on the configured local policy model. After executing the access control policy, a reward value is obtained based on the hybrid reward function, and the local policy model is updated based on the reward value; The parameters of the updated local policy models of each agent are uploaded to the aggregation server at a preset period. Based on the role type of each network node, the uploaded parameters are grouped and federated in the aggregation server to generate a global model corresponding to each role type, and the global model is then distributed to each network node of the corresponding role type. Based on the global model issued, update the local policy model of each network node.

2. The policy dynamic adjustment method based on multi-agent reinforcement learning according to claim 1, characterized in that, The construction of a hybrid reward function for each network node includes: Based on the total number of access requests and the total number of policy violation events for each network node, a global collaborative reward item is constructed for each network node; Configure a competition preference vector for each network node, and construct a local competition reward term for each network node based on the competition preference vector; The global collaboration reward and the local competition reward are weighted and summed to obtain the hybrid reward function.

3. The policy dynamic adjustment method based on multi-agent reinforcement learning according to claim 2, characterized in that, The global collaborative reward item for each network node is constructed based on the total number of access requests and the total number of policy violation events for each network node, including: The global policy violation rate is obtained based on the total number of access requests and the total number of policy violation events for each network node. Obtain the causal contribution of each network node to the global policy violation rate, and obtain the individual reward for each network node based on the causal contribution. Based on the average policy violation rate of all network nodes of the same role type, obtain the group reward for each role type; Based on the global policy violation rate, obtain the team reward shared by all network nodes; The global collaboration reward item is obtained by weighted summing of the individual reward, the group reward, and the team reward.

4. The policy dynamic adjustment method based on multi-agent reinforcement learning according to claim 3, characterized in that, The step of obtaining the causal contribution of each network node to the global policy violation rate, and obtaining the individual reward for each network node based on the causal contribution, includes: Perform counterfactual reasoning for each network node to estimate the counterfactual global violation rate of each network node not enforcing the current access control policy; Based on the difference between the global policy violation rate and the counterfactual global violation rate, the causal contribution of each network node to the global policy violation rate is determined, and the individual reward is obtained based on the causal contribution.

5. The policy dynamic adjustment method based on multi-agent reinforcement learning according to claim 2, characterized in that, The configuration of the competition preference vector for each network node, and the construction of a local competition reward term for each network node based on the competition preference vector, includes: Initialize a competition preference vector for each network node, wherein the competition preference vector contains K components, each component representing the strength of the network node's preference for a class of resources; Collect the actual consumption values ​​of various resources after each network node executes the current access control policy; The local competitive reward item for each network node is obtained by weighting the competitive preference vector of each network node with the actual consumption value of the corresponding resource.

6. The policy dynamic adjustment method based on multi-agent reinforcement learning according to claim 5, characterized in that, Also includes: Obtain the similarity of the competitive preference vectors among network nodes within the same role type; The competition preference vector of each network node is adjusted based on the similarity.

7. The policy dynamic adjustment method based on multi-agent reinforcement learning according to claim 1, characterized in that, Based on the role type of each network node, the uploaded parameters are grouped and federated in the aggregation server to generate a global model corresponding to each role type, and the global model is then distributed to each network node of the corresponding role type, including: Obtain the role type identifier of each network node, and based on the role type identifier, divide the parameters of the local policy model uploaded by each network node into multiple parameter groups, wherein the parameters of network nodes with the same role type are divided into the same parameter group. The local strategy model parameters in each parameter group are federated and aggregated to generate a global model corresponding to each role type. The global model corresponding to each role type is distributed to each network node of the corresponding role type.

8. The method for dynamic policy adjustment based on multi-agent reinforcement learning according to claim 7, characterized in that, The process of federating and aggregating the local strategy model parameters within each parameter group to generate a global model corresponding to each role type includes: The local policy model parameters within each parameter group are weighted and averaged, with the weight of each network node's local policy model parameter being positively correlated with the number of samples in the network node's local area.

9. The method for dynamic policy adjustment based on multi-agent reinforcement learning according to claim 1, characterized in that, The process of updating the local policy model of each network node based on the distributed global model includes: After each network node receives the global model, it tests the fit of the global model on the local validation dataset. When the fit exceeds a preset fit threshold, the local strategy model is updated using the global model.

10. A policy dynamic adjustment system based on multi-agent reinforcement learning, characterized in that, include: The deployment module is used to deploy agents on multiple network nodes in a zero-trust network access system and configure local policy models for the agents. The building block is used to construct a hybrid reward function for each network node. The generation module is used to input the local environment state of each network node into the corresponding intelligent agent and generate access control policies based on the configured local policy model. The acquisition module is used to obtain a reward value based on the hybrid reward function after executing the access control policy, and update the local policy model based on the reward value; The upload module is used to upload the updated parameters of the local policy model of each agent to the aggregation server at a preset period. The aggregation module is used to perform grouped federated aggregation of uploaded parameters in the aggregation server based on the role type of each network node, generate a global model corresponding to each role type, and distribute the global model to each network node of the corresponding role type. The update module is used to update the local policy model of each network node based on the global model that has been issued.

Images