Server security configuration method and device, equipment, storage medium and product

Abstract

The invention relates to a server security configuration method and device, equipment, a storage medium and a product, and relates to the technical field of security. The method comprises the steps of obtaining an agent state of any target agent in an agent system and a global state of the agent system, wherein the global state is integration of the agent states of all agents contained in the agent system; through the target Actor network of the target agent, based on the agent state of the target agent and the global state, performing configuration action generation to obtain a configuration action of the target agent; performing value evaluation on the configuration action of the target agent based on the global state through a target Critic network of the target agent to obtain a value evaluation result of the configuration action of the target agent; and performing parameter updating on the target Actor network based on a value evaluation result until an updating completion condition is reached, so as to perform server security configuration on the target agent based on the updated target Actor network.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of security, in particular to a server security configuration method, device, equipment, storage medium and product. BACKGROUND

[0002] In the related art, a single intelligent agent is usually used for server security configuration systems, which usually rely on a centralized decision unit to handle security events, respond to known threats through a static rule learning mechanism, and the learning ability of the intelligent agent is usually limited to a small amount of feedback adjustment or updating rules through manual intervention, making the system have poor self-adaptation ability, insufficient adaptability and response speed to new or complex attacks, and increasing the risk of security vulnerabilities. SUMMARY

[0003] Embodiments of the present application provide a server security configuration method, device, equipment, storage medium and product, which can overcome the defects of traditional centralized systems relying on static rules and poor self-adaptation ability. The technical solution is as follows.

[0004] On the one hand, a server security configuration method is provided, which comprises: obtaining an agent state of a target intelligent agent in an intelligent agent system and a global state of the intelligent agent system, the global state being an integration of agent states of each intelligent agent contained in the intelligent agent system, the target intelligent agent being any one of the intelligent agents; generating a configuration action of the target intelligent agent based on the agent state of the target intelligent agent and the global state through a target actor network of the target intelligent agent; performing value evaluation on the configuration action of the target intelligent agent based on the global state through a target critic network of the target intelligent agent to obtain a value evaluation result of the configuration action of the target intelligent agent; updating parameters of the target actor network based on the value evaluation result until an update completion condition is reached to configure server security of the target intelligent agent based on the updated target actor network.

[0005] On the other hand, a server security configuration device is provided, which comprises: a state acquisition module configured to acquire an agent state of a target intelligent agent in an intelligent agent system and a global state of the intelligent agent system, the global state being an integration of agent states of each intelligent agent contained in the intelligent agent system, the target intelligent agent being any one of the intelligent agents; The action generation module is configured to generate a configuration action of the target agent by a target actor network of the target agent based on an agent state of the target agent and the global state. The value evaluation module is configured to evaluate a value of the configuration action of the target agent by a target critic network of the target agent based on the global state, and obtain a value evaluation result of the configuration action of the target agent. The first parameter updating module is configured to update parameters of the target actor network based on the value evaluation result until an update completion condition is reached, so as to perform server security configuration on the target agent based on the updated target actor network.

[0006] In a possible implementation, each agent is configured with a corresponding reward function. The state acquisition module is configured to acquire a new global state after the target agent performs the configuration action. The apparatus further includes: The reward value calculation module is configured to calculate a reward value of the configuration action based on the new global state and the reward function corresponding to the target agent. The target value calculation module is configured to calculate a target value of the configuration action based on the reward value of the configuration action and the new global state. The second parameter updating module is configured to update network parameters of the target critic network with minimizing a difference between the value evaluation result of the configuration action and the target value as an optimization target.

[0007] In a possible implementation, the second parameter updating module includes: The function value calculation submodule is configured to calculate a function value of a meta-learning loss function based on the value evaluation result and the target value. The parameter updating submodule is configured to update the critic network of the target agent based on the function value of the meta-learning loss function and a preset learning rate.

[0008] In a possible implementation, the value evaluation module is configured to evaluate a value of the configuration action of the target agent based on the global state by the updated target critic network after updating the network parameters of the critic network.

[0009] In a possible implementation, the agent system includes a monitoring agent, an analysis agent, and a response agent. The monitoring agent is configured to generate a monitoring action according to a monitoring agent state, the monitoring agent state including network traffic features, system activity logs, and network connection states; The analysis agent is configured to generate an analysis action according to an analysis agent state, the analysis agent state including current threat data, historical threat data, and a network security posture; The response agent is configured to generate a response action according to a response agent state, the response agent state including current security alerts, a response policy library, and system defense states.

[0010] In a possible implementation, the configuration action of the monitoring agent includes data collection strategy adjustment, monitoring range selection, and alarm sensitivity setting; The configuration action of the analysis agent includes threat assessment depth adjustment, analysis tool selection, and risk assessment update; The configuration action of the response agent includes configuration change implementation, stress measure activation, and response policy optimization.

[0011] In another aspect, a computer device is provided, which includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-mentioned server security configuration method.

[0012] In another aspect, a computer readable storage medium is provided, which stores at least one computer program, the computer program being loaded and executed by a processor to implement the above-mentioned server security configuration method.

[0013] In another aspect, a computer program product is provided, which includes a computer program stored on a non-transitory computer readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the above-mentioned server security configuration method provided in various optional implementation manners.

[0014] The technical solution provided in the present application can include the following beneficial effects: The server security configuration method provided by the embodiment of the application comprises the following steps: obtaining an agent state of a target agent in an agent system and a global state integrated by all agent states; generating a configuration action based on the agent state and the global state of the target agent through an Actor network of the target agent; performing value evaluation on the configuration action based on the global state through a Critic network of the target agent; and finally updating Actor network parameters according to the evaluation result, and realizing autonomous evolution of the agent through iterative optimization. By introducing the global state perspective and the distributed learning architecture, each agent can fully consider the overall system condition when making decisions, effectively overcoming the defects of the traditional centralized system, such as dependence on static rules and poor self-adaptability, improving the identification and response capability of the system to new threats and complex attacks, enhancing the initiative and adaptability of the overall security protection, and reducing the security risks caused by rule update lag.

[0015] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application together with the description.

[0017] Figure 1 A flow chart of a server security configuration method provided by an exemplary embodiment of the application is shown; Figure 2 A schematic diagram of an agent system provided by an exemplary embodiment of the application is shown; Figure 3 A schematic diagram of a network parameter updating process provided by an exemplary embodiment of the application is shown; Figure 4 A block diagram of a server security configuration device provided by an exemplary embodiment of the application is shown; Figure 5 A structural block diagram of a computer device shown by an exemplary embodiment of the application is shown; Figure 6 A structural block diagram of a computer device shown by an exemplary embodiment of the application is shown. DETAILED DESCRIPTION

[0018] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the application as detailed in the appended claims.

[0019] This application provides a scheme to improve the overall perception and response capability of a system to cybersecurity threats in an intelligent agent system containing multiple intelligent agents by utilizing the collaborative effect between the agents, so as to achieve global optimization and rapid response. Figure 1 This application illustrates a flowchart of a server security configuration method provided in an exemplary embodiment. This method can be executed by a computer device, which can be implemented as a server or a terminal device, such as… Figure 1 As shown, the server security configuration method may include the following steps.

[0020] Step 110: Obtain the agent state of the target agent in the agent system and the global state of the agent system. The global state is the integration of the agent states of each agent in the agent system. The target agent is any one of the agents.

[0021] In this embodiment, the intelligent agent system defines a shared environment and provides a framework for communication, coordination, and management of interactions between intelligent agents. This system can contain multiple intelligent agents, and based on different actual needs, it can be configured with different numbers and types of intelligent agents. Each intelligent agent has different functions, and there are collaborative relationships between multiple intelligent agents. The number of each type of intelligent agent in the same system can be one or more; this embodiment does not limit this. By precisely controlling data flow and action decisions to achieve collaborative work, the system's response speed and accuracy to network security threats can be improved, making the implementation of security strategies more targeted and timely.

[0022] The agent state of an agent is a description of the agent's relevant attributes. It can encapsulate the agent's internal attributes, external perception, and internal task state, etc. Among them, internal attributes may include, but are not limited to, position coordinates, energy level, and load status; external perception may include, but are not limited to, sensor readings and communication message queues; and internal task state may include, but is not limited to, current task objective, behavior policy parameters, and task execution level. Based on different agent functions, there may be different agent state parameters. The types of agent state parameters corresponding to agent states of different functions may be different, and this application does not impose any restrictions on this.

[0023] This application embodiment uses an intelligent agent system containing three types of intelligent agents as an example for illustration. In one possible application scenario, the intelligent agent system includes a monitoring intelligent agent, an analysis intelligent agent, and a response intelligent agent. Figure 2 A schematic diagram of an intelligent agent system provided in an exemplary embodiment of this application is shown, such as... Figure 2As shown, the intelligent agent system includes a monitoring intelligent agent 210, an analysis intelligent agent 220, and a response intelligent agent 230; The monitoring agent is used to generate monitoring actions based on the monitoring agent's state, which includes network traffic characteristics, system activity logs, and network connection status. The analytical agent is used to generate analytical actions based on the state of the analytical agent, which includes current threat data, historical threat data, and cybersecurity posture. The response agent is used to generate response actions based on the response agent's state, which includes the current security alert, response policy library, and system defense status.

[0024] The monitoring agent can collect network traffic and system status data to monitor potential threats. Its state can include network traffic characteristics, system activity logs, and network connection status, etc. The agent's configuration actions (i.e., monitoring actions) can include adjusting data collection strategies, selecting monitoring scope, and setting alarm sensitivity, etc. The state space of the monitoring agent is defined schematically. ,in Indicates network traffic characteristics, This represents the system activity log. Indicates network connection status; defines the action space of the monitoring agent. , This indicates an adjustment to the data acquisition strategy, including adjustments to the frequency and depth of data acquisition. This indicates the selection of the monitoring scope, including choosing the network areas and specific devices to monitor for potential threats in a targeted manner; This indicates the alarm sensitivity setting, which balances the risks of false alarms and missed alarms.

[0025] The analytical agent can receive data from the monitoring agent and perform in-depth analysis, including the type and severity of threats. The agent's state can include current threat data, historical threat data, and network security posture. The agent's configuration actions (i.e., analysis actions) can include adjusting the depth of threat assessment, selecting analysis tools, and updating risk assessments. The state space of the analytical agent is defined schematically. ,in This indicates current threat data. This represents historical threat data. Representing network security posture; defining the action space of the analytical agent. , This indicates adjusting the depth of the threat assessment, such as choosing surface analysis or delving deeper into the sources and potential impacts of threats; This indicates the selection of analysis tools; the most suitable analysis tool or technology is selected based on the type of threat currently detected. This indicates a risk assessment update, where the risk assessment of the system or asset is updated based on new analysis results.

[0026] The response agent can receive suggestions from the analysis agent, execute security policies, and adjust configurations. The response agent's state can include current security alerts, response policy repositories, and system defense status, etc. The response agent's configuration actions (i.e., analysis actions) can include implementing configuration changes, activating reactive measures, and optimizing response policies, etc. An illustrative definition of the response agent's state space is provided. ,in This indicates a current security alert. This indicates the response strategy library. Represents the system's defense state; defines the action space of the responding agent. , This indicates that configuration changes have been implemented, including updating firewall rules and adjusting access control lists. This indicates that emergency measures have been activated, including activating emergency measures when a serious threat is detected, such as cutting off network connectivity in the affected area; This indicates response strategy optimization. Based on the analysis of the agent's suggestions and historical response effects, the response measures are optimized and adjusted, including increasing or decreasing the defense strength, switching strategies, and updating strategy parameters.

[0027] By designing specific state and action spaces for different agents, the system fully considers the roles and functional requirements of different agents in the system. The customized design enables agents to perform their duties more effectively, improving the efficiency and accuracy of the entire security configuration system.

[0028] The global state integrates the local information of all agents in the system; that is, it is the set of agent states of each agent in the agent system. Illustratively, the global state of the agent system is defined as follows: ,in Let be the state space of the i-th agent. This global state is the process of merging the local observations of all agents into a global view, ensuring that decisions take into account the overall system state.

[0029] Step 120: Through the target agent Actor network, configuration actions are generated based on the target agent's agent state and global state to obtain the target agent's configuration actions.

[0030] In this embodiment, an Actor network can be constructed for each agent. The Actor networks for different agents can have different network structures. The Actor network is used to generate configuration actions corresponding to the agents. Taking an agent system that includes a monitoring agent, an analysis agent, and a response agent as an example: For monitoring agents, their Actor networks The input layer receives input data such as network traffic characteristics, system activity logs, and network connection status. Represents network parameters, with an input layer size of The Actor network monitoring the agent can have four hidden layers. The first two layers each contain 256 neurons with the ReLU activation function, and the last two layers each contain 128 neurons with the ReLU activation function. The output layer uses the Softmax activation function, and its size depends on the number of monitored action categories. .

[0031] For analyzing intelligent agents, their Actor networks The input layer receives input data such as threat data and historical threat records; its input layer size is [size missing]. The hidden layers of the Actor network analyzing the agent are the same as those of the Actor network monitoring the agent; the output layer uses the Softmax activation function, and its size depends on the number of monitored action categories. .

[0032] For a responding agent, its Actor network The input layer receives alarms and system defense status from the analytical agent, and its input layer size is [size missing]. The hidden layers of the Actor network responding to the agent can consist of four layers: the first two layers each contain 64 neurons with the ReLU activation function, and the last two layers each contain 32 neurons with the ReLU activation function; the output layer uses the Sigmoid activation function, and its size depends on the number of monitored action categories. .

[0033] In this embodiment, the input of the Actor network input layer for each agent includes not only the agent state of the corresponding agent but also the global state of the agent system. Illustratively, the Actor network for each agent generates the corresponding agent's configuration action based on the current local observations (i.e., the agent's agent state) and the global state:

[0034] in, For the first The state space of an agent. Let be the Actor network parameters for the i-th agent.

[0035] Step 130: Using the target evaluator Critic network of the target agent, the value of the target agent's configuration actions is evaluated based on the global state, and the value evaluation result of the target agent's configuration actions is obtained.

[0036] In this embodiment, a Critic network can be constructed for each agent. The Critic networks of different agents can have different network structures. The Critic network is used to evaluate the value of taking corresponding configuration actions. Taking an agent system that includes a monitoring agent, an analysis agent, and a response agent as an example: For the monitoring agent, its Critic network The input layer receives the actions and states of all monitoring agents, where The network parameters are represented; the hidden layers of the Critic network for the monitoring agent can include four layers: the first two layers each contain 128 neurons with the ReLU activation function, and the last two layers each contain 256 neurons with the ReLU activation function; the output layer outputs the value assessment of the state-action combination. , where i represents the i-th monitoring agent.

[0037] For the analytical agent, its Critic network The input layer receives all actions and states of the analytical agent; the hidden layers of the analytical agent's Critic network are the same as those of the monitoring agent's Critic network; the output layer outputs the value assessment of the state-action combination. .

[0038] For the responding agent, its Critic network The input layer receives all actions and states of the responding agents; the hidden layers of the responding agents' Critic network are the same as those of the responding agents' Critic network; the output layer outputs the value assessment of the state-action combination. .

[0039] By building Actor and Critic networks independently for each type of agent, specifically addressing their specific tasks and decision-making needs, the optimization and personalization of the decision-making process are ensured, enabling each agent to achieve optimal performance in its area of ​​expertise.

[0040] In this embodiment, after inputting the configuration actions of the target agent and the global state of the agent system into the target Critic network corresponding to the target agent, the value evaluation result of the target Critic network for the configuration action can be obtained. In other words, the Critic network can provide an expected long-term return evaluation for the configuration actions of the corresponding agent. For reference only.

[0041] Step 140: Update the parameters of the target Actor network based on the value assessment results until the update completion condition is met, and then configure the server security of the target agent based on the updated target Actor network.

[0042] The agent's Actor network continuously updates its parameters based on the value gradient provided by the Critic network to improve its behavioral strategy and generate higher-value actions to obtain higher long-term returns. The completion condition for this update can be that the value assessment result corresponding to the configuration action generated by the target Actor network reaches a value assessment result threshold, or that the number of iterations reaches an update count threshold; this application does not impose any restrictions on these conditions.

[0043] After the update is completed, the computer device can generate a configuration action corresponding to the target agent based on the updated target agent network, the target agent's current agent state, and the global state of the agent system. Then, based on this configuration action, the target agent can be configured for server security, thereby improving server security.

[0044] In one possible implementation, in addition to updating the Actor network parameters of the agent, each iteration may also include updating the parameters of the Critic network of the agent. In the next iteration, the value assessment can be performed based on the Critic network after the parameter update in the previous iteration, which improves the accuracy of the value assessment of the configuration action. Then, the Actor network parameters are updated based on the value assessment result, which can improve the effect of updating the Actor network parameters and thus improve the accuracy of the configuration action generation.

[0045] Specifically, when updating the Critic network, the network parameters can be updated with the optimization objective of minimizing the difference between the value assessment result output by the Critic network and the actual value of the target value configuration action. The value assessment result output by the Critic network is the predicted value made by the Critic network based on the current state and action, and the target value is the sum of all rewards that can be obtained in the future starting from the current state and action, and the actual reward obtained.

[0046] Therefore, after obtaining the configuration actions of the target agent, the method further includes: Obtain the new global state after the target agent performs the configuration action; The reward value is calculated based on the new global state and the reward function corresponding to the target agent to obtain the reward value of the configured action; Calculate the target value of the configuration action based on the reward value of the configuration action and the new global state; The network parameters of the target Critic network are updated with the optimization objective of minimizing the difference between the value evaluation result of the configuration action and the target value.

[0047] In one possible implementation, each agent is configured with a corresponding reward function; since different agents have different business needs, the reward function calculation method for different agents can be different; illustratively, an agent system includes a monitoring agent, an analysis agent, and a response agent as an example: For a monitoring agent, its primary responsibility is to continuously monitor and accurately identify potential security threats while minimizing system resource consumption and false alarms. Therefore, the reward function of the monitoring agent fully considers the accuracy of threat identification, resource consumption, and timeliness. Thus, illustratively, its corresponding reward function is defined as follows:

[0048] in, These are weighting coefficients that can be adjusted according to business needs and security policies; TP represents the number of correctly identified threats, FP represents the number of incorrectly identified threats, LT represents the response and detection latency, and RC represents resource consumption.

[0049] For analytical agents, it is necessary to conduct in-depth analysis of detected threats, confirm the type and severity of the threats, and accurately classify them to guide response measures. Therefore, its reward function fully considers the accuracy of threat analysis and time consumption. Schematic, its corresponding reward function is defined as follows:

[0050] in, These are weighting coefficients, which can be adjusted based on the importance of accuracy and speed in the analysis; AC is the number of threats correctly classified, MI is the number of threats incorrectly classified, and AT is the time required to complete the analysis.

[0051] For the responding agent, it is responsible for implementing specific security measures based on the data provided by the analyzing agent. It needs to quickly and effectively mitigate or prevent threats. Therefore, its reward function fully considers the number of threats successfully blocked, as well as time and resource consumption. Illustratively, its corresponding reward function is defined as follows:

[0052] in, These are weighting coefficients, which need to be adjusted according to business needs and security strategies; TM represents the number of threats in the successful process or organization, FA represents the number of times corresponding actions were taken against incorrectly identified threats, and RE represents the efficiency of the corresponding measures.

[0053] In other words, in the embodiments of this application, the reward function of each intelligent agent is dynamically designed according to its responsibilities to reflect the business needs and security strategies in different situations. The dynamic reward function enables the intelligent agent to obtain appropriate feedback in different situations, which promotes more effective and accurate learning and action adjustment.

[0054] The data introduced into the reward coefficients of the various intelligent agents mentioned above can come from server real-time monitoring tools, log management systems, security device management data, system security management tools, policy configuration files, and system resource monitoring tools, etc.

[0055] After the target agent completes the configuration action, the computer device can obtain the new global state of the agent system, and calculate the reward value by combining the reward function corresponding to the target agent with the new global state. Illustratively, the corresponding calculation formula can be expressed as:

[0056] in, This is the new global state after the action is executed. Reward value R i It directly reflects the immediate performance of the agent after a specific action is performed.

[0057] After obtaining the reward value for the configuration action, the target value of the configuration action can be calculated based on the reward value and the new global state. This calculation involves adding the currently obtained reward value to the predicted value for the future state. Illustratively, the process of calculating the target value of the configuration action can be represented as follows:

[0058] in, Indicates the target value. It is a discount factor used to represent the current value of future rewards.

[0059] In one possible implementation, with the optimization objective of minimizing the difference between the value assessment result of the configuration action and the target value, the process of updating the network parameters of the target Critic network can be implemented as follows: The function value of the meta-learning loss function is calculated based on the value assessment results and the target value; The parameters of the Critic network of the target agent are updated based on the function value of the meta-learning loss function and the preset learning rate.

[0060] The meta-learning loss function can be expressed as:

[0061] in, This represents the expected (average) calculation of the loss function for combinations of different states and actions.

[0062] In one possible implementation, the optimization of the Critic network parameters can employ a two-layer optimization mechanism. In the inner rapid adaptation phase, the agent quickly adjusts its behavioral strategy to adapt to the new environment by modifying the meta-parameters of the Critic network. Based on the Critic network parameters pre-set by meta-learning, the agent performs a gradient calculation using data from the current new environment to rapidly fine-tune the parameters to adapt to the new task. This parameter update formula can be expressed as:

[0063] in, Indicates the parameters of the Critic network. Indicates the learning rate. It is for new data The meta-learning loss function is calculated.

[0064] In the outer meta-learning phase, unlike conventional learning which focuses solely on maximizing immediate rewards, meta-learning optimizes the entire learning strategy, including how to learn, when to adjust the strategy, and how to effectively utilize past experiences. The system calculates the meta-learning loss based on the long-term cumulative rewards generated by this adapted strategy in the environment, and ultimately backpropagates this signal to adjust the initial meta-parameters. The parameter update formula can be expressed as:

[0065] in, This represents the Critic network parameters updated after meta-learning. This represents the current parameters of the Critic network for agent i. Indicates the learning rate. This indicates the possible configuration actions to choose from. This updated formula takes into account long-term rewards, improving the policy's foresight and sustainability.

[0066] In other words, the inner process quickly adjusts its behavior based on new data, while the outer process optimizes the inner process's ability to adjust itself based on the long-term results of the behavior. By applying meta-learning strategies, the system can quickly adapt and learn when faced with new or unknown threats.

[0067] After updating the network parameters of the Critic network, the method also includes: The updated target Critic network is used to evaluate the value of the target agent's configuration actions based on the global state.

[0068] Indicative, Figure 3 This illustration shows a schematic diagram of a network parameter update process provided in an exemplary embodiment of this application, such as... Figure 3As shown, after initialization, for any agent in the agent system, a configuration action is first generated through the agent's Actor network. Then, based on this configuration action and the new global state after its execution, a reward value is calculated. This reward value is then used to update the meta-learning policy of the Critic network. Finally, the updated Critic network is used to evaluate the value of the configuration action, and the Actor network parameters are updated based on this evaluation result. This process is repeated until the business requirements are met. Finally, the updated Actor network is used to perform server security configuration on the target agent. The implementation methods for each of these processes can be found in the foregoing related content and will not be elaborated here.

[0069] By evaluating the value of the configuration actions output by the Actor network of the corresponding agent through the updated Critic network, the accuracy of the value evaluation of the configuration actions can be improved. Consequently, when updating the network parameters of the Actor network based on the value evaluation results, the accuracy of the parameter update of the Actor network can be improved, the network update time can be shortened, the update effect can be improved, and the accuracy of the generated configuration actions can be improved.

[0070] Repeatedly updating the Critic network parameters driven by meta-learning enables the Actor network of each agent to generate more valuable server security configuration actions until business requirements are met. In one possible implementation, during the parameter update process, each agent stores its experience in an experience replay mechanism so that all agents can share their experiences, enabling knowledge transfer between agents and thereby improving the server's adaptability to new security threats and environmental changes.

[0071] In summary, the server security configuration method provided in this application obtains the agent state of the target agent in the intelligent agent system and the global state formed by integrating the states of all agents; generates configuration actions based on the agent state and the global state through the Actor network of the target agent; evaluates the value of the configuration actions based on the global state through the Critic network of the target agent; finally, updates the Actor network parameters according to the evaluation results, and realizes the autonomous evolution of the agent through iterative optimization; by introducing a global state perspective and a distributed learning architecture, each agent can fully consider the overall system situation when making decisions, effectively overcoming the shortcomings of traditional centralized systems that rely on static rules and have poor adaptability, improving the system's ability to identify and respond to new threats and complex attacks, enhancing the initiative and adaptability of overall security protection, and reducing the security risks caused by lagging rule updates.

[0072] Figure 4This illustration shows a block diagram of a server security configuration apparatus provided in an exemplary embodiment of this application, the apparatus being used to perform, for example... Figure 1 All or part of the steps in the illustrated embodiments, such as Figure 4 As shown, the device may include the following modules.

[0073] The state acquisition module 410 is used to acquire the agent state of the target agent in the agent system and the global state of the agent system. The global state is the integration of the agent states of each agent included in the agent system. The target agent is any one of the agents. The action generation module 420 is used to generate configuration actions for the target intelligent agent by using the target actor network of the target intelligent agent, based on the agent state of the target intelligent agent and the global state, to obtain the configuration actions of the target intelligent agent. The value assessment module 430 is used to assess the value of the configuration actions of the target intelligent agent based on the global state through the target evaluator Critic network of the target intelligent agent, and obtain the value assessment result of the configuration actions of the target intelligent agent. The first parameter update module 440 is used to update the parameters of the target Actor network based on the value assessment result until the update completion condition is met, so as to perform server security configuration on the target agent based on the updated target Actor network.

[0074] In one possible implementation, each agent is configured with a corresponding reward function; The state acquisition module is used to acquire the new global state of the target intelligent agent after it has executed the configuration action, after obtaining the configuration action of the target intelligent agent; The device further includes: The reward value calculation module is used to calculate the reward value based on the new global state and the reward function corresponding to the target agent, so as to obtain the reward value of the configured action; The target value calculation module is used to calculate the target value of the configuration action based on the reward value of the configuration action and the new global state; The second parameter update module is used to update the network parameters of the target Critic network with the optimization objective of minimizing the difference between the value evaluation result of the configuration action and the target value.

[0075] In one possible implementation, the second parameter update module includes: The function value calculation submodule is used to calculate the function value of the meta-learning loss function based on the value assessment result and the target value; The parameter update submodule is used to update the parameters of the Critic network of the target agent based on the function value of the meta-learning loss function and the preset learning rate.

[0076] In one possible implementation, the value evaluation module 430 is used to evaluate the configuration actions of the target agent based on the global state through the updated target Critic network after updating the network parameters of the Critic network.

[0077] In one possible implementation, the intelligent agent system includes a monitoring agent, an analyzing agent, and a responding agent; The monitoring agent is used to generate monitoring actions based on the monitoring agent's state, which includes network traffic characteristics, system activity logs, and network connection status. The analytical agent is used to generate analytical actions based on the state of the analytical agent, which includes current threat data, historical threat data, and network security situation. The response agent is used to generate response actions based on the response agent state, which includes the current security alert, response policy library, and system defense status.

[0078] In one possible implementation, the configuration actions of the monitoring agent include adjusting the data acquisition strategy, selecting the monitoring range, and setting the alarm sensitivity. The configuration actions of the analytical agent include in-depth adjustment of threat assessment, selection of analytical tools, and updating of risk assessment. The configuration actions of the response agent include implementing configuration changes, activating stress measures, and optimizing response strategies.

[0079] In summary, the server security configuration device of this application acquires the agent state of the target agent in the intelligent agent system and the global state formed by integrating the states of all agents; generates configuration actions based on the agent state and the global state through the Actor network of the target agent; evaluates the value of the configuration actions based on the global state through the Critic network of the target agent; and finally updates the Actor network parameters according to the evaluation results, achieving autonomous evolution of the agent through iterative optimization. In the above device, by introducing a global state perspective and a distributed learning architecture, each agent can fully consider the overall system situation when making decisions, effectively overcoming the shortcomings of traditional centralized systems that rely on static rules and have poor adaptability. This improves the system's ability to identify and respond to new threats and complex attacks, enhances the initiative and adaptability of overall security protection, and reduces the security risks caused by lagging rule updates.

[0080] Figure 5 A structural block diagram of a computer device 500 illustrated in an exemplary embodiment of this application is shown. This computer device can be implemented as a server as described above in this application. The computer device 500 includes a Central Processing Unit (CPU) 501, a system memory 504 including Random Access Memory (RAM) 502 and Read-Only Memory (ROM) 503, and a system bus 505 connecting the system memory 504 and the CPU 501. The computer device 500 also includes a mass storage device 506 for storing an operating system 509, application programs 510, and other program modules 511.

[0081] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 504 and mass storage device 506 described above can be collectively referred to as memory.

[0082] According to various embodiments of this disclosure, the computer device 500 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 500 can be connected to a network 508 via a network interface unit 507 connected to the system bus 505, or it can use the network interface unit 507 to connect to other types of networks or remote computer systems (not shown).

[0083] The memory also includes at least one instruction, at least one program, code set, or instruction set, which are stored in the memory. The central processing unit 501 executes the at least one instruction, at least one program, code set, or instruction set to implement all or part of the steps in the server security configuration method shown in the above embodiments.

[0084] Figure 6 A structural block diagram of a computer device 600 illustrating an exemplary embodiment of this application is shown. The computer device 600 can be implemented as the aforementioned terminal device, such as a smartphone, tablet computer, laptop computer, desktop computer, etc. The computer device 600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0085] Typically, computer device 600 includes a processor 601 and a memory 602.

[0086] In some embodiments, the computer device 600 may optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, and a power supply 608.

[0087] In some embodiments, the computer device 600 further includes one or more sensors 609. The one or more sensors 609 include, but are not limited to, an accelerometer 610, a gyroscope 611, a pressure sensor 612, an optical sensor 613, and a proximity sensor 614.

[0088] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the computer device 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0089] In one exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to implement all or part of the steps in the server security configuration method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0090] In one exemplary embodiment, a computer program product is also provided, comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the above-described actions. Figure 1 All or part of the steps of the server security configuration method shown in the embodiments.

[0091] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0092] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A server security configuration method, characterized in that, The method includes: The system obtains the agent state of the target agent in the agent system and the global state of the agent system. The global state is the integration of the agent states of each agent in the agent system. The target agent is any one of the agents. The configuration actions of the target intelligent agent are obtained by generating configuration actions based on the agent state of the target intelligent agent and the global state through the target agent's Actor network. The target intelligent agent's configuration actions are evaluated based on the global state through the target evaluator Critic network, and the evaluation result of the target intelligent agent's configuration actions is obtained. Based on the value assessment results, the parameters of the target Actor network are updated until the update completion condition is met, so as to perform server security configuration on the target agent based on the updated target Actor network.

2. The method according to claim 1, characterized in that, Each agent is configured with a corresponding reward function; after obtaining the configured actions of the target agent, the method further includes: Obtain the new global state of the target agent after it performs the configuration action; The reward value is calculated based on the new global state and the reward function corresponding to the target agent to obtain the reward value of the configured action; Based on the reward value of the configured action and the new global state, calculate the target value of the configured action; With the optimization objective of minimizing the difference between the value assessment result of the configuration action and the target value, the network parameters of the target Critic network are updated.

3. The method according to claim 2, characterized in that, The step of updating the network parameters of the Critic network with the optimization objective of minimizing the difference between the value evaluation result of the configuration action and the target value includes: The value of the meta-learning loss function is calculated based on the value assessment results and the target value. The parameters of the Critic network of the target agent are updated based on the function value of the meta-learning loss function and the preset learning rate.

4. The method according to claim 2 or 3, characterized in that, After updating the network parameters of the Critic network, the method further includes: The updated target Critic network is used to evaluate the value of the target agent's configuration actions based on the global state.

5. The method according to any one of claims 1 to 4, characterized in that, The intelligent agent system includes a monitoring intelligent agent, an analytical intelligent agent, and a response intelligent agent; The monitoring agent is used to generate monitoring actions based on the monitoring agent's state, which includes network traffic characteristics, system activity logs, and network connection status. The analytical agent is used to generate analytical actions based on the state of the analytical agent, which includes current threat data, historical threat data, and network security situation. The response agent is used to generate response actions based on the response agent state, which includes the current security alert, response policy library, and system defense status.

6. The method according to claim 5, characterized in that, The configuration actions of the monitoring agent include adjusting the data acquisition strategy, selecting the monitoring range, and setting the alarm sensitivity. The configuration actions of the analytical agent include in-depth adjustment of threat assessment, selection of analytical tools, and updating of risk assessment. The configuration actions of the response agent include implementing configuration changes, activating stress measures, and optimizing response strategies.

7. A server security configuration device, characterized in that, The device includes: The state acquisition module is used to acquire the agent state of the target agent in the agent system and the global state of the agent system. The global state is the integration of the agent states of each agent included in the agent system. The target agent is any one of the agents. The action generation module is used to generate configuration actions for the target intelligent agent by using the target actor network of the target intelligent agent, based on the agent state of the target intelligent agent and the global state, to obtain the configuration actions of the target intelligent agent. The value assessment module is used to assess the value of the configuration actions of the target intelligent agent based on the global state through the target evaluator Critic network of the target intelligent agent, and obtain the value assessment result of the configuration actions of the target intelligent agent. The first parameter update module is used to update the parameters of the target Actor network based on the value assessment result until the update completion condition is met, so as to perform server security configuration on the target agent based on the updated target Actor network.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to implement the server security configuration method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the server security configuration method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the server security configuration method as described in any one of claims 1 to 6.

Images