Cloud-edge-device security detection task scheduling method based on multi-agent collaboration
By adopting the security detection task scheduling method of multi-agent collaborative in the cloud-edge-end computing mode, and using reinforcement learning and reward mechanisms to optimize security detection tasks, the security detection problem of resource-constrained devices is solved and efficient network attacks are achieved Detection and resource utilization.
Patent Information
- Application Number
- CN202411251666.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-09
AI Technical Summary
The existing cloud-edge-end computing model faces resource-constrained computing, storage and energy limitations of devices in terms of security detection. The traditional task offload model cannot effectively consider the security capabilities of the device and the collaboration of heterogeneous nodes, resulting in detection in real-time environments. Poor performance.
Adopt cloud-edge-end security detection task scheduling method based on multi-agent collaboration, deploy policy networks on end devices through intelligent agents, collect and process status parameters, optimize security detection task scheduling using reinforcement learning and reward mechanisms, and consider node security design comprehensive reward functions to optimize decision-making.
It significantly improves the detection ability of end devices for network attacks, optimizes resource utilization efficiency, ensures that scheduling strategies are optimized on resource-constrained devices, takes into account the balance of real-time and resource utilization, and improves the attack detection ability of nodes.
Smart Images

Figure CN118764325B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cyberspace security technology, and in particular to a cloud-edge-end security detection task scheduling method based on multi-agent collaboration. Background Art
[0002] With the rapid development of the Internet, end-edge-cloud computing (EECC) has gradually become the mainstream computing paradigm. In the processing of resource-intensive and delay-sensitive services, such as image processing, autonomous driving, and healthcare, EECC has attracted much attention for its strong resource support and high real-time analysis. However, this cloud-edge-end collaborative model faces challenges in security detection. First, the availability of resources is in a decreasing relationship between cloud devices, edge devices, and end devices. The computing, storage, energy, and communication resources of end devices are usually limited, which makes it difficult for end devices to take care of both ordinary tasks and security detection tasks at the same time. One solution is to perform task offloading and transfer computing tasks from resource-constrained devices to devices with more abundant resources. Traditional task offloading problems can be divided into two categories: energy-saving task allocation and bandwidth-aware task allocation, which aim to minimize device energy consumption and minimize transmission delay, respectively. However, given the particularity of security detection tasks, traditional task offloading models cannot be directly applied to security detection tasks. First, for security detection tasks, the security capabilities of heterogeneous nodes need to be additionally considered. For example, if a node occupies most of the system resources while performing ordinary tasks, causing the security detection task to fail to operate normally, then such a device is very likely to become a target of attack. Secondly, if a device is attacked, its neighboring devices are also vulnerable. Therefore, the offloading problem must also take into account the security capabilities of the current device and its neighboring devices to adjust the offloading strategy.
[0003] In addition, some existing studies have considered how to effectively deploy security detection algorithms on resource-constrained devices in the EECC environment, but these studies are limited in the flexibility of resource scheduling. First, the task offloading problem dynamically allocates computing resources according to the real-time resource requirements of the system, which is more intelligent in terms of resource optimization and load balancing. However, existing studies rely on the collaboration between a certain number of distributed nodes. For example, a preliminary intrusion detection task is performed on the edge device, and then the necessary data or model is uploaded to the cloud for further detection. This approach alleviates the limitations of edge device computing resources through collaboration between nodes, but is not effective in the face of real-time changing resource requirements. Second, considering that security detection tasks often require high real-time performance, existing solutions do not explicitly discuss how to ensure the performance of the system in real-time or near-real-time environments. In addition, existing security detection research solutions based on the EECC environment mainly focus on using distributed architecture to improve the attack detection capabilities of nodes, while ignoring the security capabilities of heterogeneous nodes in the distributed architecture. Summary of the invention
[0004] In order to overcome the shortcomings of the above technologies, the present invention provides a cloud-edge-end security detection task scheduling method based on multi-agent collaboration, which can significantly improve the terminal device's detection capability of network attacks, while ensuring the optimization of scheduling strategies on resource-constrained terminal devices and balancing real-time and resource utilization efficiency.
[0005] The technical solution adopted by the present invention to overcome the technical problems is:
[0006] A cloud-edge-device security detection task scheduling method based on multi-agent collaboration includes the following steps:
[0007] (a) Deploy the intelligent agent to each end device in the entire network. The intelligent agent collects The moment Terminal equipment Status parameters , , is the number of mid-range devices in the entire network, , For the cycle;
[0008] (b) The intelligent agent Terminal equipment Deploy the policy network on Terminal equipment Status parameters Input into the policy network and output Security detection task scheduling strategy at all times ;
[0009] (c) Intelligent agent execution scheduling strategy After that, the intelligent agent collects Terminal equipment Status parameters ;
[0010] (d) Using state parameters Calculate the reward of the intelligent agent ;
[0011] (e) Central cloud controller acquisition time The information of intelligent agents is constructed The vector of global information at each moment ,in, , , , , the central cloud controller will periodically The vector of global information at all times is stored in the buffer middle;
[0012] (f) Deployment in a central cloud controller value network and target network, each value network and target network corresponds to an end device, each value network and target network from the buffer Random Sampling The vector of global information, The vector of global information is , , the value network outputs the Q value of reinforcement learning , the target network outputs the Q value of reinforcement learning at the next moment ;
[0013] (g) Using Q value and Q value Update the value network;
[0014] (h) training the policy network;
[0015] (i) Update the parameters of the target network using a soft update method;
[0016] (j) repeating steps (e) to (i) until a predetermined training cycle is reached or learning converges;
[0017] (h) Terminal equipment Execute the security detection task scheduling strategy output by its own deployed policy network, interact with the environment, and make scheduling decisions for security detection tasks.
[0018] Furthermore, the state parameter in step (a) It includes the status of the node, the status of the task, and the status of the link. The node status includes the resource occupancy rate of the end node, the resource balance index of the end node, the security metric of the end node, and the resource occupancy rate of the edge node; the task status includes the node resources requested by the current task, the amount of task data, and the urgency of the task.
[0019] Furthermore, the intelligent agent collects Terminal equipment Status parameters The method of the node status is:
[0020] (a-1) The resource occupancy rate of the end node is given by Resource usage of non-safety detection tasks at all times and Resource usage of security detection tasks at all times Composition, through the formula Calculated Resource usage of non-safety detection tasks at all times , where For the Terminal equipment The end node system resources that are occupied by non-safety detection tasks at all times, For the Terminal equipment The total resources at the moment are calculated by the formula Calculated Resource usage of security detection tasks at all times , where For the Terminal equipment Always perform security detection tasks on the occupied end node system resources. ;
[0021] (a-2) Through the formula Calculated End node resource balance index at the moment ;
[0022] (a-3) Through the formula Calculate the end node security metric , where is the balance factor, For the Terminal equipment The safety indicators , and are weight coefficients, For the In the cycle Terminal equipment The duration of the test, For the Terminal equipment In the The number of detection interruptions within a cycle, As parameters, For the Terminal equipment A collection of other directly connected end devices, For the Terminal equipment The number of directly connected end devices, For collection Middle Terminal equipment The safety indicators , For the In the cycle Terminal equipment The duration of the test, For the Terminal equipment In the Number of detection interruptions within a cycle;
[0023] (a-4) Through the formula Calculated Moment Resource occupancy rate of edge nodes , , is the number of edge nodes, for Moment The system resources that have been occupied by the edge node are for Moment The total resources of the edge nodes, the resource occupancy rate of all edge nodes is , .
[0024] Furthermore, the intelligent agent collects Terminal equipment Status parameters The status method of the task is:
[0025] (a-5) The node resources requested by the current task include the current CPU percentage requested by security detection tasks 、Current The percentage of memory requested by security detection tasks ;
[0026] (a-6) Get the current The amount of data requested by a security detection task ;
[0027] (a-7) Get the current The urgency of the security inspection task request , A positive number greater than 0 and less than 1.
[0028] Furthermore, the link status is the link bandwidth , No. Terminal equipment Status parameters .
[0029] Further, step (b) comprises the following steps:
[0030] (b-1) The policy network consists of the first fully connected layer, the second fully connected layer, the discrete action output layer, and the continuous action output layer;
[0031] (b-2) Terminal equipment Status parameters Input into the first fully connected layer of the policy network and output the feature ;
[0032] (b-3) The feature Input into the second fully connected layer of the policy network and output the feature ;
[0033] (b-4) The discrete action output layer of the policy network is composed of a fully connected layer and a softmax function. Input to the fully connected layer of the discrete action output layer, and the output is Dimensional vector , the vector Input into the softmax function of the discrete action output layer, and the output is the probability distribution , using the torch.distributions.Categorical function in the Pytorch library Dimensional probability distribution Randomly sample an element and get the element in Position index in the dimensional vector , , use PyTorch's torch.zeros function to generate a The zero vector of , using the index Set the value of the element at the corresponding index position in the zero vector to 1, and get the vector , For the For the elements, , , ,if , then the safety detection task is scheduled to If the edge nodes execute , then the safety detection task is not scheduled to the If the edge nodes execute , then the safety detection task is in the Terminal equipment Execute on;
[0034] (b-5) The continuous action output layer of the policy network is composed of a fully connected layer and a sigmoid function. Input the fully connected layer of the continuous action output layer, and output a two-dimensional vector , transform the vector The sigmoid function input to the continuous action output layer outputs a value between 0 and 1. , ,result As a task, get the CPU percentage of the selected edge or end node. The percentage of memory capacity of the selected edge or end node obtained as a task to construct a triple , .
[0035] Further, step (d) comprises the following steps:
[0036] (d-1). Through the formula
[0037] Calculate the task balance reward , where for The end node resource balance index at the time, and All are parameters. , , are all set thresholds. , for Resource occupancy rate of non-safety detection tasks at all times, for Resource usage rate of security detection tasks at all times;
[0038] (d-2) by formula Calculate the task urgency reward , where and All are positive weights. The safety inspection task is Terminal equipment When executed above, , No. When a security detection task is executed on the edge node, , For the The round trip delay of a safety detection task, , , For a cycle within, no. The amount of data sent by each security detection task, For the The transmission rate of the safety detection task sending process, , For a cycle within, no. The amount of data returned by a security detection task, For the The transmission rate of the return process of a safety detection task, is the reward value when the task is executed at the end node, , To adjust the parameters;
[0039] (d-3) by formula Calculate the edge node load balancing reward , where for Moment The resource utilization rate of the edge nodes of the platform, , , for Moment The system resources that have been occupied by the edge node are for Moment Total resources of edge nodes;
[0040] (d-4) by formula Calculate the relative security reward , where is the average value of the security metric of the end node itself and its neighboring end nodes, ;
[0041] (d-5) by formula Calculate the reward of the intelligent agent , where , , , are all weight coefficients.
[0042] Further, step (f) comprises the following steps:
[0043] (f-1). The value network consists of the first fully connected layer, the second fully connected layer, and the third fully connected layer. In the vector of global information and The input is mapped into the first fully connected layer of the value network to obtain a hidden layer, and the output is vector , the vector Input into the second fully connected layer of the value network and output the vector , the vector Input into the third fully connected layer of the value network, and output the Q value of the value network output reinforcement learning ;
[0044] (f-2) The target network consists of the first fully connected layer, the second fully connected layer, and the third fully connected layer. In the vector of global information middle The state parameters are input into the policy network respectively, and the output is Each security detection task scheduling strategy constitutes a vector , will In the vector of global information and vector The input is mapped to the first fully connected layer of the target network to obtain a hidden layer, and the output is vector , the vector Input into the second fully connected layer of the target network and output the vector , the vector Input into the third fully connected layer of the target network, and output the Q value of reinforcement learning at the next moment .
[0045] Furthermore, in step (g), the Q value is used by the time difference algorithm and Q value The value network is updated, and in step (h), the policy network is trained using the policy gradient algorithm.
[0046] The beneficial effects of the present invention are as follows: a security measurement method for nodes in an EECC distributed architecture is proposed, and a node security index is designed. The end node uses the security index of the neighboring nodes around it for joint consideration to measure its ability to resist external attacks. At the same time, the index is also used to guide the intelligent agent to schedule security detection tasks and optimize strategies. It is achieved while ensuring node security and optimizing resource utilization. The cloud-edge collaborative optimization problem is modeled as a Markov game process, which takes into account the characteristics of nodes, tasks and links, as well as the load balancing of edge nodes, to make better security detection task scheduling decisions. A comprehensive reward function is designed to optimize the decision-making of intelligent agents, improve the attack detection capabilities of nodes, and at the same time balance security detection tasks in resource allocation and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a network framework diagram of the present invention;
[0048] Figure 2 The present invention is a flow chart of the method. DETAILED DESCRIPTION
[0049] The following is combined with Figure 1 , Attachment Figure 2 The present invention is further described.
[0050] In recent years, multi-agent collaborative systems (MACS) have attracted widespread attention due to their potential in complex task processing and decision making. In MACS, each agent has a certain degree of autonomy and is able to perceive the environment, make decisions, and interact with other agents. Multiple agents learn together how to solve tasks to adapt to more complex and flexible scenarios. In combination with MACS, the present invention designs a cloud-edge-end security detection task scheduling method based on multi-agent collaboration, which includes the following steps:
[0051] (a) Deploy the intelligent agent to each end device in the entire network. The intelligent agent collects The moment Terminal equipment Status parameters , , is the number of mid-range devices in the entire network, , For cycle.
[0052] (b) The intelligent agent Terminal equipment Deploy the policy network on Terminal equipment Status parameters Input into the policy network and output Security detection task scheduling strategy at all times .
[0053] (c) Intelligent agent execution scheduling strategy After that, the intelligent agent collects Terminal equipment Status parameters .
[0054] (d) Using state parameters Calculate the reward of the intelligent agent .
[0055] (e) Central cloud controller acquisition time The information of intelligent agents is constructed The vector of global information at each moment ,in, , , , , the central cloud controller will periodically The vector of global information at all times is stored in the buffer middle.
[0056] (f) Deployment in a central cloud controller value network and target network, each value network and target network corresponds to an end device, each value network and target network from the buffer Random Sampling The vector of global information, The vector of global information is , , the value network outputs the Q value of reinforcement learning , the target network outputs the Q value of reinforcement learning at the next moment .
[0057] (g) Using Q value and Q value Update the value network.
[0058] (h) Training the policy network.
[0059] (i) Use the soft update method to update the parameters of the target network.
[0060] (j) Repeat steps (e) to (i) until a predetermined training period is reached or learning converges.
[0061] (h) Terminal equipment Execute the security detection task scheduling strategy output by its own deployed policy network. During the execution phase, the intelligent agent in the end device no longer relies on the central cloud controller, but independently interacts with the environment to schedule security detection tasks.
[0062] The present invention models the problem as a Markov game process, deploys an off-policy intelligent agent at each end node, takes the current node status, the security detection task status and the link status as input, and outputs the security detection task scheduling decision made by the agent. The goal of the present invention is to adaptively schedule some security detection tasks in resource-constrained end nodes to other resource-sufficient edge nodes for distributed execution. This strategy aims to significantly improve the end device's ability to detect network attacks, while ensuring the optimization of scheduling strategies on resource-constrained end devices, taking into account the balance between real-time performance and resource utilization efficiency.
[0063] In one embodiment of the present invention, the state parameter in step (a) It includes the status of the node, the status of the task, and the status of the link. The node status includes the resource occupancy rate of the end node, the resource balance index of the end node, the security metric of the end node, and the resource occupancy rate of the edge node; the task status includes the node resources requested by the current task, the amount of task data, and the urgency of the task.
[0064] In one embodiment of the present invention, the intelligent agent collects Terminal equipment Status parameters The method of the node status is:
[0065] (a-1) The resource occupancy rate of the end node is used to indicate the ratio of the resources occupied by the end node in the current time step to the total resources of the node. The resource occupancy rate of the end node is given by Resource usage of non-safety detection tasks at all times and Resource usage of security detection tasks at all times Composition, through the formula Calculated Resource usage of non-safety detection tasks at all times , where For the Terminal equipment The end node system resources that are occupied by non-safety detection tasks at all times, For the Terminal equipment The total resources at the moment are calculated by the formula Calculated Resource usage of security detection tasks at all times , where For the Terminal equipment Always perform security detection tasks on the occupied end node system resources. .
[0066] (a-2) Through the formula Calculated End node resource balance index at the moment . End node resource balance index It is used to measure the ratio of resource occupancy of end nodes performing non-safety detection tasks to the resource occupancy of all tasks.
[0067] (a-3) Through the formula Calculate the end node security metric . It is used to measure the security level of the end node under distributed conditions. is the balance factor, For the Terminal equipment The safety indicators , and are weight coefficients, For the In the cycle Terminal equipment The duration of the test, For the Terminal equipment In the The number of detection interruptions within a cycle, is a parameter to prevent the node's security index from being 0. For the Terminal equipment A collection of other directly connected end devices, For the Terminal equipment The number of directly connected end devices, For collection Middle Terminal equipment The security index of , without considering the distribution, for a single terminal node, at the beginning of the cycle, The purpose of the design is to hope that the increase in the number of interruptions can reduce the positive impact of the duration on the security of the end node to a certain extent. For the In the cycle Terminal equipment The duration of the test, For the Terminal equipment In the The number of detection interruptions within a cycle.
[0068] (a-4) Through the formula Calculated Moment Resource occupancy rate of edge nodes , , is the number of edge nodes, for Moment The system resources that have been occupied by the edge node are for Moment The total resources of the edge nodes, the resource occupancy rate of all edge nodes is , .
[0069] In one embodiment of the present invention, the intelligent agent collects Terminal equipment Status parameters The status method of the task is:
[0070] (a-5) The node resources requested by the current task include the current CPU percentage requested by security detection tasks 、Current The percentage of memory requested by security detection tasks .
[0071] (a-6) Get the current The amount of data requested by a security detection task .
[0072] (a-7) Get the current The urgency of the security inspection task request , It is a positive number greater than 0 and less than 1. The closer its value is to 1, the higher the real-time requirement of the security detection task is, and vice versa.
[0073] In one embodiment of the present invention, the state of the link is the link bandwidth , No. Terminal equipment Status parameters .
[0074] In one embodiment of the present invention, step (b) comprises the following steps:
[0075] (b-1) The policy network consists of the first fully connected layer, the second fully connected layer, the discrete action output layer, and the continuous action output layer.
[0076] (b-2) Terminal equipment Status parameters Input into the first fully connected layer of the policy network and output the feature .
[0077] (b-3) The feature Input into the second fully connected layer of the policy network and output the feature .
[0078] (b-4) The discrete action output layer of the policy network is composed of a fully connected layer and a softmax function. Input to the fully connected layer of the discrete action output layer, and the output is Dimensional vector , the vector Input into the softmax function of the discrete action output layer, and the output is the probability distribution , using the torch.distributions.Categorical function in the Pytorch library Dimensional probability distribution Randomly sample an element and get the element in Position index in the dimensional vector , , use PyTorch's torch.zeros function to generate a The zero vector of , using the index The value of the element at the corresponding index position in the zero vector is set to 1, so that the sampled element is represented in the form of a one-hot encoding, and the vector is obtained. , For the For the elements, , , ,if , then the safety detection task is scheduled to If the edge nodes execute , then the safety detection task is not scheduled to the If the edge nodes execute , then the safety detection task is in the Terminal equipment For example, if , it means that the current security detection task is executed on the local node
[0079] (b-5) The continuous action output layer of the policy network is composed of a fully connected layer and a sigmoid function. Input the fully connected layer of the continuous action output layer, and output a two-dimensional vector , transform the vector The sigmoid function input to the continuous action output layer outputs a value between 0 and 1. , ,result As a task, get the CPU percentage of the selected edge or end node. The percentage of memory capacity of the selected edge or end node obtained as a task to construct a triple , .
[0080] In one embodiment of the present invention, step (d) comprises the following steps:
[0081] (d-1). Through the formula
[0082] Calculate the task balance reward , where for The end node resource balance index at the time, and All are parameters. , , The resource balance index is a set threshold to ensure that the reward does not depend too much on the relative resource usage of different types of tasks. For example, when the resource usage of a certain end node is greater than the set threshold, if the end node occupies more resources for non-safety detection tasks and ignores safety detection tasks, the resource balance index is Above threshold Mission Balance Rewards The value of will decrease. As a penalty term, it is used to specify the extent to which the reward decreases when the security detection task is excessively ignored. The parameter The role of is to adjust the upper limit of the reward. , for Resource occupancy rate of non-safety detection tasks at all times, for Always check the resource usage of the task safely. Task balance reward It is used to measure whether the intelligent agent can balance the resource usage of the current end node in executing non-safety detection tasks and safety detection tasks after making a safety detection task scheduling strategy. Therefore, it is necessary to use the end node resource usage rate at the next moment and the end node resource balance index at the next moment.
[0083] (d-2) by formula Calculate the task urgency reward , Mission Urgency Reward It is used to represent the rewards obtained by the intelligent agent when handling tasks of different urgency. For example, for security detection tasks with high urgency, they should be executed on the end node as much as possible to obtain better real-time performance, while for detection tasks with lower urgency, they can be considered to be executed on the edge node. In the formula, and All are positive weights, used to balance the impact of different factors on rewards. The safety inspection task is Terminal equipment When executed above, , No. When a security detection task is executed on the edge node, , For the The round trip delay of a safety detection task, , , For a cycle within, no. The amount of data sent by each security detection task, For the The transmission rate of the safety detection task sending process, , For a cycle within, no. The amount of data returned by a security detection task, For the The transmission rate of the return process of a safety detection task, is the reward value when the task is executed at the end node, , To adjust the parameters.
[0084] (d-3) The load balancing reward of edge nodes is used to measure whether the scheduling strategy of security detection tasks can ensure that each edge node can balance its own computing resources and achieve load balancing of edge nodes. The load balancing reward of edge nodes is defined by the Jain fairness index, so the formula is Calculate the edge node load balancing reward , where for Moment The resource utilization rate of the edge nodes of the platform, , , for Moment The system resources that have been occupied by the edge node are for Moment The total resources of edge nodes. Because the load balancing reward is obtained after a security detection task scheduling decision, the expression considers the edge node resource occupancy rate at the next moment The value range of the Jain fairness index is between 0 and 1. The closer the Jain index is to 1, the better the load balancing of the node is, and the closer it is to 0, the worse the load balancing is.
[0085] (d-4) by formula Calculate the relative security reward , where is the average value of the security metric of the end node itself and its neighboring end nodes, . Relative Security Rewards is to measure in a cycle The beginning of the current node The difference in security metrics of the end nodes with their neighboring perimeters.
[0086] (d-5) by formula Calculate the reward of the intelligent agent , where , , , All are weight coefficients. Used to represent the relative security reward of the end node, the weight coefficient Used to indicate task balance reward, weight coefficient Used to represent the edge node load balancing reward, weight coefficient Used to indicate the importance of the task urgency reward to the total reward.
[0087] In one embodiment of the present invention, step (f) comprises the following steps:
[0088] (f-1). The value network consists of the first fully connected layer, the second fully connected layer, and the third fully connected layer. In the vector of global information and The input is mapped into the first fully connected layer of the value network to obtain a hidden layer, and the output is vector , the vector Input into the second fully connected layer of the value network and output the vector , the vector Input into the third fully connected layer of the value network, and output the Q value of the value network output reinforcement learning .
[0089] (f-2) The target network consists of the first fully connected layer, the second fully connected layer, and the third fully connected layer. In the vector of global information middle The state parameters are input into the policy network respectively, and the output is Each security detection task scheduling strategy constitutes a vector , will In the vector of global information and vector The input is mapped to the first fully connected layer of the target network to obtain a hidden layer, and the output is vector , the vector Input into the second fully connected layer of the target network and output the vector , the vector Input into the third fully connected layer of the target network, and output the Q value of reinforcement learning at the next moment .
[0090] In step (g), the Q value is used by the time difference algorithm and Q value The value network is updated, and in step (h), the policy network is trained using the policy gradient algorithm.
[0091] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A cloud-edge-device security detection task scheduling method based on multi-agent collaboration, characterized in that: The steps include: (a) Deploy the intelligent agent to each end device in the entire network. The intelligent agent collects The moment Terminal equipment Status parameters , , is the number of mid-range devices in the entire network, , For the cycle; (b) The intelligent agent Terminal equipment Deploy the policy network on Terminal equipment Status parameters Input into the policy network and output Security detection task scheduling strategy at all times ; (c) Intelligent agent execution scheduling strategy After that, the intelligent agent collects Terminal equipment Status parameters ; (d) Using state parameters Calculate the reward of the intelligent agent ; (e) Central cloud controller acquisition time The information of intelligent agents is constructed The vector of global information at each moment ,in, , , , , the central cloud controller will periodically The vector of global information at all times is stored in the buffer middle; (f) Deployment in a central cloud controller value network and target network, each value network and target network corresponds to an end device, each value network and target network from the buffer Random Sampling The vector of global information, The vector of global information is , , the value network outputs the Q value of reinforcement learning , the target network outputs the Q value of reinforcement learning at the next moment ; (g) Using Q value and Q value Update the value network; (h) training the policy network; (i) Update the parameters of the target network using a soft update method; (j) repeating steps (e) to (i) until a predetermined training cycle is reached or learning converges; (h) Terminal equipment Execute the security detection task scheduling strategy output by its own deployed policy network; Step (d) comprises the following steps: (d-1) by formula Calculate the task balance reward , where for The end node resource balance index at the time, and All are parameters. , , are all set thresholds. , for Resource occupancy rate of non-safety detection tasks at all times, for Resource usage rate of security detection tasks at all times; (d-2) by formula Calculate the task urgency reward , where and are all positive weights, The safety inspection task is Terminal equipment When executed above, , No. When a security detection task is executed on the edge node, , For the The round trip delay of a safety detection task, , , For a cycle within, no. The amount of data sent by each security detection task, For the The transmission rate of the safety detection task sending process, , For a cycle within, no. The amount of data returned by a security detection task, For the The transmission rate of the return process of a safety detection task, is the reward value when the task is executed at the end node, , To adjust the parameters, For the current The urgency of the security inspection task request, is a positive number greater than 0 and less than 1; (d-3) by formula Calculate the edge node load balancing reward , where for Moment The resource utilization rate of the edge nodes of the platform, , , for Moment The system resources that have been occupied by the edge node are for Moment The total resources of the edge nodes, is the number of edge nodes; (d-4) by formula Calculate the relative security reward , where is the average value of the security metric of the end node itself and its neighboring end nodes, , is the end node security measure, For the Terminal equipment The number of directly connected end devices; (d-5) by formula Calculate the reward of the intelligent agent , where , , , are all weight coefficients.
2. The cloud-edge-device security detection task scheduling method based on multi-agent collaboration according to claim 1 is characterized in that: State parameters in step (a) It includes the status of the node, the status of the task, and the status of the link. The node status includes the resource occupancy rate of the end node, the resource balance index of the end node, the security metric of the end node, and the resource occupancy rate of the edge node; the task status includes the node resources requested by the current task, the amount of task data, and the urgency of the task.
3. The cloud-edge-device security detection task scheduling method based on multi-agent collaboration according to claim 2 is characterized in that: Intelligent Agent Collection Terminal equipment Status parameters The method of the node status is: (a-1) The resource occupancy rate of the end node is given by Resource usage of non-safety detection tasks at all times and Resource usage of security detection tasks at all times Composition, through the formula Calculated Resource usage of non-safety detection tasks at all times , where For the Terminal equipment The end node system resources that are occupied by non-safety detection tasks at all times, For the Terminal equipment The total resources at the moment are calculated by the formula Calculated Resource usage of security detection tasks at all times , where For the Terminal equipment Always perform security detection tasks on the occupied end node system resources. ; (a-2) Through the formula Calculated End node resource balance index at the moment ; (a-3) Through the formula Calculate the end node security metric , where is the balance factor, For the Terminal equipment The safety indicators , and are weight coefficients, For the In the cycle Terminal equipment The duration of the test, For the Terminal equipment In the The number of detection interruptions within a cycle, As parameters, For the Terminal equipment A collection of other directly connected end devices, For the Terminal equipment The number of directly connected end devices, For collection Middle Terminal equipment The safety indicators , For the In the cycle Terminal equipment The duration of the test, For the Terminal equipment In the Number of detection interruptions within a cycle; (a-4) Through the formula Calculated Moment Resource occupancy rate of edge nodes , , is the number of edge nodes, for Moment The system resources that have been occupied by the edge node are for Moment The total resources of the edge nodes, the resource occupancy rate of all edge nodes is , .
4. The cloud-edge-device security detection task scheduling method based on multi-agent collaboration according to claim 3 is characterized in that: Intelligent Agent Collection Terminal equipment Status parameters The status method of the task is: (a-5) The node resources requested by the current task include the current CPU percentage requested by security detection tasks 、Current The percentage of memory requested by security detection tasks ; (a-6) Get the current The amount of data requested by a security detection task ; (a-7) Get the current The urgency of the security inspection task request , A positive number greater than 0 and less than 1.
5. The cloud-edge-device security detection task scheduling method based on multi-agent collaboration according to claim 4 is characterized in that: The state of the link is the link bandwidth , No. Terminal equipment Status parameters .
6. The cloud-edge-device security detection task scheduling method based on multi-agent collaboration according to claim 5 is characterized in that: Step (b) comprises the following steps: (b-1) The policy network consists of the first fully connected layer, the second fully connected layer, the discrete action output layer, and the continuous action output layer; (b-2) Terminal equipment Status parameters Input into the first fully connected layer of the policy network and output the feature ; (b-3) The characteristics Input into the second fully connected layer of the policy network and output the feature ; (b-4) The discrete action output layer of the policy network is composed of a fully connected layer and a softmax function. Input to the fully connected layer of the discrete action output layer, and the output is Dimensional vector , the vector Input into the softmax function of the discrete action output layer, and the output is the probability distribution , using the torch.distributions.Categorical function in the Pytorch library Dimensional probability distribution Randomly sample an element and get the element in Position index in the dimensional vector , , use PyTorch's torch.zeros function to generate a The zero vector of , using the index Set the value of the element at the corresponding index position in the zero vector to 1, and get the vector , For the For the elements, , , ,if , then the safety detection task is scheduled to If the edge nodes execute , then the safety detection task is not scheduled to the If the edge nodes execute , then the safety detection task is in the Terminal equipment Execute on; (b-5) The continuous action output layer of the policy network is composed of a fully connected layer and a sigmoid function. Input the fully connected layer of the continuous action output layer, and output a two-dimensional vector , transform the vector The sigmoid function input to the continuous action output layer outputs a value between 0 and 1. , ,result As a task, get the CPU percentage of the selected edge or end node. The percentage of memory capacity of the selected edge or end node obtained as a task to construct a triple , .
7. The cloud-edge-device security detection task scheduling method based on multi-agent collaboration according to claim 5 is characterized in that: Step (f) comprises the following steps: (f-1). The value network consists of the first fully connected layer, the second fully connected layer, and the third fully connected layer. In the vector of global information and The input is mapped into the first fully connected layer of the value network to obtain a hidden layer, and the output is vector , the vector Input into the second fully connected layer of the value network and output the vector , the vector Input into the third fully connected layer of the value network, and output the Q value of the value network output reinforcement learning ; (f-2) The target network consists of the first fully connected layer, the second fully connected layer, and the third fully connected layer. In the vector of global information middle The state parameters are input into the policy network respectively, and the output is Each security detection task scheduling strategy constitutes a vector , will In the vector of global information and vector The input is mapped to the first fully connected layer of the target network to obtain a hidden layer, and the output is vector , the vector Input into the second fully connected layer of the target network and output the vector , the vector Input into the third fully connected layer of the target network, and output the Q value of reinforcement learning at the next moment .
8. The cloud-edge-device security detection task scheduling method based on multi-agent collaboration according to claim 1 is characterized in that: In step (g), the Q value is used by the time difference algorithm and Q value The value network is updated, and in step (h), the policy network is trained using the policy gradient algorithm.
Citation Information
Patent Citations
Internet of Vehicles calculation unloading method and system based on multi-objective reinforcement learning
CN113961204A
Multi-priority computing unloading strategy optimization method for cross-cloud mobile edge computing
CN114143317A
Cloud edge computing power network system computing unloading method and system
CN117749796A
EBPF enabling task unloading method and eBPF enabling task unloading system for cloud side-end collaborative network
CN117891609A
Cloud and multi-edge network node collaborative micro-service deployment method
CN118337640A
