Multi-stage security detection task dynamic scheduling method and system based on agent

Abstract

The application relates to an intelligent agent-based multi-stage security detection task dynamic scheduling method and system, and belongs to the technical field of network space security; in the process, intelligent agents are deployed in each end device and edge device. The intelligent agent on the end device makes a scheduling decision of a security task by observing the environment in real time. For the edge device, the intelligent agent deployed thereon performs secondary filtering on the security task uploaded to the edge device under the condition of the limited load capacity of the edge device, so that the load capacity of the edge device is utilized to the maximum extent, the delay and rejection rate of the security task are reduced, and the task completion rate is maximized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cyberspace security, and particularly relates to a multi-stage security detection task dynamic scheduling method and system based on an intelligent agent. BACKGROUND

[0002] With the rapid development of network technology, the number of applications that perform computationally intensive tasks, such as image processing, video processing, virtual reality, etc., has increased dramatically. Cloud-edge-end collaboration (EECC) has attracted much attention due to its powerful resource support and high real-time analysis. However, this cloud-edge-end collaboration mode faces challenges in security detection.

[0003] Firstly, terminal devices are usually resource-constrained, which makes it difficult for them to efficiently handle both ordinary tasks and security tasks at the same time. A viable solution is task scheduling. Through task scheduling, part of the security tasks in the terminal device can be transferred to the edge node with more abundant resources for processing. However, traditional task scheduling methods mainly aim to minimize delay and energy consumption, while for security tasks, the timeliness and priority of the tasks also need to be considered. Existing researches mostly focus on how to minimize the total delay of all tasks, but this optimization objective does not fully consider the completion of tasks within the specified deadline. Security tasks have strict requirements for the timeliness of processing, and only taking task delay as the optimization objective may lead to some high-priority or critical security tasks being delayed, thereby potentially posing a security risk to the entire system. In addition, existing researches lack consideration of the sufficiency of edge device computing resources. Even if the terminal device tries to schedule security tasks to the edge device, the edge device may not be able to receive and process these tasks due to high load, further exacerbating the risk of high-priority security tasks not being processed in time. Therefore, how to effectively schedule security tasks between resource-constrained terminal devices and high-load edge devices to ensure the timely processing of high-priority tasks has become a problem to be solved. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the present application provides a multi-stage security detection task dynamic scheduling method based on an intelligent agent; the present application significantly improves the timeliness of security task scheduling, and reduces the security task scheduling delay and rejection rate.

[0005] The present application also provides a multi-stage security detection task dynamic scheduling system based on an intelligent agent.

[0006] The technical scheme of the present application is as follows:

[0007] The multi-stage security detection task dynamic scheduling method based on an intelligent agent comprises:

[0008] 1) training an intelligent agent located at a terminal device; comprising:

[0009] The intelligent agent located at the terminal device obtains current observation;

[0010] The intelligent agent inputs the obtained current observation into the policy network to obtain a security task scheduling policy;

[0011] The intelligent agent executes the security task scheduling policy, so that the current observation is changed, and a reward given by the environment is obtained, and the intelligent agent collects a trajectory of the time step and puts the trajectory into an experience replay buffer of the central controller;

[0012] The central controller performs small-batch sampling from the experience replay buffer, trains a value network using the sampled experience, and feeds back a state value output by the value network to the intelligent agent located at the terminal device;

[0013] The loss function of the policy network is constructed by using the importance sampling mode through the generalized advantage estimation technology, and the policy network is updated;

[0014] 2) Training the intelligent agent located at the edge device; comprising:

[0015] The intelligent agent located at the edge node obtains information in a task queue of the intelligent agent and a running state of the intelligent agent, and generates state parameters;

[0016] The state parameters are input into a policy network of the intelligent agent located at the edge node, the policy network outputs a filtering decision of a task, and a reward of a batch decision is obtained, and meanwhile, the state parameters of the edge node are changed;

[0017] The intelligent agent located at the edge node puts the state parameters, the filtering decision of the task, the reward of the decision and state parameters of a next time step into an experience replay buffer;

[0018] Small-batch random sampling is performed from the experience replay buffer, a value network of the intelligent agent located at the edge node is trained using the sampled experience, and a state value generated by the value network is fed back to the policy network of the intelligent agent;

[0019] The intelligent agent located at the edge node updates the policy network by using the importance sampling mode through the generalized advantage estimation technology according to the state value.

[0020] 3) Each terminal device processes current observation data in real time through a policy network, obtains a scheduling decision of a current security task, and allocates the task to a selected edge node, and the edge node receives the task scheduled from each terminal device, and makes a filtering decision for the task according to a policy network of the edge node.

[0021] According to the preferred embodiment of the present application, the intelligent agent located at the terminal device is trained; comprising:

[0022] a. At each terminal device p i , an intelligent agent is deployed, which obtains the observation o i (t) at time t, i∈{1,2,…,N}, N is the number of terminal devices, 0<t≤T, T is the period;

[0023] b. Each intelligent agent is deployed with a policy network, which receives the observation o i (t) at each time t, and outputs the security task scheduling policy a i (t);

[0024] c. The intelligent agent located at the terminal device p i executes the security task scheduling policy a i (t), and each edge device q m stores the scheduled tasks into its own task list L m according to its own CPU and memory load conditions and executes the tasks, m∈{1,2,...,k}, the intelligent agent located at the terminal device p i collects the updated observation o i (t+1) and the reward r i (t);

[0025] d. The central controller collects the trajectories (S t , A t , R t , S t+1 ) of all intelligent agents at time t, where S t =(o1(t),o2(t),...,o N (t)), A t =(a1(t),a2(t),...,a N (t)), R t =(r1(t),r2(t),...,r N (t)), S t+1 =(o1(t+1),o2(t+1),...,o N (t+1)), and puts the trajectories (S t , A t , R t , S t+1 ) into an experience replay buffer B, and repeats the process (a)-(d) until the length T of a period is reached;

[0026] e. A value network is deployed in the central controller, and K trajectories are randomly sampled from the experience replay buffer B, where one trajectory is (S h , A h , R h , Sh+1 ), 0 < h ≤ T; For S in the trajectory h and S h+1 are respectively input into the value network, and the outputs are the state values V(S h ) and V(S h+1 ) at the corresponding time; Construct a temporal difference loss function and update the value network;

[0027] g. Use the generalized advantage estimation technique to construct the loss function of the policy network by importance sampling and update the policy network;

[0028] h. Repeat steps a to g until a predetermined training period or learning convergence is reached.

[0029] Preferably according to the present invention, train the intelligent agent located at the edge device; including:

[0030] i. Deploy an intelligent agent in each edge device. The intelligent agent includes a policy network. The m-th edge device q m obtains all the security tasks uploaded to itself at time t from its own task list L m ;

[0031] j. The m-th edge device q m calculates the statistical information of the task list L m and generates the state parameters of the edge device q m at time t through the statistical information and the operating state of the edge device q m ;

[0032] k. Use the state parameters of the edge device q m at time t as the input of the policy network to obtain the task filtering policy ;

[0033] l. The m-th edge device q m executes the task filtering policy and then calculates the reward for this execution of the task filtering policy and the new state parameters of the edge device q m ;

[0034] Repeat the above process from (i) to (k) for I times, where 0 < e ≤ I and I is the period; For each time, the intelligent agent in the m-th edge device q m will collect the trajectory of the edge device q m at this time and store it in the buffer B m ;

[0035] In each intelligent agent of the edge devices, a value network is deployed; after a period I, the intelligent agent located at the edge device q m randomly samples G trajectories from the buffer B m where one trajectory is 0 < j ≤ I, and the and in the trajectory are respectively input into the value network, and the outputs are the state values at the corresponding times and and a temporal difference loss function is constructed based on this to update the value network;

[0036] n. Use the generalized advantage estimation technique to construct the loss function of the policy network through importance sampling and update the policy network;

[0037] o. Repeat steps (i) to step (n) until a predetermined training period or learning convergence is reached.

[0038] p. The i-th terminal device p i and the m-th edge device q m execute the security detection task scheduling policy output by their own deployed policy network;

[0039] Preferably according to the present invention, in step a,

[0040] the observation o i (t) is defined as a tuple o i (t) = (D i (t), T i (t), E(t)),

[0041] where D i (t) represents the state of the terminal device p i at the beginning of the time slot t; where D i (t) = (LC i (t), TP i (t), h i (t), cp i (t), m i [[ID=6​​​​​​​​​​​​​​​​i memory occupancy of the edge device at t time;

[0042] T i (t) represents the state of the security task generated in the terminal device at t time; wherein, represents the size of the security task, represents the CPU percentage required by the security task, represents the priority of the security task, represents the deadline of the security task;

[0043] E(t) represents the state of the edge device at t time; E(t) = (SE(t), SC(t), SM(t)), wherein SE(t) represents the set of all edge device computing capabilities, SC(t) represents the set of all edge device CPU occupancy, and SM(t) represents the set of all edge device memory occupancy; wherein,

[0044] According to the present application, preferably, in step b, each intelligent agent is deployed with a policy network, comprising the following steps:

[0045] b-1: the policy network comprises three fully connected layers, first, the policy network receives the observation value o i (t) is input, and the input value o i (t) is converted into a tensor obs_n.

[0046] b-2: input the tensor obs_n into the first fully connected layer o_fc1, and use the Relu activation function after the first fully connected layer o_fc1 to obtain the output obs_n1, take the output obs_n1 as the input of the second fully connected layer o_fc2, and use the Relu activation function after the second fully connected layer o_fc2 to obtain the output obs_n2, take the output obs_n2 as the input of the third fully connected layer o_fc3, and use the Relu activation function after the third fully connected layer o_fc3 to obtain the output obs_n3;

[0047] b-3: input the output obs_n3 into the softmax function to obtain the distribution obs_n4 of each security task scheduling policy;

[0048] b-4: use the argmax function to find the position with the maximum value from the distribution obs_n4, and return the maximum value position index ain; wherein, if ain = 0, it represents that the intelligent agent selects to place the security task generated at the moment in local execution, and if ain = i, wherein i ∈ {1, 2,..., M}, it represents to place the security task generated at the moment in the i-th edge device for execution.

[0049] Further, step c comprises the following steps:

[0050] c-1: the intelligent agent in the terminal device p i executes the security task scheduling decision a i (t), obtains the observation o i (t+1) at the next moment, and obtains SC(t+1) and SM(t+1), SC(t+1) represents the set of CPU occupancy rates of all edge devices at time t+1, and SM(t+1) represents the set of memory occupancy rates of all edge devices at time t+1;

[0051] c-2: calculate the average CPU occupancy rate of all edge devices is the CPU occupancy rate of edge device q i at time t+1; and calculate the standard deviation σ SC (t+1) of the CPU occupancy rate:

[0052] c-3: define the edge device CPU load index L CPU (t+1): wherein Max(SC(t+1)) is the maximum CPU occupancy rate among all edge devices at time t+1;

[0053] c-4: calculate the average memory occupancy rate of all edge devices calculate the standard deviation σ SM (t+1) of the memory occupancy rate:

[0054] c-5: define the edge device memory load index L MeM (t+1) at time t: wherein Max(SM(t+1)) is the maximum memory occupancy rate among all edge devices at time t+1;

[0055] c-6: define the edge device load reward L(t) at time t: L(t) = w1L CPU (t+1) + w2L MeM (t+1), wherein w1 = w2 = 0.5, representing the weight coefficients of the CPU occupancy rate and the memory occupancy rate, respectively;

[0056] c-7: obtain the total number of tasks nt(t) scheduled by all terminal devices at time t, and traverse the task lists L1, L2,..., L m of each edge device at time t to obtain the total number of tasks Vnt(t) received by all edge devices, and calculate the security task rejection rate rej(t):

[0057] c-8: Calculate the reward r for each terminal device at time t. i (t): r i (t)=k1L(t)-k2rej(t); k1 and k2 represent the weight coefficients of equipment load reward and task rejection rate, respectively, with values ​​of k1=k2=0.5.

[0058] Furthermore, step e includes the following steps:

[0059] e-1: The value network consists of two fully connected layers and one GRU unit. The value network is based on S... h For input, input value S h Convert to a tensor sta_n;

[0060] e-2: Input the tensor sta_n into the first fully connected layer s_fc1, and apply the ReLU activation function after the first fully connected layer s_fc1 to obtain the output sta_n1. Use the output sta_n1 as the input to a GRU unit, and apply the ReLU activation function after the GRU unit to obtain the output sta_n2. Use the output sta_n2 as the input to the second fully connected layer s_fc2, and apply the ReLU activation function after the second fully connected layer s_fc2 to obtain the output as the state value V(S). h ).

[0061] e-3: For S h+1 Repeat the process from e-1 to e-2 to obtain V(S) h+1 ).

[0062] Furthermore, step j includes the following steps:

[0063] j-1: Obtain statistical information of the task queue in each edge device, for the m-th edge device q m The statistical information is presented as follows: in, Represents edge device q m Average CPU percentage required for medium-security tasks Represents edge device q m The average percentage of memory required for medium-security tasks. Indicates the average deadline for security tasks. Represents edge device q m The average priority of each security task in the process;

[0064] j-2: Get the m-th edge device q m The operational status, including the m-th edge device q m Remaining CPU percentage and remaining memory percentage

[0065] j-3: edge device q at time t m The state parameter of the edge device q at time t is denoted as:

[0066] Further, step k comprises the following steps:

[0067] k-1: the policy network comprises three fully connected layers, which convert the input into a tensor sta_m;

[0068] k-2: input the tensor sta_m into the first fully connected layer sq_fc1, and use the Relu activation function after the first fully connected layer sq_fc1 to obtain the output sta_m1, take the output sta_m1 as the input of the second fully connected layer sq_fc2, and use the Relu activation function after the second fully connected layer sq_fc2 to obtain the output sta_m2, take the output sta_m2 as the input of the third fully connected layer sq_fc3, and use the Relu activation function after the third fully connected layer sq_fc3 to obtain the output sta_m3;

[0069] k-3: input the output sta_m3 into the softmax function to obtain the distribution sta_m4 of the task filtering policy;

[0070] k-4: use the argmax function to find the position with the maximum value from the distribution sta_m4, and return the maximum value position index qain;

[0071] k-5: according to the index value qain, select the task filtering policy to be executed.

[0072] Further, in step k-5, the task filtering policy is divided into three types:

[0073] If qain=0, the edge device q m sorts the tasks in its task queue L m according to the task priority from large to small, and executes the tasks in the order after sorting;

[0074] If qain=1, the edge device sorts the tasks in its task queue L m according to the task deadline from small to large, and executes the tasks in the order after sorting;

[0075] If qain=2, the edge device sorts the tasks in its task queue L m according to the ratio of the task deadline to the priority from small to large, and executes the tasks in the order after sorting.

[0076] Further, step m comprises the following steps:

[0077] m-1: The value network consists of two fully connected layers and one GRU unit. First, the value network... For input, input value Convert it to a tensor sta_q.

[0078] m-2: Input the tensor sta_q into the first fully connected layer smq_fc1, and apply the ReLU activation function after this layer to obtain the output sta_q1. Use this output as the input to a GRU unit, and apply the ReLU activation function after this layer to obtain the output sta_q2. Use this output as the input to the second fully connected layer smq_fc2, and apply the ReLU activation function after this layer to obtain the output as the state value.

[0079] m-3: for Repeat the process from m-1 to m-2 to obtain

[0080] Furthermore, step 1 includes the following steps:

[0081] l-1: Calculate the reward for the task filtering strategy, as follows:

[0082]

[0083] in, For edge devices q m Task queue L m The latency of the i-th security task. Let Y represent the deadline for the i-th security task, Y represent the number of security tasks completed at time t, and φ1 and φ2 are weighting coefficients used to adjust the importance of different parts of the reward function.

[0084] A multi-stage security detection task dynamic scheduling system based on intelligent agents includes:

[0085] The intelligent agent training module of the terminal device is configured to train the intelligent agent located on the terminal device; including:

[0086] The intelligent agent located on the terminal device acquires the current observation;

[0087] The intelligent agent will obtain a safe task scheduling policy from the current observation input policy network;

[0088] The intelligent agent executes a safe task scheduling strategy, which changes the current observation and obtains a reward from the environment. The intelligent agent collects the trajectory of this time step and puts it into the experience playback buffer of the central controller.

[0089] The central controller samples in small batches from the experience replay buffer, trains the value network using the sampled experience, the value network outputs state value, and feeds back to the intelligent agent located at the terminal device;

[0090] The loss function of the policy network is constructed by importance sampling mode using generalized advantage estimation technology, and the policy network is updated;

[0091] The intelligent agent training module of the edge device is configured to train the intelligent agent located at the edge device, and comprises:

[0092] The intelligent agent located at the edge node obtains information in the task queue and the running state of itself, and generates state parameters;

[0093] The state parameters are input into the policy network of the intelligent agent of the edge node, the policy network outputs the filtering decision of the task, and the reward of the batch decision is obtained, and the state parameters of the edge node itself are changed;

[0094] The intelligent agent located at the edge node puts the state parameters of itself, the filtering decision of the task, the reward of the decision and the state parameters of the next time step into the experience replay buffer;

[0095] Small batch random sampling is performed from the experience replay buffer, the value network of the edge node intelligent agent is trained using the sampled experience, and the value network generates state value and feeds back to the policy network of the intelligent agent;

[0096] The intelligent agent located at the edge node uses generalized advantage estimation technology to construct the loss function of the policy network by importance sampling mode according to the state value, and updates the policy network;

[0097] The dynamic scheduling module is configured to: each terminal device processes the current observation data in real time through the policy network, obtains the scheduling decision of the current security task, and distributes the task to the selected edge node, and the edge node receives the task scheduled from each terminal device, and makes filtering decision according to the policy network of itself.

[0098] The beneficial effects of the present application are:

[0099] The application provides a security task scheduling method for an EECC distributed architecture. Specifically, a multi-stage security detection task dynamic scheduling method based on an intelligent agent is designed. In the process, an intelligent agent is deployed in each terminal device and edge device. The intelligent agent on the terminal device makes a scheduling decision of the security task by observing the environment in real time. For the edge device, the intelligent agent deployed thereon filters the security task uploaded to the edge device again, so as to maximize the load capacity of the edge device, reduce the delay and rejection rate of the security task, and maximize the task completion rate. BRIEF DESCRIPTION OF DRAWINGS

[0100] Figure 1 A training flowchart of the intelligent agent located at the terminal device of the application;

[0101] Figure 2 A training flowchart of the intelligent agent located at the edge device of the application;

[0102] Figure 3 A distributed system architecture diagram of the application. DETAILED DESCRIPTION

[0103] The application is further limited by the description of the accompanying drawings and examples, but is not limited thereto.

[0104] Example 1

[0105] The multi-stage security detection task dynamic scheduling method based on the intelligent agent comprises:

[0106] 1) training the intelligent agent located at the terminal device, comprising:

[0107] The intelligent agent located at the terminal device obtains the current observation;

[0108] The intelligent agent inputs the obtained current observation into a policy network to obtain a security task scheduling policy;

[0109] The intelligent agent executes the security task scheduling policy, so that the current observation changes, and at the same time, the intelligent agent obtains a reward given by the environment, collects the trajectory of the time step, and puts it into an experience replay buffer of a central controller;

[0110] The central controller samples in small batches from the experience replay buffer, trains a value network using the sampled experience, the value network outputs a state value, and feeds back to the intelligent agent located at the terminal device;

[0111] The generalized advantage estimation technique is used to construct a loss function of the policy network through importance sampling, and the policy network is updated;

[0112] 2) Train the intelligent agent located at the edge device; including:

[0113] The intelligent agent located at the edge node obtains the information in its own task queue and its own running state, and generates state parameters;

[0114] The state parameters are input into the policy network of the edge node intelligent agent, the policy network outputs the filtering decision of the task, and the reward of the batch decision is obtained, and at the same time the state parameters of the edge node itself change;

[0115] The intelligent agent located at the edge node puts the state parameters of itself, the filtering decision of the task, the reward of the decision and the state parameters of the next time step into the experience replay buffer;

[0116] Small batch random sampling is performed from the experience replay buffer, and the value network of the edge node intelligent agent is trained using the sampled experience, and the value network generates state value feedback to the policy network of the intelligent agent;

[0117] The intelligent agent located at the edge node uses the generalized advantage estimation technique to construct the loss function of the policy network according to the state value through importance sampling, and updates the policy network.

[0118] 3) Each terminal device processes the current observation data in real time through the policy network, obtains the scheduling decision of the current security task, and assigns the task to the selected edge node, and the edge node receives the task scheduled from each terminal device, and makes filtering decision according to its own policy network.

[0119] Embodiment 2

[0120] The multi-stage security detection task dynamic scheduling method based on agent according to embodiment 1, the difference lies in:

[0121] Train the intelligent agent located at the terminal device; as Figure 1 shown, including:

[0122] a. Deploy an intelligent agent on each terminal device p i , which can make security task scheduling decisions according to the current observation, and the intelligent agent obtains the observation o i (t) at time t, i∈{1,2,…,N}, N is the number of terminal devices in the network, 0<t≤T, T is the period; The network refers to a distributed system architecture composed of terminal devices, edge devices, central controllers, experience replay buffers and other components, which is used to realize multi-stage security detection task dynamic scheduling based on agent. As Figure 3 shown.

[0123] b. Each intelligent agent is deployed with a policy network, which receives observation o i (t) at each time t and outputs a security task scheduling policy a i (t).

[0124] c. The intelligent agent located at terminal device p i executes the security task scheduling policy a i (t) and each edge device q m stores the scheduled task in its own task list L m according to its CPU and memory load and executes the task, m∈{1,2,...,k}, the intelligent agent located at terminal device p i collects updated observation o i (t+1) and reward r i (t).

[0125] d. The central controller collects the trajectory (S t , A t , R t , S t+1 ) of all intelligent agents at time t, where S t =(o1(t),o2(t),...,o N (t)), A t =(a1(t),a2(t),...,a N (t)), R t =(r1(t),r2(t),...,r N (t)), S t+1 =(o1(t+1),o2(t+1),...,o N (t+1)), and puts the trajectory (S t , A t , R t , S t+1 ) into the experience replay buffer B, and repeats the process (a)-(d) until the length T of a period is reached.

[0126] e. A value network is deployed in the central controller, and K trajectories are randomly sampled from the experience replay buffer B, where one trajectory is (S h , A h , R h , S h+1 ), 0<h≤T; S h and S h+1 in the trajectory are input into the value network, and the output is the state value V(S h ) and V(S h+1 ) at the corresponding time; a time difference loss function is constructed, and the value network is updated.

[0127] g. Use the generalized advantage estimation technique to construct the loss function of the policy network by importance sampling and update the policy network;

[0128] h. Repeat steps a to g until a predetermined training period or learning convergence is reached.

[0129] Train the intelligent agent located at the edge device; as Figure 2 shown, including:

[0130] i. Deploy an intelligent agent in each edge device. The intelligent agent includes a policy network. The m-th edge device q m obtains all the security tasks uploaded to itself at time t from its own task list L m ;

[0131] j. The m-th edge device q m calculates the statistical information of the task list L m and generates the state parameters of the edge device q m at time t through the statistical information and the operating state of the edge device q m ;

[0132] k. Use the state parameters of the edge device q m at time t as the input of the policy network to obtain the task filtering policy ;

[0133] l. The m-th edge device q m executes the task filtering policy and then calculates the reward for this execution of the task filtering policy and the new state parameters of the edge device q m ;

[0134] Repeat the above process from (i) to (k) for I times, where 0 < e ≤ I and I is the period; for each moment, the intelligent agent in the m-th edge device q m will collect the trajectory of the edge device q m at this moment and store it in the buffer B m ;

[0135] m. Deploy a value network in the intelligent agent of each edge device; after one period I, the intelligent agent located at the edge device q m randomly samples G trajectories from the buffer B m , where one trajectory is 0 < j ≤ I, and for the and The inputs are given to the value network, and the output is the state value at the corresponding time step. as well as Based on this, a temporal difference loss function is constructed to update the value network;

[0136] n. Using generalized advantage estimation, construct the loss function of the policy network through importance sampling, and update the policy network.

[0137] o. Repeat steps (i) to (n) until the predetermined training period is reached or the learning converges.

[0138] p, the i-th terminal device p i and the mth edge device q m It executes the security detection task scheduling strategy output by the policy network it has deployed;

[0139] In step a,

[0140] Observation o i (t) is defined as a tuple o i (t)=(D i (t),T i (t),E(t)),

[0141] Among them, D i (t) represents the terminal device p at the beginning of time slot t. i The state; where D i (t)=(LC i (t),TP i (t),h i (t),cp i (t),m i (t)); LC i (t) represents the current terminal device p i Computing power, measured in FLOPS and TP. i (t) represents the current terminal device p i Transmission power, measured in dBm; h i (t) represents the link gain; cp i (t) represents the current terminal device p i CPU utilization; m i (t) represents the current terminal device p i Memory usage;

[0142] T i (t) represents the state of the security task generated in the terminal device at time t; in, Indicates the size of the security task. This indicates the percentage of CPU required for security tasks. Indicates the priority of security tasks. Indicates the deadline for the security task;

[0143] E(t) represents the state of the edge device at time t; E(t) = (SE(t), SC(t), SM(t)), where SE(t) represents the set of computing power of all edge devices, SC(t) represents the set of CPU utilization of all edge devices, and SM(t) represents the set of memory utilization of all edge devices;

[0144] In step b, a policy network is deployed on each smart agent, including the following steps:

[0145] b-1: The policy network consists of three fully connected layers. First, the policy network receives observations o. i (t) is the input, and the input value is converted into o using the torch.tensor method in PyTorch. i (t) is transformed into a tensor obs_n. The torch.tensor method in PyTorch refers to the fact that PyTorch is an open-source machine learning library based on Python, primarily used in applications such as computer vision and natural language processing in artificial intelligence. The torch.tensor method is used to create a tensor, a fundamental data structure in PyTorch that can be used on GPUs to accelerate computation. In this invention, the torch.tensor method is used to transform input data (such as observations, state parameters, etc.) into tensors so that they can be input into neural networks (policy networks, value networks, etc.) for computation.

[0146] b-2: Input the tensor obs_n into the first fully connected layer o_fc1, and apply the ReLU activation function after the first fully connected layer o_fc1 to obtain the output obs_n1. Use the output obs_n1 as the input of the second fully connected layer o_fc2, and apply the ReLU activation function after the second fully connected layer o_fc2 to obtain the output obs_n2. Use the output obs_n2 as the input of the third fully connected layer o_fc3, and apply the ReLU activation function after the third fully connected layer o_fc3 to obtain the output obs_n3.

[0147] b-3: Input the output obs_n3 into the softmax function to obtain the distribution obs_n4 for each security task scheduling policy; obs_n4 refers to the probability distribution output by the policy network. This distribution describes the probability that the intelligent agent will choose different scheduling actions (such as executing the task locally or scheduling it to different edge devices) under the current observation state. The intelligent agent forms the security task scheduling policy by selecting the action with the highest probability, which is the returned index value ain.

[0148] b-4: Use the argmax function to find the position of the maximum value in the distribution obs_n4 and return the index ain of the maximum value position; where ain = 0, it means that the intelligent agent chooses to place the security task generated at this moment to be executed locally, and ain = i, where i∈{1,2,....,M}, it means that the security task generated at this moment is placed to be executed on the i-th edge device. Argmax is a mathematical function used to find the parameter value when a given function reaches its maximum value. In machine learning and deep learning, the argmax function is often used to select the category or action with the highest probability or score. In this invention, the argmax function is used to select the security task scheduling policy or task filtering policy with the highest probability from the probability distribution output by the policy network, that is, to select the best action that the intelligent agent should take in the current state.

[0149] Step c includes the following steps:

[0150] c-1: Located in terminal device p i The intelligent agent in the process performs security task scheduling decisions. i (t), to obtain the observation at the next time step o i (t+1), and obtain SC(t+1) and SM(t+1), where SC(t+1) represents the set of CPU utilization rates of all edge devices at time t+1, and SM(t+1) represents the set of memory utilization rates of all edge devices at time t+1;

[0151] c-2: Calculate the average CPU utilization of all edge devices. This refers to the edge device q at time t+1. i The CPU utilization rate; and the standard deviation σ of the CPU utilization rate. SC (t+1):

[0152] c-3: Define the edge device CPU load metric L CPU (t+1): Where Max(SC(t+1)) is the maximum CPU utilization rate among all edge devices at time t+1;

[0153] c-4: Calculate the average memory usage of all edge devices Calculate the standard deviation σ of memory usage. SM (t+1):

[0154] c-5: Define the edge device memory load metric L at time t. MeM (t+1): Where Max(SM(t+1)) is the maximum memory usage rate among all edge devices at time t+1;

[0155] c-6: Define the edge device load reward L(t) at time t: L(t) = w1L CPU (t+1)+w2L MeM (t+1), where w1=w2=0.5, representing the weighting coefficients of CPU utilization and memory utilization, respectively;

[0156] c-7: Obtain the total number of tasks scheduled by all terminal devices at time t, nt(t), and iterate through the task lists L1, L2, ..., L of each edge device at time t. m Obtain the total number of tasks received by all edge devices, Vnt(t), and calculate the security task rejection rate, rej(t):

[0157] c-8: Calculate the reward r for each terminal device at time t. i (t): r i (t)=k1L(t)-k2rej(t); k1 and k2 represent the weight coefficients of equipment load reward and task rejection rate, respectively, with values ​​of k1=k2=0.5.

[0158] Step e includes the following steps:

[0159] e-1: The value network consists of two fully connected layers and one GRU unit. The value network is based on S... h As input, the input value S is processed using the torch.tensor method in PyTorch. h Convert to a tensor sta_n;

[0160] e-2: Input the tensor sta_n into the first fully connected layer s_fc1, and apply the ReLU activation function after the first fully connected layer s_fc1 to obtain the output sta_n1. Use the output sta_n1 as the input to a GRU unit, and apply the ReLU activation function after the GRU unit to obtain the output sta_n2. Use the output sta_n2 as the input to the second fully connected layer s_fc2, and apply the ReLU activation function after the second fully connected layer s_fc2 to obtain the output as the state value V(S). h ).

[0161] e-3: For S h+1 Repeat the process from e-1 to e-2 to obtain V(S) h+1 ).

[0162] Step j includes the following steps:

[0163] j-1: Obtain statistical information of the task queue in each edge device, for the m-th edge device q m The statistical information is presented as follows: in, Represents edge device q m Average CPU percentage required for medium-security tasks Represents edge device q m The average percentage of memory required for medium-security tasks. Indicates the average deadline for security tasks. Represents edge device q m The average priority of each security task in the process;

[0164] j-2: Get the m-th edge device q m The operational status, including the m-th edge device q m Remaining CPU percentage and remaining memory percentage

[0165] j-3: Edge device q at time t m The state parameters are represented as follows:

[0166] Step k includes the following steps:

[0167] k-1: The policy network consists of three fully connected layers, using PyTorch's `torch.tensor` method to process the input... Convert to tensor sta_m;

[0168] k-2: Input the tensor sta_m into the first fully connected layer sq_fc1, and apply the ReLU activation function after the first fully connected layer sq_fc1 to obtain the output sta_m1. Use the output sta_m1 as the input of the second fully connected layer sq_fc2, and apply the ReLU activation function after the second fully connected layer sq_fc2 to obtain the output sta_m2. Use the output sta_m2 as the input of the third fully connected layer sq_fc3, and apply the ReLU activation function after the third fully connected layer sq_fc3 to obtain the output sta_m3.

[0169] k-3: Input the output sta_m3 into the softmax function to obtain the distribution sta_m4 of the task filtering policy; sta_m4 refers to the probability distribution output by the policy network. This distribution describes the probability that the intelligent agent will choose different task filtering policies under the current state parameters. The intelligent agent forms the task filtering policy by selecting the task filtering policy with the highest probability, which is the returned index value qain.

[0170] k-4: Use the argmax function to find the position of the maximum value in the distribution sta_m4 and return the index qain of the maximum value position;

[0171] k-5: Select the task filtering strategy to be executed based on the index value qain.

[0172] In step k-5, the task filtering strategy is divided into three types:

[0173] If qain = 0, then the edge device q m Set its task queue L m Tasks are sorted by priority from highest to lowest and executed in that order. Task priority is an indicator of the urgency and importance of a task. Task priority is assessed based on the operator's preset rules.

[0174] If qain = 1, then the edge device will set its task queue L. m The tasks are sorted in ascending order of their deadlines and executed in that order.

[0175] If qain = 2, then the edge device will move its task queue L m The tasks are sorted in ascending order of the ratio of task deadline to priority, and then executed in that sorted order.

[0176] Step m includes the following steps:

[0177] m-1: The value network consists of two fully connected layers and one GRU unit. First, the value network... As input, the input values ​​are processed using the `torch.tensor` method in PyTorch. Convert it to a tensor sta_q.

[0178] m-2: Input the tensor sta_q into the first fully connected layer smq_fc1, and apply the ReLU activation function after this layer to obtain the output sta_q1. Use this output as the input to a GRU unit, and apply the ReLU activation function after this layer to obtain the output sta_q2. Use this output as the input to the second fully connected layer smq_fc2, and apply the ReLU activation function after this layer to obtain the output as the state value.

[0179] m-3: for Repeat the process from m-1 to m-2 to obtain

[0180] Furthermore, step 1 includes the following steps:

[0181] l-1: Calculate the reward for the task filtering strategy, as follows:

[0182]

[0183] in, For edge devices q m Task queue L m The latency of the i-th security task. Let Y represent the deadline for the i-th security task, Y represent the number of security tasks completed at time t, and φ1 and φ2 are weighting coefficients used to adjust the importance of different parts of the reward function.

[0184] Example 3

[0185] A multi-stage security detection task dynamic scheduling system based on intelligent agents includes:

[0186] The intelligent agent training module of the terminal device is configured to train the intelligent agent located on the terminal device; including:

[0187] The intelligent agent located on the terminal device acquires the current observation;

[0188] The intelligent agent will obtain a safe task scheduling policy from the current observation input policy network;

[0189] The intelligent agent executes a safe task scheduling strategy, which changes the current observation and obtains a reward from the environment. The intelligent agent collects the trajectory of this time step and puts it into the experience playback buffer of the central controller.

[0190] The central controller performs small-batch sampling from the experience replay buffer, uses the sampled experience to train the value network, the value network outputs state values, and feeds them back to the intelligent agent located on the terminal device.

[0191] Using generalized advantage estimation, a loss function for the policy network is constructed through importance sampling, and the policy network is then updated.

[0192] The edge device intelligent agent training module is configured to train an intelligent agent located on the edge device; including:

[0193] The intelligent agent located at the edge node obtains information from its own task queue and its own running status, and generates status parameters;

[0194] The state parameters are input into the policy network of the edge node's intelligent agent. The policy network outputs the filtering decision for the task and receives the reward for the batch decision. At the same time, the state parameters of the edge node itself change.

[0195] The intelligent agent located at the edge node puts its own state parameters, task filtering decisions, decision rewards, and state parameters for the next time step into the experience replay buffer;

[0196] Small-batch random sampling is performed from the experience replay buffer, and the sampled experience is used to train the value network of the edge node intelligent agent. The value network generates state value feedback to the policy network of the intelligent agent.

[0197] The intelligent agent located at the edge node constructs the loss function of the policy network based on the state value using the generalized advantage estimation technique and the importance sampling method, and updates the policy network accordingly.

[0198] The dynamic scheduling module is configured such that each terminal device processes the current observation data in real time through the policy network, obtains the scheduling decision of the current security task, and assigns the task to the selected edge node. After receiving the task scheduled by each terminal device, the edge node makes filtering decisions on these tasks according to its own policy network.

[0199] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic scheduling method for multi-stage security detection tasks based on intelligent agents, characterized in that, include: 1) Training the intelligent agent located on the terminal device; including: The intelligent agent located on the terminal device acquires the current observation; The intelligent agent will obtain a safe task scheduling policy from the current observation input policy network; The intelligent agent executes a safety task scheduling strategy, which changes the current observation and obtains rewards from the environment. The intelligent agent collects the trajectory and puts it into the experience playback buffer of the central controller. The central controller performs small-batch sampling from the experience replay buffer, uses the sampled experience to train the value network, the value network outputs state values, and feeds them back to the intelligent agent located on the terminal device. Using generalized advantage estimation, a loss function for the policy network is constructed through importance sampling, and the policy network is then updated. 2) Training intelligent agents located on edge devices; including: The intelligent agent located at the edge node obtains information from its own task queue and its own running status, and generates status parameters; The state parameters are input into the policy network of the edge node's intelligent agent. The policy network outputs the filtering decision for the task and receives the reward for the batch decision. At the same time, the state parameters of the edge node itself change. The intelligent agent located at the edge node puts its own state parameters, task filtering decisions, decision rewards, and state parameters for the next time step into the experience replay buffer; Small-batch random sampling is performed from the experience replay buffer, and the sampled experience is used to train the value network of the edge node intelligent agent. The value network generates state value feedback to the policy network of the intelligent agent. The intelligent agent located at the edge node constructs the loss function of the policy network based on the state value using the generalized advantage estimation technique and the importance sampling method, and updates the policy network accordingly. 3) Each terminal device processes the current observation data in real time through the policy network to obtain the scheduling decision of the current security task and assigns the task to the selected edge node. After receiving the task scheduled by each terminal device, the edge node makes filtering decisions on these tasks according to its own policy network.

2. The agent-based multi-stage security detection task dynamic scheduling method according to claim 1, characterized in that, Training the intelligent agent located on the terminal device; including: a. On each terminal device p i deploy an intelligent agent, and the intelligent agent obtains the observation o i (t), i ∈ {1, 2, …, N}, where N is the number of terminal devices, 0 < t ≤ T, and T is the period; b. A policy network is deployed on each smart agent. At each time t, the policy network receives observations o. i (t), output the safe task scheduling strategy a i (t); c. Located in terminal device p i Intelligent agent execution security task scheduling strategy a i (t), each edge device q m Based on its own CPU and memory load, the scheduled tasks are stored in its own task list L. m The task is executed concurrently, m∈{1,2,...,k}, located on terminal device p. i The intelligent agent collects updated observations. i (t+1) and reward r i (t); d. The central controller collects the trajectories (S) of all intelligent agents within time t. t A t ,R t ,S t+1 ), where S t =(o1(t),o2(t),...,o N (t)), A t =(a1(t),a2(t),...,a N (t)), R t =(r1(t),r2(t),...,r N (t)), S t+1 =(o1(t+1),o2(t+1),...,o N (t+1)), and the trajectory (S) t A t ,R t ,S t+1 Place it into the experience playback buffer B, and repeat the process (a)-(d) until the duration T of one cycle is reached; e. A value network is deployed in the central controller, randomly sampling K trajectories from the experience replay buffer B, where one trajectory is (S h , A h , R h , S h+1 ), 0 < h ≤ T; input S h and S h+1 in the trajectory into the value network respectively, and the outputs are the state values V(S h ) and V(S h+1 ) at the corresponding times; construct a temporal difference loss function and update the value network; g. Using generalized advantage estimation, construct the loss function of the policy network through importance sampling and update the policy network; h. Repeat steps a through g until the predetermined training period is reached or the learning converges.

3. The agent-based multi-stage security detection task dynamic scheduling method according to claim 1, characterized in that, Training intelligent agents located on edge devices; including: i. Deploy a smart agent in each edge device. The smart agent includes a policy network, and the m-th edge device q m From its own task list L m Retrieve all security tasks uploaded to itself at time t; j, the mth edge device q m Calculate the task list L m Statistical information, and through this statistical information, edge device q m The running status of the edge device q at time t is generated. m State parameters k, the edge device q at time t m State parameters As input to the policy network, the task filtering policy is obtained. l, the mth edge device q m Execute task filtering strategy The reward for this task filtering strategy is then calculated. and edge devices q m New state parameters Repeat the above process (i) to (k) I times. For each time step, the m-th edge device q m The intelligent agent in the middle will collect the edge device q at that moment m trajectory Store in buffer B m middle; In the intelligent agents of each edge device, a value network is deployed; after a cycle I, the intelligent agent located at edge device q m randomly samples G trajectories from buffer B m where one trajectory is 0 < j ≤ I, and the and in the trajectory are respectively input into the value network, and the outputs are the state values at the corresponding times and and a temporal difference loss function is constructed based on this to update the value network; n. Using generalized advantage estimation, construct the loss function of the policy network through importance sampling, and update the policy network. o. Repeat steps (i) to (n) until the predetermined training period is reached or the learning converges. p, the i-th terminal device p i and the mth edge device q m It executes the security detection task scheduling strategy output by the policy network it has deployed.

4. The agent-based multi-stage security detection task dynamic scheduling method according to claim 2, characterized in that, In step a, observe o i (t) is defined as a tuple o i (t)=(D i (t),T i (t),E(t)), where D i (t) represents the terminal device p at the beginning of time slot t. i State; D i (t)=(LC i (t),TP i (t),h i (t),cp i (t),m i (t)); LC i (t) represents the current terminal device p i Computing resources, in GHz / s, TP i (t) represents the current terminal device p i The computing power, measured in FLOPS; h i (t) represents the link gain; cp i (t) represents the current terminal device p i CPU utilization; m i (t) represents the current terminal device p i Memory usage; T i (t) represents the state of the security task generated in the terminal device at time t; in, Indicates the size of the security task. This indicates the percentage of CPU required for security tasks. Indicates the priority of security tasks. Indicates the deadline for the security task; E(t) represents the state of the edge device at time t; E(t) = (SE(t), SC(t), SM(t)), where SE(t) represents the set of computing power of all edge devices, SC(t) represents the set of CPU utilization of all edge devices, and SM(t) represents the set of memory utilization of all edge devices; 5. The agent-based multi-stage security detection task dynamic scheduling method according to claim 2, characterized in that, In step b, a policy network is deployed on each smart agent, including the following steps: b-1: The policy network consists of three fully connected layers. First, the policy network receives observations o. i (t) is the input, and the input value o i (t) is transformed into the tensor obs_n; b-2: Input the tensor obs_n into the first fully connected layer o_fc1, and apply the ReLU activation function after the first fully connected layer o_fc1 to obtain the output obs_n1. Use the output obs_n1 as the input of the second fully connected layer o_fc2, and apply the ReLU activation function after the second fully connected layer o_fc2 to obtain the output obs_n2. Use the output obs_n2 as the input of the third fully connected layer o_fc3, and apply the ReLU activation function after the third fully connected layer o_fc3 to obtain the output obs_n3. b-3: Input the output obs_n3 into the softmax function to obtain the distribution obs_n4 of each safe task scheduling policy; b-4: Use the argmax function to find the position of the largest value in the distribution obs_n4 and return the index of the maximum value position ain; where, if ain = 0, it means that the smart agent chooses to place the security task generated at this moment to be executed locally, and if ain = i, where i∈{1,2,....,M}, it means that the security task generated at this moment is placed to be executed on the i-th edge device.

6. The agent-based multi-stage security detection task dynamic scheduling method according to claim 2, characterized in that, Step c includes the following steps: c-1: Located in terminal device p i The intelligent agent in the process performs security task scheduling decisions. i (t), to obtain the observation at the next time step o i (t+1), and obtain SC(t+1) and SM(t+1), where SC(t+1) represents the set of CPU utilization rates of all edge devices at time t+1, and SM(t+1) represents the set of memory utilization rates of all edge devices at time t+1; c-2: Calculate the average CPU utilization of all edge devices. This refers to the edge device q at time t+1. i The CPU utilization rate; and the standard deviation σ of the CPU utilization rate. SC (t+1): c-3: Define the edge device CPU load metric L CPU (t+1): Where Max(SC(t+1)) is the maximum CPU utilization rate among all edge devices at time t+1; c-4: Calculate the average memory usage of all edge devices : Calculate the standard deviation σ of memory usage. SM (t+1): c-5: Define the edge device memory load metric L at time t. MeM (t+1): Where Max(SM(t+1)) is the maximum memory usage rate among all edge devices at time t+1; c-6: Define the edge device load reward L(t) at time t: L(t) = w1L CPU (t+1)+w2L MeM (t+1), where w1=w2=0.5, representing the weighting coefficients of CPU utilization and memory utilization, respectively; c-7: Obtain the total number of tasks scheduled by all terminal devices at time t, nt(t), and iterate through the task lists L1, L2, ..., L of each edge device at time t. m Obtain the total number of tasks received by all edge devices, Vnt(t), and calculate the security task rejection rate, rej(t): c-8: Calculate the reward r for each terminal device at time t. i (t): r i (t)=k1L(t)-k2rej(t); k1 and k2 represent the weight coefficients of equipment load reward and task rejection rate, respectively, with values ​​of k1=k2=0.

5.

7. The agent-based multi-stage security detection task dynamic scheduling method according to claim 2, characterized in that, Step e includes the following steps: e-1: The value network consists of two fully connected layers and one GRU unit. The value network is based on S... h For input, input value S h Convert to a tensor sta_n; e-2: Input the tensor sta_n into the first fully connected layer s_fc1, and apply the ReLU activation function after the first fully connected layer s_fc1 to obtain the output sta_n1. Use the output sta_n1 as the input to a GRU unit, and apply the ReLU activation function after the GRU unit to obtain the output sta_n2. Use the output sta_n2 as the input to the second fully connected layer s_fc2, and apply the ReLU activation function after the second fully connected layer s_fc2 to obtain the output as the state value V(S). h ); e-3: For S h+1 Repeat the process from e-1 to e-2 to obtain V(S) h+1 ).

8. The agent-based multi-stage security detection task dynamic scheduling method according to claim 3, characterized in that, Step j includes the following steps: j-1: Obtain statistical information of the task queue in each edge device, for the m-th edge device q m The statistical information is presented as follows: in, Represents edge device q m Average CPU percentage required for medium-security tasks Represents edge device q m The average percentage of memory required for medium-security tasks. Indicates the average deadline for security tasks. Represents edge device q m The average priority of each security task in the process; j-2: Get the m-th edge device q m The operational status, including the m-th edge device q m Remaining CPU percentage and remaining memory percentage j-3: Edge device q at time t m The state parameters are represented as follows:

9. The agent-based multi-stage security detection task dynamic scheduling method according to claim 3, characterized in that, Step k includes the following steps: k-1: The policy network consists of three fully connected layers, which will receive the input... Convert to tensor sta_m; k-2: Input the tensor sta_m into the first fully connected layer sq_fc1, and apply the ReLU activation function after the first fully connected layer sq_fc1 to obtain the output sta_m1. Use the output sta_m1 as the input of the second fully connected layer sq_fc2, and apply the ReLU activation function after the second fully connected layer sq_fc2 to obtain the output sta_m2. Use the output sta_m2 as the input of the third fully connected layer sq_fc3, and apply the ReLU activation function after the third fully connected layer sq_fc3 to obtain the output sta_m3. k-3: Input the output sta_m3 into the softmax function to obtain the distribution sta_m4 of the task filtering policy; k-4: Use the argmax function to find the position of the maximum value in the distribution sta_m4 and return the index qain of the maximum value position; k-5: Select the task filtering strategy to be executed based on the index value qain; In step k-5, the task filtering strategy is divided into three types: If qain = 0, then the edge device q m Set its task queue L m The tasks in the system are sorted from highest to lowest priority and executed in that order. If qain = 1, then the edge device will set its task queue L. m The tasks are sorted in ascending order of their deadlines and executed in that order. If qain = 2, then the edge device will move its task queue L m The tasks are sorted in ascending order of the ratio of task deadline to priority, and then executed in that sorted order. Step m includes the following steps: m-1: The value network consists of two fully connected layers and one GRU unit. First, the value network... As input, the input values ​​are processed using the `torch.tensor` method in PyTorch. Convert to a tensor sta_q; m-2: Input the tensor sta_q into the first fully connected layer smq_fc1, and apply the ReLU activation function after this layer to obtain the output sta_q1. Use this output as the input to a GRU unit, and apply the ReLU activation function after this layer to obtain the output sta_q2. Use this output as the input to the second fully connected layer smq_fc2, and apply the ReLU activation function after this layer to obtain the output as the state value. m-3: for Repeat the process from m-1 to m-2 to obtain Step 1 includes the following steps: l-1: Calculate the reward for the task filtering strategy, as follows: in, For edge devices q m Task queue L m The latency of the i-th security task. Let Y represent the deadline for the i-th security task, Y represent the number of security tasks completed at time t, and φ1 and φ2 are weighting coefficients used to adjust the importance of different parts of the reward function.

10. A multi-stage security detection task dynamic scheduling system based on intelligent agents, characterized in that, include: The intelligent agent training module of the terminal device is configured to train the intelligent agent located on the terminal device; including: The intelligent agent located on the terminal device acquires the current observation; The intelligent agent will obtain a safe task scheduling policy from the current observation input policy network; The intelligent agent executes a safety task scheduling strategy, which changes the current observation and obtains rewards from the environment. The intelligent agent collects the trajectory and puts it into the experience playback buffer of the central controller. The central controller performs small-batch sampling from the experience replay buffer, uses the sampled experience to train the value network, the value network outputs state values, and feeds them back to the intelligent agent located on the terminal device. Using generalized advantage estimation, a loss function for the policy network is constructed through importance sampling, and the policy network is then updated. The edge device intelligent agent training module is configured to train an intelligent agent located on the edge device; including: The intelligent agent located at the edge node obtains information from its own task queue and its own running status, and generates status parameters; The state parameters are input into the policy network of the edge node's intelligent agent. The policy network outputs the filtering decision for the task and receives the reward for the batch decision. At the same time, the state parameters of the edge node itself change. The intelligent agent located at the edge node puts its own state parameters, task filtering decisions, decision rewards, and state parameters for the next time step into the experience replay buffer; Small-batch random sampling is performed from the experience replay buffer, and the sampled experience is used to train the value network of the edge node intelligent agent. The value network generates state value feedback to the policy network of the intelligent agent. The intelligent agent located at the edge node constructs the loss function of the policy network based on the state value using the generalized advantage estimation technique and the importance sampling method, and updates the policy network accordingly. The dynamic scheduling module is configured such that each terminal device processes the current observation data in real time through the policy network, obtains the scheduling decision of the current security task, and assigns the task to the selected edge node. After receiving the task scheduled by each terminal device, the edge node makes filtering decisions on these tasks according to its own policy network.

Images