End-side cooperative defense optimization deployment method for advanced persistent threats

By using deep reinforcement learning networks and end-edge collaborative defense methods in the IoT environment, simulating attacker behavior and device status, and optimizing the uninstall rate and strategy of security defense tasks, the difficulty of responding to advanced persistent threats in the existing technology is solved. , achieving efficient defense performance improvement and energy consumption reduction.

CN119030730APending Publication Date: 2024-11-26STATE GRID FUJIAN ELECTRIC POWER CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410795173.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-19
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When facing advanced persistent threats, prior art requires prior information for model learning and training, and deep reinforcement learning models are not close to actual usage scenarios, which limits their application effects.

Method used

A method of optimization deployment of end-edge collaborative defense for advanced persistent threats is proposed. By initializing the deep reinforcement learning network on edge servers, simulating attacker behavior, defining device status and actions, and updating parameters using experience pools and stochastic gradient descent method. Optimize the uninstall rate and execution strategy of security defense tasks to achieve coordinated defense between device and edge servers.

Benefits of technology

There is no need to obtain attacker information in advance. By constantly updating deep reinforcement learning algorithms, we can effectively analyze and respond to advanced persistent threats, improve the defense performance of IoT devices, reduce energy consumption on the device side, and achieve more detailed defense analysis and strategy optimization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119030730A_ABST
    Figure CN119030730A_ABST
Patent Text Reader

Abstract

The invention relates to an advanced persistent threat-oriented end-side cooperative defense optimization deployment method. The method comprises the following steps of performing initialization processing on a deep reinforcement learning network on an edge computing server and an equipment end security defense strategy; the host infected by the zombie program virus executes an instruction sent by a simulation attacker; determining the state of a certain device in a certain time slot, taking the state as the input of deep reinforcement learning, and obtaining the action of the device in the time slot; in a certain time slot, according to the states and actions of all the devices, calculating a total reward which can be obtained by the actions and the next state of each device; setting an experience pool for storing the states, the next state, the action and the reward of all equipment; extracting data from the experience pool, and optimizing the deep reinforcement learning network by taking reward maximization as a target; when the maximum number of iterations is reached, iteration is ended, a final reward result is obtained, and the unloading rate of the security defense task and the execution strategy of the security defense task are obtained through the final reward result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of network attack and defense security, and more specifically, to an end-edge collaborative defense optimization deployment method for advanced persistent threats. Background Art

[0002] With the development of the Internet of Things, IoT technologies such as smart cities and smart homes have provided great convenience to people's lives, but at the same time, various security vulnerabilities and attack methods continue to emerge, bringing huge risks to users. However, the computing power and resources of the device side in the IoT environment are usually very limited and cannot cope with advanced persistent threats. Therefore, the security defense tasks on the device side can be offloaded to the edge server for execution, making full use of the end-to-end collaboration to defend against advanced persistent threats. By deploying reinforcement learning algorithms on the edge server, the end-to-end offloading rate of the device-side security defense tasks and its security defense strategy can be obtained. The end-to-end collaborative defense architecture can minimize the energy consumption of the device side and improve the level of device-side defense against advanced persistent threats without overloading the edge server.

[0003] End-edge collaboration and reinforcement learning are widely used in the field of modern network security. For example, the Chinese patent authorization publication number CN109194505A proposes a power network security defense alarm system, which includes modules such as security supervision, regional supervision, collaborative defense, real-time alarm and security analysis. By monitoring the security status information of the network host and the real-time communication status of the network, the power grid terminal and the edge collaborate to provide security defense, and conduct network security analysis on the network, which can comprehensively protect the security of the entire power network. The Chinese patent authorization publication number CN117609734A proposes a risk assessment method based on deep reinforcement learning, which uses a deep reinforcement learning algorithm to find attack paths, simulate the behavior and thinking of attackers, and analyze the training process of deep reinforcement learning at the same time, calculate the probability of the device being attacked, and realize the security risk assessment of the device. The Chinese patent authorization publication number CN117220995A proposes a network deception defense strategy optimization method and system based on intelligent real-time game. The optimal deception defense strategy is selected through a network deception differential game model and based on deep reinforcement learning, so that the defender can adjust the deployment of deception assets in real time according to the attacker's strategy, and can balance the deception defense effect and deception resource overhead while preventing the attacker from discovering the deception strategy.

[0004] The problem with the above-mentioned existing technologies is that certain prior information is required for model learning and training, or the designed deep reinforcement learning model is not close to the actual usage scenario and is subject to too many restrictions during use. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes an optimized deployment method of end-edge collaborative defense against advanced persistent threats.

[0006] The technical solution of the present invention is as follows:

[0007] On the one hand, the present invention proposes an optimized deployment method for end-edge collaborative defense against advanced persistent threats, which is applied in an environment where a device end is communicated with an edge computing server, and the device end includes several devices. The method is characterized in that the optimized deployment method for end-edge collaborative defense includes the following steps:

[0008] Step 1: Initialize the deep reinforcement learning network set up on the edge computing server. The device side obtains the offloading rate of the security defense task and its execution strategy through the deep reinforcement learning network. The offloading rate is the proportion of the security defense task offloaded by the device to the edge server.

[0009] Step 2: Simulate the attacker's behavior and use the zombie virus to infect the host. The infected host acts as a zombie host to execute the instructions issued by the simulated attacker and confirm the attack time k and attack interval y (k) ;

[0010] Step 3: Define the state of the i-th device as Will As the input of deep reinforcement learning, obtain the action of the device in this time slot

[0011] Step 4: In the kth time slot, according to the state s of all devices in the environment (k) and action a (k) , calculate the total reward u that the action can obtain (k) and the next state of each device

[0012] Step 5: Set up an experience pool to store the status of all devices Next state action and rewards

[0013] Step 6: Randomly take samples from the experience pool according to uniform distribution to form batch data, and update the parameters of the deep reinforcement learning network according to the stochastic gradient descent method based on the batch data with the goal of maximizing the reward;

[0014] Step 7: Repeat steps 2 to 6. When the maximum number of iterations is reached, the iteration ends and the final reward result is obtained. The uninstall rate of the security defense task and its execution strategy are obtained through the final reward result.

[0015] 2. According to the method for optimizing deployment of device-edge collaborative defense for advanced persistent threats in claim 1, it is characterized in that: in step 3, the state of the i-th device is defined as It is expressed by the following formula:

[0016]

[0017] In the formula Indicates the importance of the i-th device, represents the detection delay of the i-th device, represents the damaged data of the i-th device, represents the remaining computing power of the ith device, It represents the transmission time for the i-th device to upload the task to the edge server, and N is the total number of devices.

[0018] 3. According to the method for optimizing deployment of end-edge collaborative defense for advanced persistent threats in claim 1, it is characterized in that: the action of the device in the time slot is obtained in step 3 It is expressed by the following formula:

[0019]

[0020] In the formula is the detection interval of the ith device, is the detection duration of the i-th device, is the device CPU frequency, is the CPU processing frequency of the edge server, is the task offloading rate of the i-th device.

[0021] 4. According to the method for optimizing deployment of end-edge collaborative defense for advanced persistent threats in claim 1, it is characterized in that: the total reward u that can be obtained by calculating the action in step 4 (k) ; expressed by the following formula

[0022]

[0023] In the formula represents the cost of all tasks being completed locally, represents the cost of completing all tasks collaboratively on the edge and the total reward u (k) It is regarded as the ratio of cost saved by completing tasks through end-edge collaboration.

[0024] 5. According to the method for optimizing deployment of end-edge collaborative defense for advanced persistent threats in claim 1, it is characterized in that: in step 6, the maximum reward is taken as the goal, and the maximum reward is set as the target value Specifically:

[0025]

[0026] Where γ is the attenuation coefficient. When γ approaches 0, it means that reinforcement learning considers immediate rewards. When γ approaches 1, it means that reinforcement learning considers future rewards. To determine the next state of the device and all possible actions The maximum value of the reward received.

[0027] 6. The method according to claim 4 (k) It can be regarded as the ratio of the cost saved by the end-edge collaboration to complete the task. The feature is that the end-edge collaboration cost includes the cost of executing a task locally. and the cost of completely offloading a task to the edge server and The calculation formula is as follows:

[0028]

[0029]

[0030] Where W i is the transmission power of the ith device.

[0031] 7. The method of claim 5, wherein the maximum reward is the goal of minimizing the loss function, and the specific formula of the loss function is as follows:

[0032]

[0033] 8. An end-edge collaborative defense optimization deployment system for advanced persistent threats, characterized by comprising:

[0034] Initialization module, which initializes the deep reinforcement learning network designed by the edge computing server. The device side can obtain the offloading rate and execution strategy of the security defense task based on the network;

[0035] Attack module, the attacker uses a zombie program virus to infect a large number of hosts. The infected hosts act as zombie hosts to execute the instructions issued by the attacker and confirm the attack time k and attack interval y (k) ;

[0036] The data acquisition module defines the state of the i-th device as Will As the input of deep reinforcement learning, the action of the device in this time slot can be obtained At the kth time slot, the states s of all devices in the environment are known (k) and action a (k), we can further calculate the total reward u that the action can obtain (k) and the next state of the device

[0037] Data storage module: Set up an experience pool to store the status of all devices Next state action and rewards

[0038] In the learning optimization module, samples are randomly taken from the experience pool according to a uniform distribution to form batch data, and the parameters of the deep reinforcement learning DQN are updated according to the stochastic gradient descent method with the goal of maximizing the reward.

[0039] On the other hand, the present invention also provides an electronic device having a computer program stored thereon, wherein when the computer program is executed by a processor, an end-edge collaborative defense optimization deployment method for advanced persistent threats as described in any embodiment of the present invention is implemented.

[0040] On the other hand, the present invention also provides a computer-readable medium for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement an optimized deployment method for end-edge collaborative defense against advanced persistent threats as described in any embodiment of the present invention.

[0041] The present invention has the following beneficial effects:

[0042] The present invention applies end-edge collaboration and reinforcement learning to the IoT environment to deal with the advanced persistent threat scenario launched by botnets. This method does not need to obtain attacker information in advance, and can respond to advanced persistent threats through continuous exploration of the deep reinforcement learning algorithm deployed on the edge server during the network attack and defense process. At the same time, compared with other reinforcement learning methods for network attack and defense security, this method analyzes the advanced persistent threats launched by botnets in more detail, designs the state, action and utility in the deep reinforcement learning algorithm and gives a detailed description. In order to improve the defense performance of the Internet of Things and reduce the energy consumption of the device side, the device side can obtain the task offloading rate, detection interval, detection time, CPU processing frequency of the device side and CPU processing frequency of the edge server through the deep reinforcement learning algorithm deployed on the edge server, and select the execution method of the security defense task according to the task offloading rate, including local execution, partial offloading or full offloading. The deep reinforcement learning algorithm can continuously balance the energy consumption of the device side and its defense level through the continuous updating of parameters in each time slot, and effectively defend against advanced persistent threats. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 This is a schematic diagram of the method flow of Example 1;

[0045] Figure 2 This is a scenario diagram of the optimized deployment solution for end-edge collaborative defense against advanced persistent threats described in the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0048] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0049] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0050] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0051] Embodiment 1:

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following will be combined with the specific embodiments of the present application and refer to the attached Figure 1 , clearly and completely describe the technical solution of the present invention.

[0053] To solve the problems in the prior art, the present invention provides an optimized deployment method for end-edge collaborative defense against advanced persistent threats, comprising the following steps:

[0054] Step 1: Assume that the experiment includes K = 1000 time slots; there is 1 edge server and N = 10 IoT devices in the environment, and end-to-end collaborative defense against advanced persistent threats is performed. Initialize the designed four cascaded deep reinforcement learning networks on the edge computing server (the four cascaded deep reinforcement learning networks respectively estimate the detection duration, device CPU frequency, edge server CPU processing frequency, and task offloading rate for each device in the current time slot, that is, the Q value of the model network, and thus form an end-to-end strategy for security defense, thereby greatly reducing the computational complexity of the classic deep reinforcement learning algorithm. Specifically, in the designed four cascaded Q neural networks, each network consists of three fully connected layers, where the weight parameter of the jth (1≤j≤4) network is θ j , the number of nodes in the first two fully connected layers are f j,1 and f j,2 , and activated by the rectified linear unit function. The number of output nodes of each network is determined by the optional action set of each action). The device can obtain the offloading rate and execution strategy of the security defense task based on the network. Among them, the transfer learning technology is used to load the initialization parameters for the deep reinforcement learning network, thereby accelerating the startup speed of the model.

[0055] Step 2: In the kth time slot, the simulated attacker uses a zombie program virus to infect a large number of hosts. The infected hosts act as zombie hosts to execute the instructions issued by the simulated attacker, and carry out advanced persistent threats to the devices in the environment. The attack interval is y (k) ∈[0,5].

[0056] Step 3: In the kth time slot, define the state of the i-th device as in Indicates the importance of the i-th device, which is determined by the amount of data on the device. represents the detection delay of the i-th device, represents the damaged data of the i-th device, Indicates the remaining computing power of the device. Indicates the transmission time for the device to upload the task to the edge server, which is determined by the amount of tasks. As the input of DQN, the action of the device in this time slot can be obtained in To detect the interval, is the detection duration, is the CPU processing frequency of the device. is the CPU processing frequency of the edge server, is the offloading rate of security defense tasks, where Indicates that the computing task is executed on the device side. Indicates that the computing tasks are completely offloaded to the edge server for execution. Other values ​​indicate that part of the computing task is offloaded to the edge server for execution.

[0057] Step 4: At the kth time slot, the state s of all devices in the environment is known (k) and action a (k) , we can further calculate the total reward that the action can obtain and the next state of the device in represents the cost of all tasks being completed locally, represents the cost of completing all tasks collaboratively on the edge and the total reward u (k) It can be regarded as the ratio of the cost saved by the end-edge collaboration to complete the task; the cost of a task when it is executed locally This includes the increased overhead due to higher detection latency, the CPU energy consumed by the device, and the economic overhead caused by data corruption; the overhead of completely offloading a task to an edge server This includes the increased overhead due to higher detection latency, the CPU operating energy of the edge server, the economic overhead due to data corruption, and the transmission energy consumption of offloading tasks to the edge server; and The calculation formula is as follows:

[0058]

[0059]

[0060] Among them, W i is the transmit power of the device.

[0061] Step 5: Set up an experience pool to store the status of all devices Next state action and rewards

[0062] Step 6: The edge server randomly selects batch_size=128 samples from the experience pool according to uniform distribution to form a batch of data, and updates the parameters of DQN according to the stochastic gradient descent method, so that the predicted Q value gradually approaches the target Q value, that is, the loss function Minimum. Target Q value Where γ is the attenuation coefficient. When γ approaches 0, it means that reinforcement learning considers immediate rewards more. When γ approaches 1, it means that reinforcement learning considers future rewards more. To determine the next state of the device and all possible actions The maximum value of the reward received.

[0063] Step 7: Repeat steps 2 to 6. When k = 1000, the iteration ends and the final reward result is obtained. The unloading rate of the security defense task and its execution strategy are obtained through the final reward result.

[0064] Embodiment 2:

[0065] This embodiment provides an optimized deployment system for device-edge collaborative defense against advanced persistent threats, including:

[0066] Initialization module, which initializes the deep reinforcement learning network designed by the edge computing server. The device side can obtain the offloading rate and execution strategy of the security defense task based on the network;

[0067] Attack module, the attacker uses a zombie program virus to infect a large number of hosts. The infected hosts act as zombie hosts to execute the instructions issued by the attacker and confirm the attack time k and attack interval y (k) ;

[0068] The data acquisition module defines the state of the i-th device as Will As the input of deep reinforcement learning, the action of the device in this time slot can be obtained At the kth time slot, the states s of all devices in the environment are known (k) and action a (k) , we can further calculate the total reward u that the action can obtain (k) and the next state of the device

[0069] Data storage module: Set up an experience pool to store the status of all devices Next state action and rewards

[0070] In the learning optimization module, samples are randomly taken from the experience pool according to a uniform distribution to form batch data, and the parameters of the deep reinforcement learning DQN are updated according to the stochastic gradient descent method with the goal of maximizing the reward.

[0071] Embodiment three:

[0072] This embodiment provides an electronic device having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements an optimized deployment method for end-edge collaborative defense against advanced persistent threats as described in any embodiment of the present invention.

[0073] Embodiment 4:

[0074] This embodiment provides a computer-readable medium for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement an optimized deployment method for end-edge collaborative defense against advanced persistent threats as described in any embodiment of the present invention.

[0075] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0076] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0078] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0079] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An optimized deployment method for end-to-edge collaborative defense against advanced persistent threats, applied in an environment where a device end is connected to an edge computing server for communication, wherein the device end includes several devices, and is characterized in that: The optimized deployment method of device-edge collaborative defense includes the following steps: Step 1: Initialize the deep reinforcement learning network set up on the edge computing server. The device side obtains the offloading rate of the security defense task and its execution strategy through the deep reinforcement learning network. The offloading rate is the proportion of the security defense task offloaded by the device to the edge server. Step 2: Simulate the attacker's behavior and use the zombie virus to infect the host. The infected host acts as a zombie host to execute the instructions issued by the simulated attacker and confirm the attack time k and attack interval y (k) ; Step 3: Define the state of the i-th device as Will As the input of deep reinforcement learning, obtain the action of the device in this time slot Step 4: In the kth time slot, according to the state s of all devices in the environment (k) and action a (k) , calculate the total reward u that the action can obtain (k) and the next state of each device Step 5: Set up an experience pool to store the status of all devices Next state action and rewards Step 6: Randomly take samples from the experience pool according to uniform distribution to form batch data, and update the parameters of the deep reinforcement learning network according to the stochastic gradient descent method based on the batch data with the goal of maximizing the reward; Step 7: Repeat steps 2 to 6. When the maximum number of iterations is reached, the iteration ends and the final reward result is obtained. The uninstall rate of the security defense task and its execution strategy are obtained through the final reward result.

2. The method for optimizing deployment of end-edge collaborative defense against advanced persistent threats according to claim 1 is characterized in that: In step 3, the state of the i-th device is defined as It is expressed by the following formula: Where P i (k) Indicates the importance of the i-th device, represents the detection delay of the i-th device, represents the damaged data of the i-th device, represents the remaining computing power of the ith device, T i (k) It represents the transmission time for the i-th device to upload the task to the edge server, and N is the total number of devices.

3. The method for optimizing deployment of end-edge collaborative defense against advanced persistent threats according to claim 1 is characterized in that: The action of the device in the time slot is obtained in step 3 It is expressed by the following formula: In the formula is the detection interval of the ith device, is the detection duration of the i-th device, is the device CPU frequency, is the CPU processing frequency of the edge server, is the task offloading rate of the i-th device.

4. The method for optimizing deployment of end-edge collaborative defense against advanced persistent threats according to claim 1 is characterized in that: In step 4, the total reward u that can be obtained by the action is calculated (k) ; expressed by the following formula In the formula represents the cost of all tasks being completed locally, represents the cost of completing all tasks collaboratively on the edge and the total reward u (k) It is regarded as the ratio of cost saved by completing tasks through end-edge collaboration.

5. The method for optimizing deployment of end-edge collaborative defense against advanced persistent threats according to claim 1 is characterized in that: In step 6, the maximum reward is taken as the goal, and the maximum reward is set as the target value Specifically: Where γ is the attenuation coefficient. When γ approaches 0, it means that reinforcement learning considers immediate rewards. When γ approaches 1, it means that reinforcement learning considers future rewards. To determine the next state of the device and all possible actions The maximum value of the reward received.

6. The method according to claim 4 (k) It can be regarded as the ratio of the cost saved by the end-edge collaboration to complete the task. It is characterized by: The edge-to-edge collaboration overhead includes the overhead of executing a task locally. and the cost of completely offloading a task to the edge server and The calculation formula is as follows: Where W i is the transmission power of the ith device.

7. The method according to claim 5, wherein the goal is to maximize the reward, characterized in that: The goal of maximizing the reward is to minimize the loss function. The specific formula of the loss function is as follows:

8. An end-edge collaborative defense optimization deployment system for advanced persistent threats, characterized in that: include: Initialization module, which initializes the deep reinforcement learning network designed by the edge computing server. The device side can obtain the offloading rate and execution strategy of the security defense task based on the network; Attack module, the attacker uses a zombie program virus to infect a large number of hosts. The infected hosts act as zombie hosts to execute the instructions issued by the attacker and confirm the attack time k and attack interval y (k) ; The data acquisition module defines the state of the i-th device as Will As the input of deep reinforcement learning, the action of the device in this time slot can be obtained At the kth time slot, the states s of all devices in the environment are known (k) and action a (k) , we can further calculate the total reward u that the action can obtain (k) and the next state of the device Data storage module: Set up an experience pool to store the status of all devices Next state action and rewards In the learning optimization module, samples are randomly taken from the experience pool according to a uniform distribution to form batch data, and the parameters of the deep reinforcement learning DQN are updated according to the stochastic gradient descent method with the goal of maximizing the reward.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements an end-edge collaborative defense optimization deployment method for advanced persistent threats as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements an end-edge collaborative defense optimization deployment method for advanced persistent threats as described in any one of claims 1 to 7.