Distributed task migration method and system based on deep reinforcement learning and storage medium

Abstract

The invention provides a distributed task migration method and system based on deep reinforcement learning and a storage medium, and the method comprises the steps: obtaining resource state information of all nodes in a distributed task system and migration task state information sent by a migration node, inputting the information as decision basis information into a decision network, and outputting a task migration decision; sending the task migration decision to a migration node, and obtaining task migration time delay and execution time delay; obtaining updated decision basis information in the system, and calculating a load balancing index; on the basis of the total time delay and the responsible equilibrium index, calculating a reward value and constructing an experience tuple, and adding the experience tuple into an experience playback pool; and under the condition that the number of the experience tuples is greater than the batch size of the training samples, sampling the experience tuples from the experience playback pool as the training samples, performing synchronous iterative training on the decision network, the at least two value networks and the corresponding target value networks, and updating parameters of all the networks. The performance and the stability of the distributed system can be improved.

Description

Technical Field

[0001] The present invention relates to the field of resource scheduling technology, and in particular to a distributed task migration method, system and storage medium based on deep reinforcement learning. Background Art

[0002] With the rapid development of information technology, distributed systems have become critical infrastructure supporting various security protection and computing-intensive tasks due to their high availability, scalability, and load balancing capabilities. However, in the actual operation of the system, due to uneven initial task distribution, sudden task surges, or dynamic resource fluctuations, some nodes often fall into a state of resource overload due to carrying too many containerized tasks. This overload will not only significantly reduce the local task processing efficiency of the node, but may also cause a surge in task delays, service degradation, and even node failure, thereby threatening the stability and security protection effectiveness of the entire system. Therefore, in distributed security protection systems, designing efficient and adaptive task scheduling and migration mechanisms to dynamically achieve load balancing and optimize the overall system performance is a core challenge to ensure its continuous and reliable operation.

[0003] To address node overload, previous studies have proposed a variety of heuristic rule-based task scheduling and migration strategies. These methods typically pre-set fixed priority rules, load thresholds, or simple scheduling strategies (such as round-robin and lowest load first), which can alleviate overload and improve resource utilization in specific static or quasi-static scenarios. However, in dynamic and complex distributed environments, it is difficult to coordinately optimize task migration efficiency and global system load balancing, resulting in suboptimal decision-making, high risk of node overload, and reduced overall system performance and stability.

[0004] However, heuristic-based task scheduling and migration strategies struggle to adapt to rapid, uncertain changes in environmental conditions in real time, and they are even more incapable of effectively co-optimizing the often conflicting core objectives of task migration efficiency and global system load balancing. In a dynamic and complex distributed environment, this struggle leads to suboptimal decision-making, a high risk of node overload, and decreased overall system performance and stability. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a distributed task migration method, system, and storage medium based on deep reinforcement learning to eliminate or improve one or more deficiencies in the prior art. This method addresses the difficulty of prior art in collaboratively optimizing task migration efficiency and system global load balancing in dynamic and complex distributed environments, resulting in suboptimal decision-making, high risk of node overload, and decreased overall system performance and stability.

[0006] One aspect of the present invention provides a distributed task migration method based on deep reinforcement learning, the method comprising the following steps:

[0007] Obtaining decision-making basis information, including resource status information of all nodes in the distributed task system and migration task status information of the migration node;

[0008] Input the decision-making basis information into the pre-built decision network and output the task migration decision; the task migration decision is used to indicate the node information of the target node that will undertake the migration task;

[0009] Sending the task migration decision to the migration node, so that the migration node can migrate the migration task to the target node for execution based on the node information after receiving the task migration decision;

[0010] The total delay is obtained based on the task migration delay between the migration node and the target node and the execution delay of the migration task on the target node;

[0011] After determining that the migration node and the target node have completed the task migration and execution, obtaining updated decision basis information in the distributed task system, and calculating the load balancing index of the distributed task system based on the updated decision basis information;

[0012] Based on the total delay and the balance index, the reward value corresponding to the task migration decision is calculated;

[0013] Based on the reward value, task migration decision, and the decision basis information before and after the update, the experience tuple is constructed and added to the experience replay pool;

[0014] When the number of experience tuples in the experience replay pool is greater than the batch size of training samples, experience tuples are sampled from the experience replay pool as training samples according to the batch size of training samples;

[0015] Based on the training samples, the decision network, at least two value networks and their corresponding target value networks are synchronously iteratively trained to update the parameters of all networks.

[0016] In some embodiments of the present invention, the number of value networks and target value networks is two each;

[0017] Based on the training sample set, the decision network, at least two value networks and their corresponding target value networks are trained iteratively simultaneously to update the parameters of all networks, including:

[0018] Input the training samples into the two value networks respectively to obtain the decision basis information before the update in the training samples and the value estimation value of the corresponding task transfer decision;

[0019] The updated decision-making basis information in the training sample and the task migration decision corresponding to the updated decision-making basis information are input into the two target value networks. The obtained value estimate is combined with the policy entropy term to generate a fusion target value for updating the two value networks.

[0020] Inputting the target value into a first preset loss function to obtain a first loss function value;

[0021] Using the first loss function value, the gradient descent method is used to update the network parameters of the two value networks;

[0022] Inputting the minimum value among the estimated values ​​into a second preset loss function to obtain a second loss function value;

[0023] Using the second loss function value, the network parameters of the decision network are updated using the gradient descent method;

[0024] Based on the updated network parameters of the two value networks, the network parameters of the two target value networks are updated using a soft update method.

[0025] In some embodiments of the present invention, the first predetermined loss function is used to reduce the difference between the target value and the value estimate;

[0026] The second preset loss function includes a target value term and an entropy regularization term, and is used to maximize the weighted sum of the target value term and the entropy regularization term.

[0027] In some embodiments of the present invention, the target value for updating the two value networks is generated by fusing the value estimates output by the two target value networks, including:

[0028] Calculate the average of the value estimates output by the two target value networks;

[0029] The average is combined with an entropy regularization term to obtain the target value used to update the value network.

[0030] In some embodiments of the present invention, extracting experience tuples from the experience replay pool to construct training samples includes:

[0031] Based on the reward value corresponding to each experience tuple in the experience replay pool, a tuple weight is assigned to each experience tuple; the tuple weight is positively correlated with the reward value;

[0032] Based on the tuple weight corresponding to each experience tuple, calculate the corresponding experience tuple sampling probability;

[0033] Sampling is performed from the experience replay pool based on the experience tuple sampling probability to construct training samples.

[0034] In some embodiments of the present invention, after sampling from the experience replay pool based on the experience tuple sampling probability and constructing the training sample, the following steps are further included:

[0035] For each experience tuple in the training sample, the corresponding learning rate is adjusted according to the corresponding sampling probability.

[0036] In some embodiments of the present invention, the task migration delay includes the transmission delay between the migration node and the target node, and the image acquisition delay corresponding to the container image used to execute the migration task by the target node;

[0037] The migration task includes subtasks executed in a CPU environment and / or subtasks executed in a DPU environment; the execution delay includes the processing delay of each subtask in the corresponding execution environment.

[0038] In some embodiments of the present invention, the updated decision basis information includes updated decision basis information;

[0039] Based on the updated decision-making information, the load balancing index of the distributed task system is calculated, including:

[0040] Based on the updated decision-making information, calculate the resource utilization of each node for each resource type, as well as the average resource utilization of all nodes under each resource type;

[0041] Based on the resource utilization and the average resource utilization, calculate the standard deviation of resource utilization for the corresponding resource type;

[0042] According to the preset weight corresponding to each resource type, the resource utilization standard deviation of each resource type is weighted and summed to obtain the load balancing index of the distributed task system.

[0043] Another aspect of the present invention provides a distributed task migration system based on deep reinforcement learning, including a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction. When the computer program / instruction is executed, the system implements the steps of the distributed task migration method based on deep reinforcement learning as described above.

[0044] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the distributed task migration method based on deep reinforcement learning as described above.

[0045] The distributed task migration method and system based on deep reinforcement learning of the present invention can solve the problem that the existing technology is difficult to coordinately optimize the task migration efficiency and the global load balancing of the system in a dynamic and complex distributed environment, resulting in suboptimal decision-making, high risk of node overload, and decreased overall system performance and stability. By collecting node resource status and migration task status information in real time as the input basis of the decision network, task migration decisions are generated and the execution of migration nodes is guided. After the migration is executed, the total delay is calculated based on the actual measured migration delay and execution delay, and the load balancing index is accurately calculated in combination with the updated system status information. Based on this, a fusion efficiency (total delay) and stability (load balancing) index is constructed. The dual-objective reward function of the efficiency and stability reward function drives the decision network to simultaneously optimize the task processing speed and the overall health of the system. At the same time, the experience replay pool is used to systematically store historical decision-making experience, and the parameters of the decision network and value network are continuously and iteratively updated through batch sampling training, which significantly enhances the decision robustness and adaptability of the model in a dynamic environment. Finally, by dynamically adjusting the weight factors of efficiency and stability in the reward function, an adaptive balance is achieved between reducing the cost of a single migration and maintaining global load balancing, realizing efficient and stable migration of container-based distributed tasks on the migration node, effectively alleviating the problem of node overload, and thus significantly improving the overall performance and operational stability of the distributed system.

[0046] In addition, the distributed tasks on the migration nodes are trimmed and split based on the embedded trimming integration technology, so that some tasks can be executed by containers deployed in the DPU environment, thereby improving the execution efficiency of the migration tasks.

[0047] Furthermore, the SAC-TM algorithm network architecture includes a decision network, at least two value networks, and a target value network that corresponds to each value network. When solving for the target value, the average of the target value network's outputs is used instead of the minimum, avoiding overly conservative strategy selection and improving the algorithm's exploration capabilities and overall performance.

[0048] In addition, considering that random sampling will cause some important experience tuples to be ignored, which in turn slows down the learning speed, reward-based priority sampling is used as a new experience replay mechanism to prioritize sampling of experiences with higher reward values, which can improve the learning speed.

[0049] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0050] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0052] Figure 1 A schematic diagram of the structure of a distributed task migration system based on deep reinforcement learning provided by one embodiment of the present invention.

[0053] Figure 2 A schematic diagram of a process flow of a distributed task migration method based on deep reinforcement learning provided by one embodiment of the present invention.

[0054] Figure 3 A schematic flow chart of a distributed task migration method based on deep reinforcement learning provided in another embodiment of the present invention.

[0055] Figure 4 A network architecture diagram of the SAC-TM algorithm for task migration provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0057] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0058] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0059] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0060] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0061] Figure 1This is a schematic diagram of a distributed task system based on deep reinforcement learning provided by an embodiment of the present application. Figure 1 As shown, the system at least includes: a centralized scheduling device 110, a migration node 120 and a node 130 available for migration.

[0062] The centralized scheduling device 110 is configured to receive distributed task migration requests sent by target nodes and make migration decisions based on the global state information in the distributed task system, thereby implementing the migration of distributed tasks. In some embodiments of the present invention, the centralized scheduling device 110 may be a server or a server cluster. In actual implementation, the centralized scheduling device 110 may also be implemented as a cloud service or a virtual machine. This embodiment does not limit the implementation of the centralized scheduling device 110.

[0063] Among them, the distributed task system includes but is not limited to a distributed security protection task system, a computing-intensive task system, a data-intensive task system, a transaction processing system, etc. This embodiment does not limit the distributed task type of the distributed task system.

[0064] The migration node 120 is a node that needs to perform distributed task migration. The migration node 130 is a node that can receive the complete information and control information of the task to be migrated sent by the migration node 120 and build a corresponding container to execute the migration task.

[0065] In some embodiments of the present invention, the migration node 120 and the node available for migration 130 can be mobile phones, computers, etc. The device types of the migration node 120 and the node available for migration 130 are the same or different. This implementation does not limit the implementation method of the migration node 120 and the node available for migration 130.

[0066] Figure 1 The number of migration nodes 120 and nodes 130 available for migration is taken as one for illustration. In actual implementation, the number of migration nodes 120 and nodes 130 available for migration is one or at least two. This implementation does not limit the number of first terminals 120 and nodes 130 available for migration.

[0067] Each node in a distributed task system can be a device with different computing resources and capabilities. These nodes are responsible for executing the distributed tasks assigned to them, and containerization technology is used to improve task isolation and portability. Each node can select an appropriate container instance to perform a specific task based on its hardware configuration and current load. A node includes components such as a CPU, data processing unit (DPU), or FPGA to support different types of task processing requirements. By rationally allocating and scheduling tasks in the CPU and DPU environments, the distributed task system can flexibly cope with various complex application scenarios while ensuring high efficiency.

[0068] For example, taking the distributed task system as a distributed security protection system and the nodes in the system as security access nodes, each node is provided with a data processing unit element; the security protection node has certain computing power, storage capacity and communication capacity, and can create corresponding container instances in the CPU environment or DPU environment to perform intrusion monitoring, attack interception, data encryption and other security protection tasks.

[0069] In a distributed task system, when a node is in a resource overload state due to taking on a large number of container-based tasks, it can be regarded as a migration node 120, and its position in the distributed task system can be expressed as (x o ,y o To reduce its own pressure and ensure efficient execution of tasks, the migration node 120 will migrate some tasks to other non-overloaded nodes in the system, that is, the nodes 120 available for migration.

[0070] refer to Figure 2 First, based on the concept of microservice decomposition, migration node 120 tailors the tasks to be migrated, identifying subtasks that can be offloaded to the DPU environment and those that can only be processed in the CPU environment. Next, migration node 120 estimates the resource requirements corresponding to each subtask, encapsulates the state information of the tasks to be migrated, and sends it to the centralized scheduling device 120.

[0071] Specifically, in the migration node 120, the set of all subtasks obtained after splitting and pruning can be expressed by the following formula:

[0072] T={t1,t2,...,t |T|}

[0073] Where |T| represents the number of subtasks obtained after splitting and pruning; t1 represents the first subtask in the set; and t2 represents the second subtask in the set. The resource requirements corresponding to each subtask include computing resource requirements, memory resource requirements, and storage resource requirements.

[0074] For any subtask t i , whose task type τi ∈{τ cpu ,τ dpu}. Among them, τ cpu Indicates that this task is only applicable to the CPU environment, τ dpu Indicates that the task can be executed in both the CPU environment and the DPU environment.

[0075] In subtask t i The task type is τ cpu , that is, subtask t i If the subtask can only be executed in a CPU environment, the resource requirements corresponding to the subtask include CPU computing resource requirements, host memory resource requirements, and storage resource requirements.

[0076] In subtask t i The task type is τ dpu , that is, subtask ta i When the subtask can be executed in a CPU or DPU environment, the resource requirements corresponding to the subtask also include DPU computing resource requirements and DPU independent memory resource requirements.

[0077] After receiving the status information of the tasks to be migrated sent by the migration node 120, the centralized scheduling device 120 makes a decision based on the resource status information of each node, and determines a target node from multiple nodes 120 available for migration to receive the tasks required to be migrated by the migration node 110.

[0078] In some embodiments of the present invention, the centralized scheduling device 110 is also used to monitor the resource status information of each node in the distributed system in real time. In any distributed system, the set of nodes 120 available for migration can be expressed by the following formula:

[0079] N={n1,n2,......,n |N|}

[0080] In the formula, |N| represents the number of nodes 120 available for migration; n1 represents the first node 120 available for migration; and n2 represents the second node 120 available for migration.

[0081] For any node n among the nodes 120 available for migration i , its physical coordinate position in the distributed task system It can be expressed as The maximum resources that can be provided can be expressed as follows:

[0082]

[0083] Where, Represents node n i The maximum value of CPU computing resources that can be provided; Represents node n i The maximum value of DPU computing resources that can be provided; Represents node n i The maximum value of memory resources that can be provided; Indicates the maximum value of DPU independent memory resources that can be provided; S i Represents node n i The maximum value of the storage resources that can be provided.

[0084] Correspondingly, at time t, for any node n i , the resource status information obtained by the centralized scheduling device 110 through real-time monitoring can be expressed as follows:

[0085]

[0086] Where, Represents node n i The CPU computing resources available at time t; Represents node n i The DPU computing resources available at time t; Represents node n i The available memory resources at time t; Represents node n i The DPU independent memory resources that can be provided at time t; S i (t) represents node n i The storage resources available at time t.

[0087] like Figure 2 As shown, after the centralized scheduling device 120 makes a task migration decision, it sends the task migration decision to the migration node 120. After receiving the task migration decision, the migration node 120 forwards the task migration decision, the complete information of the task to be migrated (including subtask t1 and subtask t2), and the control information to the target node indicated by the task migration decision.

[0088] After receiving the information sent by the migration node 120, the target node builds a container in the CPU environment or DPU environment using the container image stored in the local or cloud image repository, executes the corresponding subtasks, and finally returns the execution results to the migration node 120.

[0089] Among them, the container image contains the program for executing the subtask and all the dependencies required for its operation to ensure the availability of the container image.

[0090] In some embodiments of the present invention, the container image may be stored locally on the node or in a cloud image repository.

[0091] If the node does not store the required container image locally, it must be downloaded from the cloud image repository. The cloud image repository pre-stores the container images required for various subtasks after trimming. The collection of container images can be expressed as follows:

[0092] I={img1,img2,...,img |I|}

[0093] In the formula, |I| represents the number of image types stored in the cloud image repository, and the container image img i The size can be used To represent, the cloud location of the container image can be expressed as (x img ,y img ).

[0094] The following is a detailed introduction to the distributed task migration method based on deep reinforcement learning provided by this application.

[0095] like Figure 3 As shown, the embodiment of the present application provides a distributed task migration method based on deep reinforcement learning. This embodiment uses this method to Figure 1 Taking the centralized scheduling device in the system shown as an example, the method includes at least the following steps:

[0096] Step S301: Acquire decision basis information, where the decision basis information includes resource status information of all nodes in the distributed task system and migration task status information of the migration node.

[0097] The resource status information refers to the usage of various resources of each node in the distributed task system at the current moment, which is obtained by the centralized scheduling device through real-time monitoring.

[0098] Taking the current time as time t as an example, for any node n i , the resource status information can be expressed as follows:

[0099]

[0100] Where, Represents node n i The CPU computing resources available at time t; Represents node n i The DPU computing resources available at time t; Represents node n i The available memory resources at time t; Represents node n i The DPU independent memory resources that can be provided at time t; S i (t) represents node n iThe storage resources available at time t.

[0101] The migration task status information includes resource requirement information corresponding to the migration task, the task type of the migration task, the image type of the container image used to execute the migration task, and the amount of data to be sent for the migration task.

[0102] In some embodiments of the present invention, the migration task includes migrating the node based on the microservice splitting concept, cutting out at least one subtask. i , whose task type τ i ∈{τ cpu ,τ dpu}.in, cpu Indicates that this task is only applicable to the CPU environment, τ dpu Indicates that the task can be executed in both the CPU environment and the DPU environment.

[0103] In subtask t i The task type is τ cpu , that is, subtask t i If it can only be executed in a CPU environment, its corresponding resource requirements include CPU computing resource requirements, host memory resource requirements, and storage resource requirements. i The task type is τ dpu , that is, subtask ta i When it can be executed in a CPU or DPU environment, its corresponding resource requirements also include DPU computing resource requirements and DPU independent memory resource requirements.

[0104] Step S302: input the decision basis information into the pre-built decision network and output the task migration decision.

[0105] After obtaining the migration task status information and resource status information, the migration decision is made through the pre-built Soft Actor-Critic for Task Migration (SAC-TM) framework.

[0106] The network architecture of the SAC-TM algorithm includes a pre-built decision (Actor) network, at least two value (Critic) networks, and a target value network corresponding to at least two value networks. Figure 4 , Figure 4 This is a network architecture diagram of the SAC-TM algorithm provided in this embodiment.

[0107] like Figure 4 As shown, the decision network is used to obtain the current resource status information in the distributed system. and migration task status information Information as a basis for decision making t , based on decision-making information s t Determine task migration decision a t .

[0108] Specifically, the decision-making basis information obtained at time t can be expressed as follows:

[0109]

[0110] Where s t Indicates information on which decisions are based; Represents the resource status information of all nodes in the distributed task system at time t; Indicates the migration task status information at time t.

[0111] Among them, the resource status information of all nodes It can be expressed by the following formula:

[0112]

[0113] Where, They represent the CPU computing resources that each node can provide at time t among the |N| nodes available for migration; Respectively represent the DPU computing resources that each node can provide at time t; Represents the memory resources that each node can provide at time t; They represent the DPU independent memory resources that each node can provide at time t; S1(t), S2(t),…, S |N| (t) represent the storage resources available at time t; I1(t), I2(t),…, I |N| (t) The types of image containers that each node can provide; Represent the coordinates of each node in the system.

[0114] Migration task status information at time t This includes the amount of migration task data sent to the target node, the size of the data returned by the target node after executing the migration task, the task type of the migration task, the resource requirements, and the required container image type. Computing resources include CPU and DPU computing resources, and memory resources include general host memory resources and DPU independent memory resources.

[0115] Specifically, migration task status information It can be expressed by the following formula:

[0116]

[0117] Where, d t Indicates the size of the data sent to the target node; d t′ represents the size of the data returned after the target node executes the migration task; τ represents the migration task type, including task types that are only applicable to the CPU environment and task types that can be executed in both the CPU environment and the DPU environment; c c Indicates the required CPU computing resources; c d Indicates the required DPU computing resources; m c Indicates the required host memory resources; m d Indicates the required DPU independent memory resources; img indicates the required container image type.

[0118] In some embodiments of the present invention, the task migration decision a t It is used to indicate the node information of the target node that will undertake the migration task. Specifically, the task migration decision set of the decision network can be expressed as follows:

[0119] A={a1,a2,…,a |N|}

[0120] Where a1 represents the migration strategy of migrating the migration task to the available node n1; a2 represents the migration strategy of migrating the migration task to the available node n2; |N| Indicates migrating the migration task to the available node n |N| Migration strategy.

[0121] In the SAC-TM algorithm framework provided in this embodiment, the input of the decision network is the current decision basis information s t , the output is a probability distribution, expressed as π(a|s t ). This probability distribution is used to indicate the decision-making basis information s. t Next, select each action, that is, select the probability of each task migration decision a. Then, based on the probability distribution π(a|s t ) to implement action selection.

[0122] In some embodiments of the present invention, a random sampling method is used to select the probability distribution π(a|s t )Select a task migration decision to execute.

[0123] Step S303: Send the task migration decision to the migration node, so that the migration node can migrate the migration task to the target node for execution based on the node information after receiving the task migration decision.

[0124] After receiving the task migration decision, the migration node will forward the task migration decision, the complete information of the task to be migrated, and the control information to the target node indicated by the task migration decision.

[0125] After receiving the information sent by the migration node, the target node builds a container in the CPU environment or DPU environment using the container image stored locally or in the cloud image repository based on the received container image type, the task type of the migration task, and resource requirements. It schedules the corresponding resources to execute the corresponding subtasks and returns the execution results to the migration node.

[0126] Step S304: Obtain a total delay based on the task migration delay between the migration node and the target node and the execution delay of the migration task executed by the target node.

[0127] The task migration latency includes the transmission latency between the migration node and the target node, as well as the image acquisition latency required for the target node to obtain the container image used to execute the migration task. Accordingly, the total latency is the sum of the transmission latency, image acquisition latency, and execution latency, and can be expressed as follows:

[0128]

[0129] Where, represents the transmission delay between the migration node and the target node; Indicates the image acquisition latency for the target node to obtain the container image used to perform the migration task; Indicates the execution latency of the migration task on the target node.

[0130] In some embodiments of the present invention, the transmission delay includes the transmission delay of the migrating node sending the relevant data of the migration task to the target node, and the transmission delay of receiving the migration task execution result returned by the target node. The transmission rate between the migrating node and the target node can be calculated using the following formula:

[0131]

[0132] Where B represents the communication link bandwidth, P represents the transmit power of the migration node or target node, h is the channel gain, σ 2 is the noise power. In a wireless channel, the channel gain h is related to the migration node and the target node n. z It is inversely proportional to the distance between them, which can be expressed by the following formula:

[0133]

[0134] Where α represents the path loss index, which is used to describe the attenuation of the signal during propagation; x o is the horizontal coordinate of the migration node position coordinate; The horizontal coordinate of the target node position; y o The vertical coordinate representing the position coordinate of the migration node; The vertical coordinate representing the target node's position coordinates.

[0135] In the process of the migration node sending the relevant data of the migration task to the target node, the transmission rate can be expressed as C o ; In the process of the target node returning the execution result of the migration task to the migration node, the transmission rate can be expressed as C z . Thus, the migration node and the target node n z About tasks that need to be migrated j The transmission delay can be expressed as:

[0136]

[0137] Where, Represents the task t sent by the migration node to the target node j The size of the relevant data; Indicates the data size of the execution result returned by the target node to the migration node; C o Indicates the transmission rate when the migration node sends data to the target node; C z Indicates the transmission rate when the target node sends data to the migration node.

[0138] After receiving the migration task data from the migration node, the target node compares the container image type in the data with the type of the locally stored container image to determine whether there is a container image that can execute the task.

[0139] If the container image required to perform the migration task is not stored locally, a container image request is generated based on the received container image type and sent to the cloud image repository, so that when the cloud image repository receives the container image request, it returns the corresponding container image based on the container image type information carried in the request.

[0140] Based on this, the image acquisition latency for the target node to obtain the container image used to perform the migration task can be expressed as follows:

[0141]

[0142] Where, Ext zj ∈{0,1} is used to represent the target node n z Is there a migration task stored on t j Required container image img j , where 0 represents the target node n zMigration tasks are not stored locally. j Required container image img j , 1 represents the target node n z Local storage has migration tasks j Required container image img j ; C img Indicates that the target node downloads the container image img from the cloud j Download speed; Represents the container image img j Data size; T wait Indicates the time that the container image request sent by the target node waits for processing in the cloud.

[0143] The execution latency of the target node to execute the migration task refers to the time required for the target node to execute the migration task, which depends on the task type, task volume, execution environment and the task processing capability of the target node itself.

[0144] In some embodiments of the present invention, a migration task includes subtasks that can only be executed in a CPU environment and / or subtasks that can be executed in a DPU environment. Accordingly, during the execution of the migration task, the execution latency generated includes the execution latency of each subtask in the corresponding execution environment. Specifically, the execution latency can be expressed as follows:

[0145]

[0146] Where W represents the amount of tasks to be processed; Indicates the CPU processing capacity of the target node; Indicates the processing capacity of the DPU of the target node; considering that subtasks that can be executed on the DPU environment can still be executed on the CPU environment, but the performance may be degraded due to the lack of dedicated hardware acceleration, the performance degradation coefficient π is set to measure the type of τ dpu The degree to which the execution efficiency of the subtask decreases due to environmental inconsistency.

[0147] The processing capabilities of the CPU and DPU of the target node vary with the computing resource utilization, which can be expressed by the following formula:

[0148]

[0149] Where, Indicates the CPU processing capacity of the target node; Indicates the processing capacity of the DPU of the target node; μ1 and μ2 are weight coefficients used to measure the basic processing capacity of the CPU and DPU. and Respectively represent the utilization of the target node's CPU computing resources and DPU computing resources.

[0150] Step S305 , after determining that the migration node and the target node complete the task migration, obtain the updated decision basis information of all nodes in the distributed task system, and calculate the load balancing index of the distributed task system based on the updated decision basis information.

[0151] In some embodiments of the present invention, after completing the migration task, the target node returns the execution result to the migration node and generates task completion information and sends it to the centralized scheduling device to inform the centralized scheduling device that the task migration is completed.

[0152] In other embodiments of the present invention, after completing the migration task, the target node returns the execution result to the migration node; after receiving the execution result returned by the target node, the migration node generates task completion information and sends it to the centralized scheduling device to inform the centralized scheduling device that the task migration is completed.

[0153] In a distributed system, different nodes may have different or identical resource supply capabilities and resource usage, and different migration tasks may have different or identical resource requirements. Inconsistent resource usage between nodes can lead to significant resource waste and, to a certain extent, affect a node's resource utilization.

[0154] Therefore, in order to avoid the load imbalance caused by the heterogeneity between different nodes when migrating tasks from the migration node to the target node, load balancing of the distributed system is also an important consideration when the decision network makes task migration decisions.

[0155] In some embodiments of the present invention, the standard deviation of resource utilization of each resource is weighted and summed by a preset weight to obtain the load balancing index of the distributed task system. Specifically, the load balancing index can be expressed by the following formula:

[0156]

[0157] Where r represents the resource type; ω r Represents the preset weight corresponding to resource r, which is used to evaluate the importance of different resources; σ r (t) represents the standard deviation of resource utilization corresponding to resource r at time t, which is calculated based on the resource utilization of resource r and the average resource utilization of all nodes in the distributed task system at time t.

[0158] Specifically, based on the updated decision basis information, the load balancing index of the distributed task system is calculated, including: based on the resource utilization of each node on each resource type, and the average resource utilization of all nodes under each resource type, the resource utilization standard deviation of the corresponding resource type is calculated; according to the preset weight corresponding to each resource type, the resource utilization standard deviation of each resource type is weighted and summed to obtain the load balancing index of the distributed task system.

[0159] For any node n in the distributed system i and any resource r in the node (including CPU computing resources C c , DPU computing resources C d , host memory resources M c , DPU independent memory resource M d Or storage resource S), at time t, the resource utilization of resource r can be expressed by the following formula:

[0160]

[0161] Where, is node n i The maximum value of the resource r that can be provided, is node n i The amount of resource r owned at time t.

[0162] For each resource r, the average resource utilization of all nodes in the distributed system at time t is calculated, which can be expressed as follows:

[0163]

[0164] Where, Represents node n i The resource utilization of resource r at time t; |N| represents the |N|th node in the distributed system; N represents the total number of nodes.

[0165] For each resource r, the standard deviation of resource utilization of all nodes in the distributed system at time t can be expressed as follows:

[0166]

[0167] Where, Represents node n i The resource utilization rate of resource r at time t; |N| represents the |N|th node in the distributed system; N represents the total number of nodes; It represents the average resource utilization of resource r of all nodes in the distributed system at time t.

[0168] Step S306: Based on the total delay and the responsibility balance index, the reward value corresponding to the task migration decision is calculated.

[0169] In deep reinforcement learning, the reward value refers to the immediate feedback signal from the environment to the actions taken by the agent, which is used to measure the quality of the agent's behavior.

[0170] In the SAC-TM algorithm framework provided in this embodiment, the reward value is determined by the total latency of the migration node migrating the task to the target node and obtaining the processing result, as well as the load balancing index of each node in the distributed system after the migration. Furthermore, to significantly amplify the impact of small changes and provide more refined feedback signals for the reward mechanism, in some embodiments of the present invention, the reward value can be calculated using the following formula:

[0171] R=-ω t logT-ω e log(1+E)

[0172] Where T represents the total delay; E represents the load balancing index; ω t Represents the preset weight corresponding to the total delay, which is used to reflect the relative importance of the reward value corresponding to the total delay; ω e Indicates the preset weight corresponding to the complex balance index, which is used to reflect the relative importance of the load balance index to the reward value and satisfies ω t +h e =1.

[0173] In step S307 , an experience tuple is constructed based on the reward value, the task migration decision, and the decision basis information before and after the update, and is added to the experience replay buffer.

[0174] In some embodiments of the present invention, the experience tuple is in the form of a data structure and can be represented as e i =(s t ,a t ,r t ,s t+1 ). Among them, s t represents the decision-making basis information at time t, that is, the decision-making basis information before the update; a t Represents the migration task status information at time t; r t represents the reward value at time t; s t+1 It represents the decision basis information at time t+1, that is, the updated decision basis information.

[0175] Step S308 : When the number of experience tuples in the experience replay pool is greater than the batch size of training samples, experience tuples are sampled from the experience replay pool as training samples according to the batch size of training samples.

[0176] The approved training sample size refers to the number of samples used to calculate the gradient during each parameter update of the decision network. The approved training sample size determines the scale of empirical data relied upon for a single update and has an impact on the stability and efficiency of the training process.

[0177] For example, if the experience replay pool can store up to 10,000 experience tuples and the approved training sample size is 500, when the number of experience tuples currently stored in the experience replay pool is 50, no experience tuples will be sampled from the experience replay pool as training samples; if the number of experience tuples currently stored in the experience replay pool is 1,000, 500 experience tuples will be sampled from these 1,000 experience tuples to construct training samples.

[0178] In some embodiments of the present invention, experience tuples are sampled from the experience replay pool as training samples by random sampling.

[0179] Experience tuples are sampled randomly, meaning that when sampling from the experience replay pool, each experience tuple has the same probability of being sampled. This sampling method does not introduce any bias and helps ensure fairness and stability in the learning process.

[0180] However, the random sampling method may sample experience tuples that are not very helpful for learning, resulting in some important experiences being ignored, which in turn slows down the learning speed and has a certain impact on the convergence speed of the algorithm.

[0181] Based on this, in some other embodiments of the present invention, experience tuples are sampled from the experience replay pool as training samples through a reward-based priority sampling method.

[0182] A weight is assigned to each experience tuple, which is proportional to the reward value corresponding to the experience tuple. The sampling probability of each experience tuple is calculated based on the assigned weight. Then, sampling is performed from the experience replay pool based on the experience tuple sampling probability, and experience tuples with higher reward values ​​are sampled first to construct training samples.

[0183] Specifically, experience tuples are extracted from the experience replay pool to construct training samples, including: assigning a tuple weight to each experience tuple based on the reward value corresponding to each experience tuple in the experience replay pool; the tuple weight is positively correlated with the reward value; based on the tuple weight corresponding to each experience tuple, the corresponding experience tuple sampling probability is calculated; and sampling is performed from the experience replay pool based on the experience tuple sampling probability to construct training samples.

[0184] Among them, the calculation of each experience tuple assigned tuple weight can be expressed by the following formula:

[0185]

[0186] Where, ω i Represents the calculated experience tuple e i tuple weight of r i Represents the experience tuple e i The reward value of α sample It is a hyperparameter that controls the influence of the reward value on the weight.

[0187] In some embodiments of the present invention, sample = 0, experience tuples are randomly sampled from the experience replay pool as training samples.

[0188] Based on the tuple weight corresponding to each experience tuple, the corresponding experience tuple sampling probability is calculated, which can be expressed by the following formula:

[0189]

[0190] Where, P(e i ) represents the experience tuple e i The probability of being sampled; ω i Represents the experience tuple e i tuple weight of ω; j Represents the experience tuple e j , where j ranges from 1 to B, and B represents the total number of experience tuples in the experience replay pool.

[0191] In actual implementation, in a reward-priority-based experience sampling mechanism, non-uniform sampling can lead to some experience tuples being frequently used, while others are rarely involved in training. This can cause deviations to accumulate during network parameter updates, impacting the stability and effectiveness of network parameter convergence. Therefore, while sampling experience tuples, it is also necessary to adaptively adjust the learning rate of the corresponding experience tuple based on its sampling probability to ensure that each experience tuple contributes evenly to network parameter training.

[0192] Specifically, after sampling from the experience replay pool based on the experience tuple sampling probability and constructing the training sample, the method further includes: for each experience tuple in the training sample, adjusting the corresponding learning rate according to the corresponding sampling probability.

[0193] For each sampled experience tuple e i , whose adjusted learning rate η i It can be expressed as:

[0194]

[0195] Where η represents the experience tuple e i The original learning rate; η irepresents the adjusted learning rate; P(e i ) represents the experience tuple e i The probability of being sampled.

[0196] Step S309: Based on the training samples, synchronous iterative training is performed on the decision network, at least two value networks and their corresponding target value networks to update the parameters of all networks.

[0197] After obtaining training samples from the experience replay pool, the network parameters of the decision network, the value network and the corresponding target value network in the SAC-TM algorithm framework provided in this embodiment are iteratively trained using the training samples.

[0198] refer to Figure 2 In this embodiment, the number of value networks and target value networks is two. In actual implementation, the number of value networks and their corresponding target value networks is at least two. This embodiment does not limit the number of value networks and their corresponding target value networks.

[0199] Specifically, based on the training sample set, synchronous iterative training is performed on the decision network, at least two value networks and their corresponding target value networks, and the parameters of all networks are updated, which at least includes the following steps:

[0200] In step S3091, the training samples are input into the two value networks respectively to obtain the decision basis information before updating in the training samples and the value estimation value of the corresponding task migration decision.

[0201] Among them, the two value networks can be expressed as follows:

[0202]

[0203] Where, Represents the i-th value network Q i The network parameters of i are 1 or 2; s t Represents the decision-making basis information input to the i-th value network; a t Represents the task migration decision corresponding to the decision basis information input to the i-th value network.

[0204] The output of the value network is the estimated value of the state-action value, which means that the decision is based on the information s t Next, select task migration decision a t The expected return can be expressed as Q i (s t ,a t ). The state-action value estimate is used to provide the decision network with performance feedback on the current experience tuple, guiding the decision network to optimize its decision to improve the long-term cumulative reward.

[0205] In step S3092, the task migration decision corresponding to the updated decision basis information in the training sample and the updated decision basis generated by the decision network are input into the two target value networks, and the obtained value estimation value is combined with the policy entropy term to generate a fusion to update the target value of the two value networks.

[0206] In deep reinforcement learning algorithms, a single target-value network may learn overly optimistic estimates of the Q-values ​​of certain state-action pairs, resulting in overly risky task transfer decisions, thus affecting the stability and performance of the algorithm.

[0207] Based on this, in one embodiment of the present invention, two target value networks are used to independently output value estimates, and the minimum value output by the two target value networks is combined with the policy entropy term to construct the target value.

[0208] The network structure of the target value network is the same as that of the value network. The two target value networks can be expressed as follows:

[0209]

[0210] Where, Represents the i-th target value network Q i The network parameters, i is 1 or 2; s t Represents the decision-making basis information input to the i-th value network; a t Represents the task migration decision corresponding to the decision basis information input to the i-th target value network.

[0211] The policy entropy term is a quantitative indicator used to measure the randomness or uncertainty of the policy network's decision to select task migration under given decision-making information. It reflects the diversity of the probability distribution of the policy network's output and is used to balance the relationship between exploration and utilization.

[0212] Specifically, the higher the entropy, the more random the policy network is in selecting task migration decisions, and the more inclined it is to explore new action spaces; the lower the entropy, the more inclined the policy network is to select task migration decisions with higher certainty, and the more focused it is on utilizing known optimal task migration decisions. The action space of the SAC-TM algorithm provided by the present invention is a discrete space, so the policy entropy H can be expressed as follows:

[0213]

[0214] Where π(a|s) is the probability of selecting task migration decision a under the decision basis information s.

[0215] Correspondingly, the target value constructed by combining the minimum value output by the two target value networks and the policy entropy term can be expressed as follows:

[0216]

[0217] In the formula, y represents the target value; r represents the reward value; γ represents the discount factor; s t+1 Indicates the updated decision-making information of the distributed system after the task migration is completed; a t+1 Represents the policy network based on the updated decision-making information s t+1 The task migration decision is made by random sampling method according to the probability from the probability distribution π(s t+1 |a t+1 ) is determined in

[0218] However, although the minimum value method can effectively reduce the problem of overestimation of target value, the minimum value may be affected by the extreme estimation of a single target value network, which may lead to the problem of overly pessimistic estimation of the target value network to a certain extent.

[0219] To address this issue, some embodiments of the present invention use the average of the value estimates output by the two target value networks instead of the minimum value to provide a smoother value estimate. This balances the value estimates of the two target value networks to a certain extent, avoiding overly conservative strategy selection and thus improving the algorithm's exploration capabilities and overall performance.

[0220] Specifically, based on the value estimation values ​​output by the two target value networks, a fusion is generated to generate a target value for updating the two value networks, including: calculating the average value of the value estimation values ​​output by the two target value networks; combining the average value with the entropy regularization term to obtain the target value for updating the value network.

[0221] Among them, combining the average value with the entropy regularization term to obtain the target value for updating the value network can be expressed as follows:

[0222]

[0223] In the formula, y represents the target value; r represents the reward value; γ represents the discount factor; s t+1 Indicates the updated decision-making information of the distributed system after the task migration is completed; a t+1 Represents the policy network based on the updated decision-making information s t+1 The task migration decision is made by random sampling method according to the probability from the probability distribution π(s t+1 |a t+1 ) is determined; α is the temperature parameter.

[0224] Step S3093: Input the target value into the first preset loss function to obtain a first loss function value.

[0225] The first preset loss function is used to reduce the difference between the target value and the estimated value. Specifically, the first preset loss function can be expressed as follows:

[0226]

[0227] Where D represents the experience replay pool; represents the average value of the training samples sampled from the experience replay pool; represents the estimated value output by the i-th value network; y t Indicates the target value.

[0228] Step S3094: Use the first loss function value and the gradient descent method to update the network parameters of the two value networks.

[0229] Step S3095: Input the minimum value of the estimated value into the second preset loss function to obtain a second loss function value.

[0230] The second preset loss function includes a target value term and an entropy regularization term, and is used to maximize the weighted sum of the target value term and the entropy regularization term. The second preset loss function can be expressed as follows:

[0231]

[0232] Where D represents the experience replay pool; represents the average value of the training samples sampled from the experience replay pool; θ π represents the network parameters of the decision network; π(s t |a t ) indicates that the decision is based on information s t Under this condition, the probability distribution of the decision network output is: α is the temperature parameter. Q(s t ,a t ) Minimum value among the tabular value estimates.

[0233] Step S3096: Use the second loss function value and adopt the gradient descent method to update the network parameters of the decision network.

[0234] Step S3097: Based on the updated network parameters of the two value networks, the network parameters of the two target value networks are updated using a soft update method.

[0235] In some embodiments of the present invention, the network structure of the target value network is the same as the network structure of the value network. Gradually approach the network parameters of the value network through soft updates It can be expressed by the following formula:

[0236]

[0237] Where τ represents the soft update coefficient, which enables the network parameters of the target value network to smoothly track the network parameters of the value network, thereby providing a more stable reference value in the calculation of the target value.

[0238] In addition, this embodiment provides a training method for each network in the SAC-TM algorithm framework, which method includes at least the following steps:

[0239] Step S401, initialize the decision network parameters θ π .

[0240] To alleviate the overload problem of some nodes and ensure that migrating tasks from the migration node can improve the overall operating efficiency of the distributed task system, the overall optimization goal is set to minimize the total delay and load balancing index while satisfying the constraints. The overall optimization goal can be expressed as:

[0241]

[0242] Where β is a weight coefficient used to indicate the importance; T represents the total delay and E represents the load balancing index; Indicates the CPU computing resources that the target node can provide; Indicates the DPU computing resources that the target node can provide; Indicates the memory resources that the target node can provide; Indicates the DPU independent memory resources that the target node can provide; S z Indicates the storage resources that the target node can provide; ω z Indicates the ratio of storage resources occupied by container images to the storage resources provided by the target node; Indicates the storage resources required for the migration task.

[0243] Step S402: Initialize two value network parameters and

[0244] Step S403: Initialize two target value network parameters:

[0245] Step S404: Initialize the experience replay pool D.

[0246] Step S405: setting hyper parameters.

[0247] Among them, the hyperparameters include the weight coefficient ω of the reward function t and ω e , the soft update coefficient τ in the soft update function of the target value network, the discount factor γ and temperature parameter α in the target value function, the learning rate η of the sampled experience tuple, and the reward sensitivity α in the weight distribution function of the experience tuple sample .

[0248] Specifically, the reward function is expressed as:

[0249] R=-ω t logT-ω e log(1+E)

[0250] Where T represents the total delay; E represents the load balancing index; ω t Represents the preset weight corresponding to the total delay, which is used to reflect the relative importance of the reward value corresponding to the total delay; ω e Indicates the preset weight corresponding to the complex balance index, which is used to reflect the relative importance of the load balance index to the reward value and satisfies ω t +ω e =1.

[0251] The target value network soft update function is expressed as:

[0252]

[0253] Where τ represents the soft update coefficient, which enables the network parameters of the target value network to smoothly track the network parameters of the value network, thereby providing a more stable reference value in the calculation of the target value.

[0254] The target value function is expressed as:

[0255]

[0256] In the formula, y represents the target value; r represents the reward value; γ represents the discount factor; s t+1 Indicates the updated decision-making information of the distributed system after the task migration is completed; a t+1 Represents the policy network based on the updated decision-making information s t+1 The task migration decision is made by random sampling method according to the probability from the probability distribution π(s t+1 |a t+1 ) is determined; α is the temperature parameter.

[0257] The weight distribution function of the experience tuple is expressed as:

[0258]

[0259] Where, ω iRepresents the calculated experience tuple e i tuple weight of r i Represents the experience tuple e i The reward value of α sample It is a hyperparameter that controls the influence of the reward value on the weight.

[0260] Step S406: Based on the training samples, the decision network, at least two value networks and their corresponding target value networks are synchronously iteratively trained to update the network parameters of all networks until the network parameters converge or the preset number of iterations is reached.

[0261] Among them, based on the training samples, the decision network, at least two value networks and their corresponding target value networks are synchronously iteratively trained, and the steps of updating the network parameters of all networks are the same as the steps in the above-mentioned distributed task migration method embodiment based on deep reinforcement learning, and will not be repeated here.

[0262] In summary, the distributed task migration method based on deep reinforcement learning provided by this embodiment obtains the resource status information of all nodes in the distributed task system and the migration task status information sent by the migration node as decision-making basis information; inputs the decision-making basis information into a pre-built decision network and outputs a task migration decision; the task migration decision is used to indicate the node information of the target node that will undertake the migration task; the task migration decision is sent to the migration node so that the migration node migrates the migration task to the target node for execution based on the node information after receiving the task migration decision; obtains the task migration delay between the migration node and the target node and the execution delay of the migration task by the target node, and calculates The total delay is calculated; after the migration node completes the task migration and execution, the updated decision basis information in the distributed task system is obtained, and based on the updated decision basis information, the load balancing index of the distributed task system is calculated; based on the total delay and the load balancing index, the reward value corresponding to the task migration decision is calculated; based on the reward value, the task migration decision, and the decision basis information before and after the update, the experience tuple is constructed and added to the experience replay pool; when the number of experience tuples in the experience replay pool is greater than the training sample batch size, the experience tuple is sampled from the experience replay pool as a training sample according to the training sample batch size; based on the training samples, the decision network and at least two value networks are The network and its corresponding target value network are trained synchronously and iteratively to update the parameters of all networks. It can solve the problem that existing technologies are difficult to coordinately optimize task migration efficiency and system global load balancing in dynamic and complex distributed environments, resulting in suboptimal decision-making, high risk of node overload, and decreased overall system performance and stability. By collecting node resource status and migration task status information in real time as the input basis of the decision network, task migration decisions are generated and the execution of migration nodes is guided. After the migration is executed, the total delay is calculated based on the actual measured migration delay and execution delay, and the load balancing index is accurately calculated in combination with the updated system status information. Based on this, a fusion efficiency (total delay) and stability (load balancing) model is constructed. The dual-objective reward function of the efficiency and stability in the reward function is used to drive the decision network to simultaneously optimize the task processing speed and the overall health of the system. At the same time, the experience replay pool is used to systematically store historical decision-making experience, and the parameters of the decision network and the value network are continuously and iteratively updated through batch sampling training, which significantly enhances the decision robustness and adaptability of the model in a dynamic environment. Finally, by dynamically adjusting the weight factors of efficiency and stability in the reward function, an adaptive balance is achieved between reducing the cost of a single migration and maintaining global load balancing, realizing efficient and stable migration of container-based distributed tasks on the migration node, effectively alleviating the problem of node overload, and thus significantly improving the overall performance and operational stability of the distributed system.

[0263] In addition, the distributed tasks on the migration nodes are trimmed and split based on the embedded trimming integration technology, so that some tasks can be executed by containers deployed in the DPU environment, thereby improving the execution efficiency of the migration tasks.

[0264] Furthermore, the SAC-TM algorithm network architecture includes a decision network, at least two value networks, and a target value network that corresponds to each value network. When solving for the target value, the average of the target value network's outputs is used instead of the minimum, avoiding overly conservative strategy selection and improving the algorithm's exploration capabilities and overall performance.

[0265] In addition, considering that random sampling will cause some important experience tuples to be ignored, which in turn slows down the learning speed, reward-based priority sampling is used as a new experience replay mechanism to prioritize sampling of experiences with higher reward values, which can improve the learning speed.

[0266] Corresponding to the above method, the present invention also provides a distributed task migration system based on deep reinforcement learning, the device / system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the device / system implements the steps of the distributed task migration method based on deep reinforcement learning as described above.

[0267] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned distributed task migration method based on deep reinforcement learning. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0268] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0269] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0270] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0271] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A distributed task migration method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Acquiring decision-making basis information, wherein the decision-making basis information includes resource status information of all nodes in the distributed task system and migration task status information of the migration node; Input the decision basis information into a pre-built decision network and output a task migration decision; the task migration decision is used to indicate the node information of the target node that will undertake the migration task; Sending the task migration decision to the migration node, so that the migration node can migrate the migration task to the target node for execution based on the node information after receiving the task migration decision; Obtaining a total delay based on a task migration delay between the migration node and the target node and an execution delay of the migration task by the target node; After determining that the migration node and the target node have completed the task migration and execution, obtaining updated decision basis information in the distributed task system, and calculating the load balancing index of the distributed task system based on the updated decision basis information; Based on the total delay and the responsible balance index, a reward value corresponding to the task migration decision is calculated; Constructing an experience tuple based on the reward value, the task migration decision, and the decision basis information before and after the update, and adding the tuple to the experience replay pool; When the number of experience tuples in the experience replay pool is greater than the training sample batch size, sampling experience tuples from the experience replay pool as training samples according to the training sample batch size; Based on the training samples, the decision network, at least two value networks and their corresponding target value networks are synchronously iteratively trained to update the parameters of all networks.

2. The method according to claim 1, characterized in that The number of the value network and the number of the target value network are two respectively; The step of performing synchronous iterative training on the decision network, the at least two value networks, and their corresponding target value networks based on the training sample set, and updating parameters of all networks includes: Inputting the training samples into the two value networks respectively to obtain the decision basis information before updating in the training samples and the value estimation value of the corresponding task migration decision; Inputting the updated decision basis information in the training sample and the task migration decision corresponding to the updated decision basis information into the two target value networks, combining the obtained value estimate with the policy entropy term to generate a target value for updating the two value networks; Inputting the target value into a first preset loss function to obtain a first loss function value; Using the first loss function value, updating the network parameters of the two value networks using a gradient descent method; Inputting the minimum value among the value estimates into a second preset loss function to obtain a second loss function value; Using the second loss function value, updating the network parameters of the decision network using a gradient descent method; Based on the updated network parameters of the two value networks, the network parameters of the two target value networks are updated using a soft update method.

3. The method according to claim 2, characterized in that The first preset loss function is used to reduce the difference between the target value and the value estimate; The second preset loss function includes a target value term and an entropy regularization term, and is used to maximize the weighted sum of the target value term and the entropy regularization term.

4. The method according to claim 2, characterized in that The value estimation values ​​outputted by the two target value networks are fused to generate target values ​​for updating the two value networks, including: Calculating an average of the value estimates output by the two target value networks; The average is combined with an entropy regularization term to obtain a target value for updating the value network.

5. The method according to claim 1, wherein The extracting experience tuples from the experience replay pool to construct training samples includes: Based on the reward value corresponding to each experience tuple in the experience replay pool, a tuple weight is assigned to each experience tuple; the tuple weight is positively correlated with the reward value; Based on the tuple weight corresponding to each experience tuple, calculate the corresponding experience tuple sampling probability; Sampling is performed from the experience replay pool based on the experience tuple sampling probability to construct a training sample.

6. The method according to claim 5, characterized in that After sampling from the experience replay pool based on the experience tuple sampling probability and constructing a training sample, the method further includes: For each experience tuple in the training sample, the corresponding learning rate is adjusted according to the corresponding sampling probability.

7. The method according to claim 1, characterized in that The task migration delay includes the transmission delay between the migration node and the target node, and the image acquisition delay corresponding to the target node acquiring the container image for executing the migration task; The migration task includes a subtask executed in a CPU environment and / or a subtask executed in a DPU environment; the execution delay includes a processing delay of each subtask in a corresponding execution environment.

8. The method according to claim 1, characterized in that The updated decision basis information includes updated decision basis information; The calculating the load balancing index of the distributed task system based on the updated decision basis information includes: Based on the updated decision basis information, calculate the resource utilization of each node for each resource type and the average resource utilization of all nodes under each resource type; Calculating a standard deviation of resource utilization corresponding to a resource type based on the resource utilization and the resource utilization average; According to the preset weight corresponding to each resource type, the resource utilization standard deviation of each resource type is weighted and summed to obtain the load balancing index of the distributed task system.

9. A distributed task migration system based on deep reinforcement learning, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is configured to execute the computer program / instructions. When the computer program / instructions are executed, the system implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images