Dependent task unloading and service placement method based on HDQN

Abstract

Task unloading and service placement are two key technologies for reducing task completion time delay in the field of edge computing research, however, task unloading decisions and service placement decisions are optimized separately in existing research, the problem is simplified to a certain degree through the method, but the close relation and mutual influence between the task unloading decisions and the service placement decisions are neglected. In order to deal with the challenge, the invention provides a joint optimization scheme, a hierarchical deep reinforcement learning (HDQN) architecture is provided aiming at dual constraints of task dependence and service deployment in a complex scene, a joint optimization problem is decoupled into two decision levels, an upper layer carries out resource allocation through service placement optimization, and a lower layer carries out resource allocation through service deployment optimization. And the lower layer performs real-time task scheduling based on a dynamic environment. Through the layering mechanism, task unloading and resource constraint are effectively coordinated, and the dynamic change of the edge computing environment is adaptively coped with. Experimental results show that the method has excellent performance in the aspect of reducing task completion time.

Description

Technical Field

[0001] The present invention relates to the field of edge computing task offloading and service placement, and realizes efficient task scheduling and resource allocation in an environment with multi-task dependencies and limited computing resources. Background Art

[0002] With the deep integration of 5G communication technology and the Internet of Things (IoT) technology, latency-sensitive applications such as smart cities and industrial automation are experiencing explosive growth. Traditional cloud computing architecture is limited by the physical distance between the cloud and the terminal, and faces bottlenecks in transmission latency and bandwidth consumption. Multi-access edge computing (MEC) utilizes computing resources at the edge of the network to effectively reduce transmission latency and enhance user experience.

[0003] Task offloading and service placement have been widely studied as key technologies for reducing latency in MEC. Task offloading focuses on how to properly schedule tasks, transferring them from resource-constrained devices (such as mobile phones and cameras) to suitable MEC servers with sufficient computing power. Service placement, on the other hand, focuses on configuring appropriate service types on MEC servers with limited storage space to maximize resource utilization and improve overall computing efficiency.

[0004] Existing designs often assume that user-provided tasks are independent. However, in real-world applications, many computing tasks are not independent but rather interdependent. For example, in scenarios such as video analysis, autonomous driving, and industrial inspection, prerequisite tasks (such as image preprocessing and object detection) must be completed before their results can be used by subsequent tasks (such as object tracking and decision control). Therefore, decisions about task offloading and service placement must not only consider the computing resource requirements of individual tasks but also comprehensively analyze the dependencies between tasks to optimize overall execution efficiency.

[0005] Furthermore, in existing research, optimization strategies for task offloading and service placement are often studied independently, failing to fully consider the coupling relationship between the two. For example, in the field of task offloading research, a general computing task model is commonly adopted, meaning that tasks can be executed on MEC as long as their computing resource constraints are met. However, in real applications, many tasks not only require computing resources but also rely on specific data or service support. For another example, in the field of service placement research, researchers typically model user-generated tasks as task flows, focusing on the overall task's service requirements or computing demand trends to optimize service placement strategies, while ignoring the fine-grained requirements and offloading requirements of individual tasks.

[0006] In summary, task offloading and service placement are two closely related and mutually influential key technologies in edge computing. Task offloading determines how computing tasks are dynamically allocated between end devices, edge servers, and the cloud, while service placement focuses on how to rationally deploy computing services on edge nodes to optimize resource utilization and task execution efficiency. Furthermore, it is necessary to comprehensively consider the dependencies between tasks to simulate more realistic user data flows, which has important practical significance for intelligent applications in MEC environments. Summary of the Invention

[0007] In view of this, this paper proposes an intelligent decision-making scheme for jointly optimizing task offloading and service placement. By using the hierarchical deep reinforcement learning (HDQN) method, task offloading and service placement are optimized in layers. The upper layer optimizes the service placement strategy, and the lower layer optimizes the task offloading strategy. Through the powerful dynamic adaptability and hierarchical processing capabilities of HDQN, an effective strategy optimization solution is provided to users.

[0008] The purpose of the present invention is to provide a method for optimizing task offloading and service placement based on HDQN. To achieve the above purpose, the present invention provides the following technical solutions:

[0009] 1. Build system models, service models, task models, and computing models in a multi-user, multi-MEC environment;

[0010] 2. Obtain information such as the MEC environment, user-generated task information, and the connection status of each device during each time slot, and feed this information into a feature extraction model to obtain comprehensive information required for decision-making;

[0011] 3. Build an HDQN upper-layer decision model, using the previous time slot's task offloading strategy and MEC environment information as input to the upper-layer network. Through continuous experience learning and environmental feedback, the strategy is continuously adjusted to output the optimal service placement strategy.

[0012] 4. The service placement strategy and feature extraction model output from step 3 are used as input to the HDQN lower-level decision model. The dual Q network (DDQN) dynamically selects the optimal offloading path and optimizes the scheduling and execution order of tasks.

[0013] 5. Build a joint optimization model. Upper-level service placement decisions guide lower-level decisions, while lower-level task offloading decisions influence upper-level decisions. Through continuous dynamic optimization of the two-layer decision model, the optimal service placement and task offloading strategy is determined.

[0014] 6. Model Effectiveness Evaluation: Using task completion latency as the evaluation metric, the proposed model is compared with current mainstream algorithms (Greedy, DQN, and Local) to determine the effectiveness of the proposed technical solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0016] Figure 1 It is the system model diagram.

[0017] Figure 2 This is a diagram of the dependent task structure.

[0018] Figure 3 Flowchart of the HDQN method.

[0019] Figure 4 This is a two-level decision effectiveness analysis diagram.

[0020] Figure 5 Validity analysis diagram for different numbers of users.

[0021] Figure 6 The effectiveness analysis diagram for different numbers of MEC servers.

[0022] Figure 7 This is a diagram analyzing the effectiveness of different types of services. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0024] The accompanying drawings are for illustrative purposes only and are schematic, not physical, representations. They should not be construed as limiting the present invention. To better illustrate the embodiments of the present invention, some components in the accompanying drawings may be omitted, enlarged, or reduced in size, and do not represent the actual dimensions of the products. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted from the accompanying drawings. The same or similar reference numerals in the accompanying drawings of the embodiments of the present invention correspond to the same or similar components.

[0025] The present invention provides a method for optimizing dependent task offloading and service placement based on HDQN, which realizes the joint optimization of task offloading and service placement strategies under the premise of considering task dependencies, thereby reducing task completion time. The method includes the following steps:

[0026] Step 1: Build the system model, task and service model, and computing model in a multi-user multi-MEC environment. The specific steps are as follows.

[0027] 1. Consider multiple users and MEC server scenarios and establish a system model.

[0028] like Figure 1 As shown in Figure 1, this system considers a three-tier service architecture of cloud, edge, and end, consisting of a cloud server, multiple user equipment (UE), and MEC servers. UEs are connected to multiple MEC servers via wireless networks. Each MEC server has computing and storage resources and can provide computing services for specific applications.

[0029] UEs have limited computing and storage capabilities and can generate computing tasks. UEs have different hardware configurations, and the scale of the tasks they generate may also vary. MEC servers are deployed at the edge of the network, and each MEC server can handle tasks for multiple UEs. MEC servers have limited computing and storage capabilities and can configure limited services.

[0030] In traditional MEC systems, service placement is often static, meaning the location of services is determined at system startup. In the system of the present invention, each user equipment (UE) is statically configured with a service during initialization, based on its needs and the pre-set application environment. The MEC server flexibly adjusts and configures the deployment of multiple services based on the dynamic changes in its storage space. Cloud servers, with their ample storage space, can configure all services.

[0031] 2. Establish a service model

[0032] The system provides a set of service libraries ,in is the total number of services. Each service The data size is recorded as The present invention divides the system scheduling time into multiple equal time slots. , ,in represents the length of the total scheduling cycle, The number of the current time slot. Tasks can be run locally if the user device is configured with the corresponding service. If the device is not configured with the required service, the user must offload the task to an MEC server that has the service configured via the wireless network. If neither the user device nor the corresponding MEC server is configured with the service, the task is offloaded to the cloud server for processing.

[0033] 3. Establish a task model

[0034] The task model defines the dependencies, data size, computational requirements, and task types of user-generated tasks. Each user generates a set of tasks with internal dependencies (e.g. Figure 2 As shown in Figure 2), the complexity of the dependencies between tasks, the number of subtasks, and the services required for each subtask are determined by the user based on their needs.

[0035] 4. Build a computational model

[0036] The computing model consists of local latency, edge computing latency, cloud computing latency, and service migration latency.

[0037] Specifically, in the time slot When MEC server Migration Services Delay It can be expressed as:

[0038]

[0039] in It's a service The amount of data required, It is the data transmission rate between the cloud server and the target MEC server.

[0040] For users, if a task needs to be executed locally and the user has not configured the corresponding service, the task must be offloaded to the cloud server for processing. For MEC servers, if the user offloads a task to a MEC server and the server does not have the corresponding service configured, the task needs to be further offloaded to the cloud server for processing.

[0041] When the user Provided subtasks When running locally, its ready time It can be expressed as:

[0042]

[0043] in, Indicates a task The completion time, Represents a user The computing idle time of the local device, express The set of predecessor tasks. Completion time It can be expressed as:

[0044]

[0045] in, Represents a user The computing power of the local device.

[0046] When the task is offloaded to the directly connected MEC server No. When a VM is available, its ready time It can be expressed as:

[0047]

[0048] Completion time It can be expressed as:

[0049]

[0050] When tasks are offloaded to a non-directly connected server , then the task needs to be unloaded to , and then through Transfer to In this case, its ready time It can be expressed as:

[0051]

[0052] Completion time It can be expressed as:

[0053]

[0054] Step 2: Environmental information collection and feature extraction. The specific steps are as follows.

[0055] 1. Environmental state extraction

[0056] In a multi-user, multi-MEC (edge computing) environment, the system needs to collect and process information at multiple levels, including MEC environment information, user-generated task information, and the connection status of each device. By collecting this information, comprehensive information support is provided for subsequent task offloading and optimization of service placement strategies.

[0057] MEC environment information includes the computing power, storage, and network load of each MEC server, as well as the number and load of services currently configured on the server. Task information needs to include the computing and storage resources required for each subtask, the dependencies between tasks, and the required service types. Device connection information needs to consider the connection status between the user device and the MEC server, as well as the current computing and storage capabilities of each device.

[0058] 2. Feature Extraction

[0059] The feature extraction model transforms environmental information into high-dimensional feature representations for decision-making. These features serve as input to subsequent decision-making models, helping them select the optimal task offloading path and service placement strategy.

[0060] In this invention, the system collects multidimensional environmental information and task features in real time and uses a feature extraction model to convert this information into a high-dimensional feature representation suitable for decision-making. The system first collects key environmental information from user devices and MEC servers, collects task information generated by user devices, and monitors the network connection status and computing power of the devices. Then, using normalization techniques, it extracts numerical and categorical features from the collected raw data. Based on the different requirements of the tasks, these features are converted into high-dimensional feature vectors suitable for input into the decision-making model. This feature vector serves as the basis for subsequent task offloading and service placement decisions.

[0061] Step 3: Build an HDQN upper-layer service placement decision optimization model.

[0062] In the present invention, both the upper and lower layer decision models of HDQN are constructed using DDQN. The upper layer decision model outputs the optimal service placement decision based on the task offloading strategy of the previous time slot and the current MEC environment information. The specific steps are as follows.

[0063] The input to the upper-level decision model is the task offloading strategy and MEC environment information for the previous time slot. This information is converted into a high-dimensional feature representation through the feature extraction model and used as the input to the DDQN network. The specific state space is as follows:

[0064]

[0065] in, Indicates the serial number of the current service. Indicates the storage resources required for the current service deployment. Indicates the service deployment status of the previous time slot, Indicates the available storage resources of the MEC server in the current time slot.

[0066] The action space of the upper-level decision-making includes different service placement strategies, that is, determining which services should be deployed on which virtual machines. The specific action space is as follows:

[0067]

[0068] in Indicates the number of the virtual machine in the system.

[0069] Service placement reward function The reward is calculated based on the migration time of the service. The longer the migration time, the lower the reward.

[0070]

[0071] in Represents the MEC server set, Represents a set of service types. Indicates the migration time of the service.

[0072] Step 4: Build a lower-level task offloading decision optimization model.

[0073] In step 3, the upper-level decision model has output the optimal service placement strategy, determining which services should be deployed on which MEC servers. The feature extraction model also outputs high-dimensional features related to task characteristics, MEC environment status, network quality, etc. These outputs serve as input to the lower-level decision model, enabling more refined task offload path selection and task scheduling.

[0074] The lower-level decision model's primary task is to select an offload path based on the input service placement strategy and task characteristics. This involves deciding whether each task should be executed locally or offloaded to an MEC server or cloud server. The lower-level model also needs to optimize task scheduling and execution order, scheduling across multiple tasks to meet task dependencies and deadlines.

[0075] Scheduling time using task offloading To represent the optimal time slot for task offloading. Definition Represents the computing capabilities of all VMs, which are defined as follows:

[0076]

[0077] in express Previous The computing power of a virtual machine, Indicates the MEC server The number of virtual machines, Indicates the total number of MEC servers

[0078] State space for task offloading It can be expressed as:

[0079] in , , Respectively represent the data size of the task, the amount of computing required, and the type of service required, Indicates the user's computing available time, Indicates the user's computing power, Indicates the transmission speed of the server directly connected to the user.

[0080] The action space describes the All possible unloading strategies that the system may adopt can be expressed as:

[0081]

[0082] when When , it means the task is executed locally. When it is non-zero, it means offloading to the corresponding VM. If the selected device does not have the service corresponding to the task, it will be offloaded to the cloud server.

[0083] The design of the reward function is shown in the following formula:

[0084]

[0085] in Indicates that the task is being performed Previous completion time, Indicates that the task is being performed The completion time afterwards. Defined as user Average time to process all tasks locally.

[0086] Step 5: Establish a two-level decision-making joint optimization model.

[0087] Detailed process as follows Figure 3 As shown, first, at the upper level decision time When the agent obtains the current upper state from the environment , the upper behavior network selects an action based on the current state , and perform the action, and then obtain immediate rewards through environmental feedback After each time slot, the agent uses the target Q network to calculate the target value and updates the parameters of the behavior network by gradient descent to optimize the action selection strategy. When the lower layer task offloading network is based on the upper layer action Update your status , and conduct independent training. The lower-level network selects the appropriate offloading strategy based on the current service placement strategy, task requirements, computing resources and network status. The lower-level task offloading decision It will also affect the upper-layer service placement strategy, because the choice of offloading tasks to MEC servers or cloud servers will directly affect the location of service deployment and resource allocation. The task offloading strategy is also trained using the Q learning method. In the network, the Q value of the task offloading is updated through immediate feedback. The agent optimizes the offloading decision in the lower layer network to ensure that the task can be executed on the most suitable device, thereby optimizing the service task strategy.

[0088] Both the upper and lower layers use the DDQN network for optimization decisions. Their training and optimization methods are similar, so the general formula calculation method is given below. The optimal actions for the upper and lower layers are selected as follows:

[0089]

[0090] Then, the actions are calculated using the respective target networks The target Q value , the update method is as follows:

[0091]

[0092] In order to train the behavior network, it is necessary to minimize the mean square error between the predicted value of the behavior network and the target value. Defined as follows:

[0093]

[0094] Step 6: Model effect evaluation

[0095] The method designed in this invention is compared with existing mainstream algorithms (such as Local, Greedy and DQN algorithms). Figure 4 As shown, although the two-layer decision model of the present invention converges slower than the static optimization solution, it shows better performance in the final convergence result by simultaneously optimizing task offloading and service placement strategies.

[0096] In addition, the two-tier decision model designed by the present invention shows better performance than the comparison algorithm in multiple different scenarios (number of users, number of MEC servers, and changes in service types). Figure 5), as the number of users increases, traditional algorithms such as Local and Greedy often cannot effectively balance the load, resulting in overload of some MEC servers. The two-tier decision model of the present invention can better distribute the load and avoid overload by dynamically optimizing the task offloading path and service placement strategy, thereby maintaining high resource utilization and task completion time.

[0097] In scenarios with different numbers of MEC servers ( Figure 6 ), although the DQN algorithm has good adaptability, the two-layer decision model of the present invention optimizes the joint decision of service placement and task offloading, and can adjust task scheduling more finely in a multi-MEC server environment, thereby improving the overall throughput and response speed of the system. At the same time, in scenarios with different service types ( Figure 7 ), because the present invention can coordinate and optimize among multiple task types and service categories, it can better cope with changes in different service requirements and provide more flexible and efficient decision-making compared to other algorithms.

Claims

1. A method for optimizing dependent task offloading and service placement based on HDQN, characterized in that: The method comprises the following steps: S1: Establish a multi-user multi-MEC system model that relies on task constraints and service placement constraints; S2: Establish an upper-layer service placement optimization model and a lower-layer task offloading optimization model. These two models are responsible for optimizing the service deployment location and task offloading path in the system, respectively. S3: Build a joint optimization model. The upper-level service placement decision guides the lower-level decision, and the lower-level task offloading decision affects the upper-level decision. Through the two-layer decision model, continuous dynamic optimization is performed to obtain the current optimal service placement and task offloading strategy.

2. The HDQN-based dependent task offloading and service placement optimization method according to claim 1 is characterized in that: The step S1 specifically includes the following steps: S1.1: Establish a system model consisting of multiple users, multiple MEC servers, and a cloud server. Consider the collaborative optimization between multiple computing devices. User devices have limited computing and storage resources and can generate tasks and decide whether to offload them to the MEC server for processing based on demand. The MEC server has computing, storage, and network resources and can provide services to multiple user devices. At the same time, service deployment can be dynamically adjusted based on resource status. S1.2: Establish a constraint model for dependent task offloading and service placement. Tasks form a directed acyclic graph (DAG) through dependency relationships. The execution of each task depends on the completion of the predecessor task. Service placement constraints include the computing and storage resource limits of each MEC server. The offloading path selection must meet the resource availability of the MEC server. S1.3: Establish a computing model. The computing model consists of local latency, edge computing latency, cloud computing latency, and service migration latency. The computing model consists of local latency, edge computing latency, cloud computing latency, and service migration latency.

3. The HDQN-based dependent task offloading and service placement optimization method according to claim 1 is characterized in that: The step S2 specifically includes the following steps: S2.1: Establish an upper-layer service placement optimization model. The input of the upper-layer decision model is the task offloading strategy and MEC environment information of the previous time slot. This information is converted into a high-dimensional feature representation through a feature extraction model and used as the input of the DDQN network. The specific state space is as follows: ; The action space of the upper-level decision-making includes different service placement strategies, that is, determining which services should be deployed on which virtual machines. The specific action space is as follows: ; Service placement reward function Calculated based on the migration time of the service. The longer the migration time, the lower the reward. ; S2.2: Establish a lower-level task offloading decision model. The main task of the lower-level decision model is to select an offloading path based on the input service placement strategy and task characteristics. The lower-level model needs to optimize task scheduling and execution order, and schedule multiple tasks to meet task dependencies and deadlines. Scheduling time using task offloading To represent the optimized time slot for task offloading, define Represents the computing capabilities of all VMs, which are defined as follows: ; State space for task offloading It can be expressed as: ; The action space describes the All possible unloading strategies that the system may adopt can be expressed as: ; The design of the reward function is shown in the following formula: ; in Indicates that the task is being performed Previous completion time, Indicates that the task is being performed After completion time, Defined as user Average time to process all tasks locally.

4. The HDQN-based dependent task offloading and service placement optimization method according to claim 1 is characterized in that: The step S3 specifically includes the following steps: S3.1: Build a joint optimization model. Upper-level service placement decisions guide lower-level decisions, while lower-level task offloading decisions influence upper-level decisions. Through continuous dynamic optimization of the two-layer decision model, the optimal service placement and task offloading strategy is determined. S3.2: Decision-making time at the top level When the agent obtains the current upper state from the environment , the upper behavior network selects an action based on the current state , and perform the action, and then obtain immediate rewards through environmental feedback After each time slot, the agent uses the target Q network to calculate the target value and updates the parameters of the behavior network through gradient descent to optimize the action selection strategy. S3.3 Decision time at the corresponding decision-making level When the lower layer task offloading network is based on the upper layer action Update your status , and conduct independent training; the lower-level network selects the appropriate offloading strategy based on the current service placement strategy, task requirements, computing resources and network status; the lower-level task offloading decision It will also affect the upper-layer service placement strategy, because the choice of offloading tasks to MEC servers or cloud servers will directly affect the location of service deployment and resource allocation; the training of task offloading strategy also adopts Q learning method, at each scheduling time In the network, the Q value of the task offloading is updated through immediate feedback; the agent optimizes the offloading decision in the lower network to ensure that the task can be executed on the most suitable device, thereby optimizing the service task strategy; S3.4 uses the DDQN network for optimization decision-making in both the upper and lower layers. Their training and optimization methods are similar, so the following general formula calculation method is given. The optimal actions for the upper and lower layers are selected as follows: ; Then, the actions are calculated using the respective target networks The target Q value , the update method is as follows: ; In order to train the behavior network, it is necessary to minimize the mean square error between the predicted value of the behavior network and the target value. The loss function Defined as follows: ; in Indicates the batch size of each network training, and then updates the parameters of each network according to the gradient descent , every fixed time step, the weight parameters of the behavior network are copied to the target network.

Images