A multi-split and multi-hop task offloading system based on deep reinforcement learning

Through deep reinforcement learning technology, data segmentation and edge server resource allocation of IoT devices are optimized, and the problem of insufficient computing power of IoT devices is solved, load balancing and efficient computing resource utilization are achieved, and computing needs and energy consumption optimization are met.

CN118590940BActive Publication Date: 2024-11-12HUNAN XISAI NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411068734.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-12
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

The computing power of modern IoT devices is not enough to meet the needs of emerging applications, resulting in excessive load load on edge servers and requires load balancing through multi-hop task offloading to meet computing needs.

Method used

Adopting a multi-segment multi-hop task offloading system based on deep reinforcement learning, through information collection, optimized time division, adaptive processing analysis and processing result output modules, the data blocks are allocated using the PPO network and DRL agent to optimize edge server resource allocation and data transmission. path to realize load balancing and efficient utilization of computing resources.

Benefits of technology

It effectively reduces the average energy consumption of IoT devices, meets the long-term optimization of task deadlines, and improves the service coverage of edge networks and the throughput of computing networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118590940B_ABST
    Figure CN118590940B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-segmentation multi-hop task offloading system based on deep reinforcement learning. The present invention relates to the technical field of wireless network resource allocation, and solves the technical problem of long-term optimization of minimizing the average energy consumption of all service IoT devices and meeting their task deadlines. The present invention transmits data to a remote edge server through multi-hop communication. In addition, the present invention enables IoT devices to divide their data into multiple fragments, each of which is assigned a task execution path, from an edge server directly connected to an IoT device to an edge server responsible for processing the data segment of the specific task. The number of multi-hop steps and the number of data cut blocks can be adaptively adjusted according to the network speed between the edge servers, which greatly improves the load balancing of the edge servers and can increase the throughput of the computing network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of wireless network resource allocation, and in particular to a multi-segmentation and multi-hop task offloading system based on deep reinforcement learning. Background Art

[0002] As the Internet of Things (IoT) and Artificial Intelligence (AI) technologies become more prevalent, the concepts of data and computational intelligence are being widely applied to various use cases, including smart cities and Industry 4.0. This is accompanied by an explosion of diverse IoT data from various types of applications (e.g., smart homes, smart transportation, smart security, etc.). It is predicted that by 2025, the core IoT sector will connect approximately 75.44 billion IoT devices to the Internet, generating approximately 80ZB of data. At the same time, AI algorithms have made great progress in the same period, from pioneering small models such as AlexNet, ResNet, and VGG to today's widely recognized large language models (LLMs) such as LLaMA, Bloom, and GPT-4. As a result, the training of AI models has rapidly shifted from traditional centralized single-machine training to the innovative federated learning paradigm, a novel distributed training approach.

[0003] However, as a well-known problem, the computing power of modern IoT devices lags behind the computing needs of emerging applications, which are usually computationally demanding. With the increasing concern about personal privacy data protection since 2016, traditional cloud computing has been gradually replaced by edge computing, a faster, more reliable and more secure computing paradigm. Subsequently, cloud service providers have launched their own edge solutions. Today, edge-based applications can be seen everywhere, especially in urban areas, such as autonomous driving, traffic management, smart grids, etc.

[0004] However, edge devices or servers usually have significantly less computing resources compared to cloud computing, often by an order of magnitude. As a result, a single edge server tends to be quickly overwhelmed by the computing demands of ultra-dense IoT devices in its vicinity. To address this challenge, collaboration between edge servers using multi-hop task offloading is critical to meet the growing computing demands. This inter-server collaboration aims to achieve load balancing, thereby improving the management of unbalanced data (or IoT devices) distribution in dynamic edge environments. This will also help expand the service coverage of edge networks. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a multi-split and multi-hop task offloading system based on deep reinforcement learning, which solves the long-term optimization problem of minimizing the average energy consumption of all service IoT devices and meeting their task deadlines.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multi-segmentation and multi-hop task offloading system based on deep reinforcement learning, comprising:

[0007] An information acquisition module, used for transmitting the acquired time scale information to the optimized time division unit;

[0008] The optimized time division module is used to analyze the acquired time scale information, obtain multiple groups of time periods by dividing the time line, and obtain time slots by dividing the time periods. At the same time, the time slots are divided into the training phase q1 and the execution phase q2, and then multi-hop multi-cutting and resource allocation processing are performed to obtain data blocks, and the data blocks are transmitted to the adaptive processing and analysis module;

[0009] The adaptive processing and analysis module is used to perform DRL agent allocation on the acquired data blocks to obtain allocation information, and define the state of the PPO network, the network's actions and the network's rewards. At the same time, the path of the data segment is constructed according to the parameters of the IoT device, and the subgradient method is applied to derive the optimal resource allocation strategy of the edge server, and configure the optimal local computing speed and transmission power of the IoT device to generate the processing result, and then transmit the processing result to the processing result output module;

[0010] The processing result output module is used to display the obtained processing results to the operator.

[0011] As a further solution of the present invention: the optimized time division module also includes a task feature calculation unit and a subtask execution unit;

[0012] A task feature calculation unit, used to calculate the features of the computing tasks performed by the IoT device;

[0013] The subtask execution unit is used to analyze the transmission path of the subtask and generate path transmission information.

[0014] As a further solution of the present invention: the specific manner in which the optimized time division module divides the timeline is as follows:

[0015] Obtain the timeline in the time scale information, divide the timeline into large time periods and label them as i, where i=1, 2, ..., i1, and i1 is a positive integer, then divide the time period i into time slots and label them as t, where t=1, 2, ..., tmax, and tmax is a positive integer;

[0016] Then the time slot g is divided into the training phase q1 and the execution phase q2, and the length of the training phase q1 is recorded as Z, and the length of the execution phase q2 is recorded as -Z, the system will look for high-quality multi-hop multi-cut and resource allocation solutions during the training phase q1, and select the best solution at the end of the training phase , and in the execution phase q2, the best solution will be executed .

[0017] As a further solution of the present invention: the specific method in which the task feature calculation unit calculates the task feature is as follows:

[0018] The amount of data for the computing task to be performed by the nth IoT device is recorded as , the computational complexity is recorded as and use The task execution period is represented as Ln, and the ID of the IoT device is represented as , and n represents the nth IoT device;

[0019] Keep the No change, each task needs to be The task is completed within the time and the calculated time is used as the task feature, and task feature information is generated at the same time.

[0020] As a further solution of the present invention: the specific manner in which the subtask execution unit generates the path transmission information is as follows:

[0021] The ID of the edge server is represented by , and h represents the hth edge server. At the same time, the computing task data of each IoT device n is cut into data blocks, and each data block corresponds to a subtask of an edge server. Then, corresponding to a single subtask, a data multi-hop transmission path is constructed to generate path transmission information.

[0022] As a further solution of the present invention: the specific definition of the state of the PPO network, the action of the network and the reward of the network by the adaptive processing module is as follows:

[0023] Definition of network status: And the relationship in the PPO network state represents from top to bottom the location of the IoT device with the edge server within timestamp t, the number of computing resources and network resources available to the edge server, the connection status with each edge server, the data cutting status, the computing task characteristics, the network transmission power and computing power, the number of computing resources allocated to the IoT device by the server and the network resource allocation, the task completion time and the energy consumption of the IoT device, and the task completion degree;

[0024] The actions of the PPO network are defined as: And from top to bottom, they represent the connection selection and data cutting of each edge server. The present invention sets a variable K to limit the maximum number of times data can be cut. Then according to The size of the edge servers is used to sort them, and the largest K edge servers are selected. The k+1th data block is assigned to The k-th edge server in performs remote computing;

[0025] The reward of the PPO network is defined as: in represents the energy consumption of the nth IoT device, Indicates the task completion of the entire system;

[0026] The state of the PPO network, the actions of the network, and the rewards of the network are comprehensively defined, and the data block is brought into the DRL agent to obtain the distribution information.

[0027] As a further solution of the present invention: the specific manner in which the adaptive processing module analyzes the allocation information is:

[0028] At the end of the last time slot t-1, collect the device information corresponding to each IoT device, bring the device information into the DRL agent, and return the agent to the solution of time slot t;

[0029] According to the division of subtasks, the computing speed of a single IoT device is calculated, and the calculation formula is: ,and represents the maximum computational speed, and Represented as a data block;

[0030] Based on the breadth search algorithm, a multi-hop data transmission path is found for each subtask, which is recorded as: , and calculate the data transmission time of each subtask according to the data transmission path and the transmission rate between the edge servers, which is recorded as: ;

[0031] Allocate the subtasks of IoT devices in proportion as follows: where n and are the IDs of any two IoT devices. It is a built-in parameter of the system, and the calculation time of a single IoT device computing subtask is calculated according to the formula ;

[0032] Based on the calculation time of the subtask Calculate the data transmission time. The specific calculation method is: .

[0033] As a further solution of the present invention: the specific method of analyzing the data transmission time is:

[0034] When the data transmission time is greater than zero, the minimum transmission rate is calculated based on the data transmission time. The specific calculation method is: in ,and is the data transmission speed, given by Divide by get;

[0035] When the data transmission time is less than zero, the transmission rate is calculated based on the data transmission time, and the IoT device sets its transmission power to the maximum value , the specific calculation method is:

[0036] ;

[0037] Based on the transmission rate, the data transmission time between the IoT device and the directly connected edge server is calculated: ;

[0038] The specific method of recalculation is: .

[0039] As a further solution of the present invention: The specific way of analysis is: when there is When Updates: After the update is completed, the resources are reallocated, otherwise the next step is executed, and so on, all IoT devices perform the training phase and generate processing results.

[0040] The present invention provides a multi-segmentation and multi-hop task offloading system based on deep reinforcement learning. Compared with the prior art, it has the following beneficial effects:

[0041] The present invention can transmit task data to a remote edge server through multi-hop communication. In addition, the present invention enables the IoT device to divide its data into multiple fragments, each fragment is assigned a task execution path, from the edge server directly connected to a certain IoT device to the edge server responsible for processing the specific task data segment, and can adaptively adjust the number of multi-hop steps and the number of data cut blocks according to the network speed between the edge servers, which greatly improves the load balancing of the edge servers and can increase the throughput of the computing network. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a block diagram of the system principle of the present invention;

[0043] Figure 2 A time partition diagram is optimized for the system of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1

[0046] See also Figure 1 to Figure 2 The present application provides a multi-split and multi-hop task offloading system based on deep reinforcement learning, including an information collection module, an optimized time division module, an adaptive processing module and a processing result output module.

[0047] The information acquisition module is used to obtain time scale information and transmit the obtained time scale information to the optimized time division unit.

[0048] Optimize the time division module. This functional module is used to analyze the acquired time scale information, and the specific analysis method is as follows:

[0049] Get the timeline in the time scale information and divide the timeline into large time periods. , and the time period usually lasts for one minute or several minutes. The divided time periods are labeled as i, and i=1, 2, ..., i1, and i1 is a positive integer;

[0050] Then each time period i is further divided into tmax smaller time slots , and the time slot is recorded as t, and t=1, 2, ..., tmax, and tmax is a positive integer. Usually the time slot lasts from one second to several seconds;

[0051] Finally, the time slot g of each time period T is divided into a training phase q1 (the length of the training phase q1 is recorded as Z) and an execution phase q2 (the length of the execution phase q1 is recorded as -Z), in the training phase q1, the system will continuously search for high-quality multi-hop multi-cut and resource allocation solutions, and select the best solution at the end of the training phase .

[0052] In the execution phase q2, the system will always execute this optimal solution. ,Until the end of the current time slot, an IoT device can only connect to at most one edge server in any time slot within any given time period.

[0053] Embodiment 2

[0054] At the same time, the time division optimization module also includes a task feature calculation unit and a subtask execution unit, and the task feature calculation unit is used to calculate the computing task features executed by the IoT device, and the specific calculation method is as follows:

[0055] use To represent the amount of data for the computing task to be performed by the nth IoT device (e.g. 2MB) and computational complexity (e.g. 2MB / s), Ln represents the execution period of the task (e.g. 1 second), They represent the IDs of IoT devices respectively, and n represents the nth IoT device. In each time period T, it remains unchanged, but in different time periods, it may change, and each task needs to be re- The task is completed within the time and the calculated time is used as the task feature. At the same time, the task feature information is generated and transmitted to the adaptive processing module.

[0056] Subtask execution unit, this functional unit is used to analyze the subtask and generate the path transmission information of the subtask, and the specific analysis method is as follows:

[0057] The ID of the edge server is represented by , and h represents the hth edge server. Then, the computing task data of each IoT device n is cut into multiple data blocks and recorded as , and the multiple data blocks obtained at the same time satisfy the constraints , each data block The system identifies it as a subtask that will be assigned to the hth edge server. For each subtask, a multi-hop data transmission path (denoted as ) to remotely execute the subtask and generate path transmission information at the same time, and transmit the path transmission information to the adaptive processing and analysis module.

[0058] Embodiment 3

[0059] Adaptive processing and analysis module: This functional module is used to analyze and process according to the acquired path transmission information and task feature information, and perform DRL agent allocation to obtain allocation information, and then perform secondary analysis and processing on the allocation information. The specific analysis method is as follows:

[0060] The state of the PPO network is defined as: And the relationship in the PPO network status represents, from top to bottom, the location of the IoT device with the edge server within timestamp t, the amount of computing resources and network resources available to the edge server, the connection status with each edge server, the data cutting status, the computing task characteristics, the network transmission power and computing power, the amount of computing resources allocated to the IoT device by the server and the allocation of network resources, the task completion time and the energy consumption of the IoT device, and the task completion degree.

[0061] The action of the PPO network is defined as: And from top to bottom, they represent the connection selection and data cutting of each edge server. The present invention sets a variable K to limit the maximum number of times data can be cut. Then according to The size of the edge servers is used to sort them, and the largest K edge servers are selected. The k+1th data block is assigned to The k-th edge server in performs remote computing.

[0062] The reward of the PPO network is defined as: in represents the energy consumption of the nth IoT device, Indicates the task completion degree of the entire system.

[0063] In addition, combined with the actual description, the present invention defines the structure and parameters of the PPO network as follows: 3 layers, 128 neural network parameters in each layer, a learning rate of 0.0001, a parameter update frequency of 300 time slots, each update is repeated 10 times, and a discount factor of 0.9.

[0064] After the above analysis, the allocation information is obtained, and the allocation information is further subjected to secondary analysis and processing, and the specific processing method is as follows:

[0065] S1: Repeat the above steps in each time period T, and the specific repetition method is: at the end of the previous time slot t-1, each IoT device collects corresponding information, and the collected information includes: the location of the IoT device and the edge server , computing resources at IoT devices and edge server locations and the number of network resources , the connection status between the device and each edge server , data cutting situation , the network transmission power and computing power of current IoT devices , the total data volume and computational complexity of current IoT device computing tasks , the number of computing resources currently allocated to the IoT device by each server , for each edge server h that satisfies the constraint , and finally the task completion time and the power consumption of the IoT device ;

[0066] S2: Input the above information of each IoT device into the DRL agent, and the agent returns the solution for the current time slot t:

[0067] Each IoT device needs to directly connect to the edge server ID, i.e. , data cutting , according to the data cutting, implement the subtasks, and the DRL agent allocation step is the same here;

[0068] S3: According to the division of subtasks, the computing speed of each IoT device is calculated according to the following formula: Indicates the maximum calculation speed;

[0069] S4: Based on the breadth search algorithm, find a multi-hop data transmission path for each subtask, denoted as: Based on the data transmission path and the transmission rate between the edge servers, the data transmission time of each subtask is calculated and recorded as ;

[0070] S5: Each edge server h distributes its computing tasks according to the following proportions based on the subtasks it undertakes from various IoT devices: where n and are the IDs of any two IoT devices. It is a built-in parameter of the system;

[0071] S6: The computing time of each IoT device computing subtask is: ;

[0072] S7: Each IoT device calculates the available data transmission time between it and the directly connected edge server: ;

[0073] S8: If Greater than 0, each IoT device Calculate the minimum required transfer rate at: in ,and is the data transmission speed, given by Divide by get;

[0074] S9: If Less than or equal to 0, each IoT device sets its transmission power to the maximum value . Calculate the transmission rate according to the following formula: Based on the transmission rate, the data transmission time between the IoT device and the directly connected edge server is calculated: Recalculate: ;

[0075] S10: When there is When Updates: After the update is completed, jump to step S5 to reallocate resources, otherwise proceed to the next step.

[0076] S11: Each IoT device executes according to the above steps S1-S10 until the training phase of the current time period ends.

[0077] S12: Record each action in the training phase and the reward returned by the system, and find the action plan that returns the maximum reward according to the following formula: .

[0078] S13: For each time slot of the execution phase within each time period T, use the best solution selected in step 12 , until the end of the time period.

[0079] S14: Every experience time period, reducing the duration of the training phase by 10, that is, Z=Z-10.

[0080] S15: Enter the next time period, T=T+1, and jump to the first step.

[0081] S16: After the above steps, the processing result is obtained and transmitted to the processing result output module.

[0082] The processing result output module is used to display the obtained processing results to the operator.

[0083] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0084] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multi-split multi-hop task offloading system based on deep reinforcement learning, characterized in that: include: An information acquisition module, used for transmitting the acquired time scale information to the optimized time division module; The optimized time division module is used to analyze the acquired time scale information, obtain multiple groups of time periods by dividing the time line, and obtain time slots by dividing the time periods. At the same time, the time slots are divided into the training phase q1 and the execution phase q2, and then multi-hop multi-cutting and resource allocation processing are performed to obtain data blocks, and the data blocks are transmitted to the adaptive processing and analysis module. The specific method of dividing the time line is as follows: Obtain the timeline in the time scale information, divide the timeline into large time periods and label them as i, where i=1, 2, ..., i1, and i1 is a positive integer, then divide the time period i into time slots and label them as t, where t=1, 2, ..., g, and g is a positive integer; Then the time slot g is divided into the training phase q1 and the execution phase q2, and the length of the training phase q1 is recorded as Z, and the length of the execution phase q2 is recorded as -Z, and Indicates the length of the execution phase. During the training phase q1, the system will look for high-quality multi-hop and multi-cut and resource allocation solutions, and select the best solution at the end of the training phase. , and in the execution phase q2, the best solution will be executed ; The task feature calculation unit is used to calculate the computing task features performed by the IoT device. The specific method of calculating the task features is as follows: The amount of data for the computing task to be performed by the nth IoT device is recorded as , and the computational complexity is expressed as and use The task execution period is represented as Ln, and the ID of the IoT device is represented as , and n represents the nth IoT device; Keep the No change, each task needs to be The task is completed within the time and the calculated time is used as the task feature, and task feature information is generated at the same time; The subtask execution unit is used to analyze the transmission path of the subtask and generate path transmission information. The specific method of generating the path transmission information is as follows: The ID of the edge server is represented by , and h represents the hth edge server. At the same time, the computing task data of each IoT device n is cut into data blocks, and each data block corresponds to a subtask of an edge server. Then, corresponding to a single subtask, a data multi-hop transmission path is constructed to generate path transmission information; The adaptive processing and analysis module is used to perform DRL agent allocation on the acquired data blocks to obtain allocation information, and define the state of the PPO network, the network's actions and the network's rewards. At the same time, the path of the data segment is constructed according to the parameters of the IoT device, and the subgradient method is applied to derive the optimal resource allocation strategy of the edge server, and configure the optimal local computing speed and transmission power of the IoT device to generate the processing result, and then transmit the processing result to the processing result output module. The specific definition of the state of the PPO network, the network's actions and the network's rewards is as follows: Definition of network status: ; And the relationship in the PPO network state represents from top to bottom the location of the IoT device with the edge server within timestamp t, the number of computing resources and network resources available to the edge server, the connection status with each edge server, the data cutting status, the computing task characteristics, the network transmission power and computing power, the number of computing resources allocated to the IoT device by the server and the network resource allocation, the task completion time and the energy consumption of the IoT device, and the task completion degree; The actions of the PPO network are defined as: ; From top to bottom, they represent the connection selection and data cutting of each edge server. A variable K is set to limit the maximum number of times data can be cut, and then according to The size of the edge servers is used to sort them, and the largest K edge servers are selected. The k+1th data block is assigned to The k-th edge server in performs remote computing; The reward of the PPO network is defined as: in represents the energy consumption of the nth IoT device, Indicates the task completion of the entire system; Comprehensively define the state of the PPO network, the actions of the network, and the rewards of the network, and bring the data block into the DRL agent to obtain the allocation information; The processing result output module is used to display the obtained processing results to the operator.

2. According to the multi-split multi-hop task offloading system based on deep reinforcement learning in claim 1, it is characterized in that: The specific method in which the adaptive processing and analysis module analyzes the allocation information is as follows: At the end of the last time slot t-1, collect the device information corresponding to each IoT device, bring the device information into the DRL agent, and return the agent to the solution of time slot t; According to the division of subtasks, the computing speed of a single IoT device is calculated, and the calculation formula is: ,and represents the maximum computational speed, and Represented as a data block; Based on the breadth search algorithm, a multi-hop data transmission path is found for each subtask, which is recorded as: , and calculate the data transmission time of each subtask according to the data transmission path and the transmission rate between the edge servers, which is recorded as: ; Allocate the subtasks of IoT devices in proportion as follows: where n and are the IDs of any two IoT devices. It is a built-in parameter of the system, and the calculation time of a single IoT device computing subtask is calculated according to the formula ; Based on the calculation time of the subtask Calculate the data transmission time. The specific calculation method is: 。 3. According to the multi-split multi-hop task offloading system based on deep reinforcement learning in claim 2, it is characterized in that: The specific method for analyzing data transmission time is as follows: When the data transmission time is greater than zero, the minimum transmission rate is calculated based on the data transmission time. The specific calculation method is: in ,and is the data transmission speed, given by Divide by get; When the data transmission time is less than zero, the transmission rate is calculated based on the data transmission time, and the IoT device sets its transmission power to the maximum value , the specific calculation method is: ; Based on the transmission rate, the data transmission time between the IoT device and the directly connected edge server is calculated: ; The specific method of recalculation is: .

4. According to claim 3, a multi-split multi-hop task offloading system based on deep reinforcement learning is characterized in that: The recalculated The specific way of analysis is: when there is When Updates: After the update is completed, the resources are reallocated, otherwise the next step is executed, and so on, all IoT devices perform the training phase and generate processing results.