Space-air-ground integrated network-oriented slice division and cooperative unloading method
By combining Transformer and improving DRL in the integrated air-space and earth network, dynamically manage slice resources, solving the problem of insufficient or excess resources in the dynamic environment, improving resource utilization efficiency and edge service benefits, and avoiding the convergence of the agent Instability.
Patent Information
- Application Number
- CN202411345059.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-25
AI Technical Summary
In the dynamically changing integrated space-space network environment, it is difficult for the existing technology to effectively manage slice resources, resulting in insufficient or oversupply of resources, affecting service quality and the benefits of edge service providers, and in-depth reinforcement learning has a Q value in resource allocation. The problems of overestimation and excessive variance lead to instability of the agent's convergence or fall into local optimality.
Using Transformer-based slicing resource division method combined with improved deep reinforcement learning (DRL) collaborative offloading and resource allocation methods, dynamically adjust slicing resources, maximize the benefits of edge service providers, and sparsely Self-attention mechanism and dual Q-value network architecture solve the problems of Q-value overestimation and high variance.
It improves system resource utilization efficiency and task completion rate, reduces the system overhead of slice resource division, and realizes more efficient resource management and maximizes the benefits of edge service providers.
Smart Images

Figure CN119071851A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of mobile edge computing technology, and in particular to a slicing division and collaborative unloading method for an air-ground-space integrated network. Background Art
[0002] With the rapid development of artificial intelligence and communication, more and more intelligent applications (such as driverless cars, virtual reality, etc.) are integrated into people's daily lives, which usually consume a lot of computing and storage resources to ensure the quality of service (QoS). However, limited by the computing architecture and energy reserves of mobile devices, the high resource requirements of such intelligent applications are difficult to meet. To address this problem, Mobile Edge Computing (MEC) is regarded as a promising new computing paradigm. In MEC, the MEC server can process the tasks offloaded from the user's mobile device and return the results to the user after completion, so as to better support the complex computing needs of intelligent applications. Compared with traditional cloud computing, MEC significantly reduces the response delay of task requests by deploying computing and storage resources at the edge of the network closer to the user, thereby effectively improving QoS.
[0003] However, the discrete location distribution and diversified service requirements of mobile users have put forward higher requirements for the flexibility of MEC network architecture. The service coverage of the fixed ground access network is limited, and it is difficult to meet the dynamic and changing user service needs. As an emerging network architecture, Space-Air-Ground Integrated Networks (SAGIN) can provide mobile users with a more comprehensive and flexible network access by coordinating air-based and space-based networks on the basis of ground networks. In SAGIN, ground networks (such as base stations) can serve users near them; while air-based and space-based networks are usually composed of satellites and drones, which can provide services to users in remote areas and alleviate the server load pressure in user-dense areas to a certain extent. But at the same time, the heterogeneous node collaboration and diversified resource allocation in SAGIN also increase the difficulty of resource management. Based on network function virtualization and software-defined network technology, network slicing technology aims to divide the physical network into multiple logically independent networks according to user needs, which is a potential feasible solution for achieving efficient heterogeneous resource management. By introducing network slicing, a SAGIN-oriented network architecture can be built, and the edge service provider (ESP) can orchestrate and configure slice resources and deploy different services to appropriate slices to meet the service needs of different users.
[0004] In real SAGIN scenarios, user traffic is complex and changeable, and the number of requests and resource requirements usually change dynamically over time. Static network slicing solutions are difficult to adapt to this dynamic environment, resulting in insufficient or excessive slice resources, which seriously affects QoS and ESP benefits. However, frequent slice resource partitioning operations will also cause additional system overhead. By rationally planning the resources and timing of slice partitioning, resources can be efficiently utilized while reducing the additional overhead caused by frequent slice resource partitioning operations. Therefore, how to design an efficient collaborative offloading and resource allocation mechanism for dynamically changing network slices is a problem worthy of further in-depth study. Existing research works usually use iterative algorithms or control theories, which are difficult to adapt to the dynamic SAGIN environment that requires efficient collaboration between heterogeneous platforms. In the field of machine learning, the emerging deep reinforcement learning (DRL) has shown initial application potential in resource management-related problems, providing new perspectives and methods for solving complex problems. By continuously interacting with the dynamic environment, DRL agents are expected to gradually optimize collaborative offloading and resource allocation decisions, thereby improving system resource utilization efficiency and ESP benefits. Although DRL is a promising solution, existing DRL-based works often suffer from the problem of overestimation of Q-values or excessive variance, which makes it easy for the intelligent agent to experience unstable convergence or fall into local optimality during training. Summary of the invention
[0005] The purpose of the present invention is to provide a slicing partitioning and collaborative unloading method for an air-space-ground integrated network, which is beneficial to improving the system resource utilization efficiency and task completion rate.
[0006] In order to achieve the above object, the technical solution adopted by the present invention is: a slicing division and collaborative unloading method for an air-ground integrated network, comprising the following steps:
[0007] (1) Collect historical user traffic of ESP, predict future user traffic, and calculate the resource requirements required by ESP. ESP performs network slicing at the beginning of each slicing window based on the resource requirements.
[0008] (2) The infrastructure provider responds to the ESP's resource allocation request and allocates network slices for it;
[0009] (3) The user accesses and uploads the computation offloading request to ESP through SAGIN;
[0010] (4) ESP assigns the appropriate edge server to the user based on the user's access location and request type, and allocates communication and computing resources based on task attributes and user priority;
[0011] (5) During the collaborative offloading process, the actions taken, the rewards obtained, and the new states entered are recorded in each time slot, and based on the above information, the system continuously optimizes its own performance.
[0012] Furthermore, the SAGIN is composed of a ground base station BS, a drone and a satellite. Users access the computing offloading service provided by ESP through SAGIN. The entire area covered by SAGIN is divided into three independent areas: an area covered by the BS, an area covered by the drone and an area not covered by the BS and the drone. The BS and the drone provide services to users in different areas respectively, while the satellite can provide services to users in areas not covered by the BS and the drone. The satellite communicates directly with the BS and the drone, while the BS and the drone communicate indirectly through satellite forwarding. All users are denoted as a set U = {u1,u2,...,u N}, the servers equipped by satellite, BS and UAV are defined as L sat , L uav and L bs , all servers are recorded as set L = L sat ∪L uav ∪L bs ={L1,L2,...,L M}; Considering the differences in offloading requests in different regions, server L will be j The number of offload requests received within the coverage area is denoted as N j Therefore, the total number of offload requests that ESP needs to process is equal to the sum of the number of requests in each area, that is,
[0013] The slicing model of SAGIN is:
[0014] The physical resources of satellites, BSs, and drones are arranged into resource-customized network slices; for server L j , and its bandwidth and computing resources are recorded as and The system time slot is divided into W slice windows, each slice window w contains a number of time slots t; the bandwidth and computing resources allocated to ESP in the slice window w are respectively denoted as and At the beginning of each slice window, ESP analyzes the historical requests received by each area to predict future demand and then divides the slice resources appropriately;
[0015] The communication model of the SAGIN is:
[0016] When user u iWhen initiating a computing offloading request to ESP, the process mainly includes four stages, namely, request access, platform collaboration, task execution and result return; the task attribute of ui is recorded as a five-tuple {a i ,c i ,o i ,ρ i ,l i}, where each element represents the amount of data, required computing power, type, priority and u of the task. i The server to which you connect; the type of task, based on the amount of task data and the computing power required i Divided into communication-intensive CMI and computation-intensive CPI; the priority of the task ρ i Indicates the level of reward that can be obtained by completing the task;
[0017] When ESP processes a computation offload request, it needs to consider u i The area where you are located; if u i If the BS or UAV is within its coverage, it will be given priority to access the computing offloading service through the BS or UAV; otherwise, u i Computing offload services can only be accessed via satellite; ESP is allocated to u i The bandwidth is denoted as b i , then u i The task access L j The uplink transmission rate is:
[0018]
[0019] Among them, p i is u i The upload power, g0 and σ 2 u i With L j The channel gain and Gaussian white noise power between u i Task request access L j The time required is:
[0020]
[0021] When the task is uploaded to L j After that, ESP will i The access location and task type are assigned to the corresponding server L j' ; when u i When computing offloading services are accessed through BS, both CMI and CPI tasks are executed on BS; i When computing offloading services are accessed through drones or satellites, CMI tasks are directly executed on the drones or satellites, while CPI tasks are forwarded to the BS with more abundant computing resources for execution; accordingly, u iThe task data from L j Forward to L j' The required transfer time is:
[0022]
[0023] Among them, R u2s represents the data transmission rate between the UAV and the satellite, R s2b Indicates the data transmission rate between BS and satellite;
[0024] The calculation model of SAGIN is:
[0025] When the user task is offloaded to the corresponding server, ESP allocates computing resources to it; user u i The CPU power allocated to the task is recorded as f i , then the time required to execute the task is:
[0026]
[0027] After the task is completed, the user will receive feedback on the results;
[0028] Complete i The total time for the task is:
[0029]
[0030] The revenue model of SAGIN is:
[0031] If the total time to complete the task is less than its maximum tolerable delay, the ESP will receive a unit reward Φ; otherwise, the ESP will not receive any reward; therefore, at time t, the ESP will receive a unit reward Φ from u i The rewards obtained are:
[0032]
[0033] At the same time, considering the differences in priorities of different tasks, the reward for completing high-priority tasks will be relatively higher; therefore, at time slot t, the total reward obtained by ESP for completing tasks is:
[0034]
[0035] In addition, ESP provides computing offloading services at a certain cost, which is related to the number of resources actually used by the task; therefore, in time slot t, the total resource rental cost of ESP is:
[0036]
[0037] in, and denote the unit rental price of bandwidth and computing resources respectively; therefore, at time slot t, the revenue obtained by ESP is expressed as R t -C t .
[0038] Furthermore, based on the slicing model, communication model, computing model and revenue model of the SAGIN, it is considered to divide the slice resources at the beginning of each slice window, and execute the corresponding cooperative offloading strategy and communication and computing resource allocation decision in each time slot to maximize the cumulative total revenue of ESP; therefore, the optimization problem is defined as:
[0039]
[0040] Among them, the constraints C1 and C2 represent L j The communication and computing resources rented by the servers in the slice window w do not exceed L j The total resources owned; Constraints C3 and C4 respectively indicate that the communication and computing resources allocated to users in each time slot do not exceed the total resources rented in the slice window; The optimization problem is decomposed into two sub-problems on the corresponding time slots: slice partitioning and collaborative offloading:
[0041] P1: Determine the appropriate slice resource allocation strategy to maximize the long-term benefits of ESP. This sub-problem is defined as:
[0042]
[0043] By deciding on resource allocation in each slice window to solve P1, the long-term benefits of ESP are maximized. Considering that in the real edge environment, it is difficult for ESP to accurately obtain the actual resource requirements of users, historical user request traffic is analyzed and converted into ESP resource requirements.
[0044] P2: Perform cooperative offloading and resource allocation to maximize the short-term benefits of ESP. This sub-problem is defined as:
[0045]
[0046] The goal of P2 is to transfer different types of tasks to appropriate servers through cooperative offloading in each time slot t, and then maximize the benefits of ESP in the current time slot through appropriate resource allocation; since the tasks and the number of requests to be processed in each time slot are different, ESP needs to adjust and optimize cooperative offloading and resource allocation decisions to meet the resource requirements of user tasks;
[0047] The slice partitioning and collaborative offloading method combines the Transformer-based slice resource partitioning method and the improved DRL-based collaborative offloading and resource allocation method. First, the Transformer is used to predict the user traffic in the future time slot, and then guide the ESP to partition the slice resources. Then, by solving the Q value overestimation and high variance problems, the collaborative offloading and resource allocation are optimized to maximize the benefits of the ESP.
[0048] Furthermore, the implementation method of the Transformer-based slice resource partitioning method is:
[0049] First, the inputs of the encoder and decoder are constructed through the user's historical request traffic. Sparse probabilistic self-attention is used when constructing the encoder, and self-attention distillation is used between layers to reduce computational overhead. The feature extraction process from the jth layer to the j+1th layer is expressed as:
[0050]
[0051]
[0052] in,[·] attention represents sparse self-attention, d is X l t Dimension, Conv1d represents one-dimensional convolution, ELU is the activation function, and MaxPool is the maximum pooling;
[0053] Next, the output of the encoder is sent to the decoder, which consists of a layer of multi-head sparse probabilistic self-attention mechanism and a layer of multi-head attention mechanism; then, the output of the decoder is sent to the MLP, and the predicted sequence of future traffic is obtained through reasoning; then, the maximum value of the time slot predicted traffic is taken as the predicted traffic in the next slice window, and it is classified according to the task type and access platform; the communication resource demand of each platform is the product of the unit communication resource demand of all types of tasks within its coverage range and the corresponding type of request traffic, that is, where b CMI and b CPI Respectively represent the average communication bandwidth required for CMI and CPI tasks, and They represent the predicted CMI and CPI task traffic respectively; the computing resources required for the CMI task within the BS coverage area are The CPI tasks in other regions are where f CMI and f CPI They represent the average computing resource requirements for CMI and CPI tasks respectively; satellites and UAVs forward CPI tasks within their coverage to BS for execution, so the computing resource requirements of BS are UAVs and satellites only need to process CMI tasks within their coverage area, so their computing resource requirements are and
[0054] Furthermore, after completing the slice resource division within the slice window, the unloading process between different slice windows is relatively independent; therefore, the long-term optimization problem of the maximum ESP cumulative benefit in P2 is transformed into a short-term optimization problem within a single slice window, and the problem is constructed as a Markov decision process; SAGIN is regarded as the environment, and ESP is abstracted as a DRL agent, which makes decisions on collaborative unloading and resource allocation by interacting with the environment, and then updates the strategy through the reward signal fed back by the environment; the state space, action space and reward function are defined as follows;
[0055] (1) State space: In time slot t, by sensing the available communication and computing resources and task attribute information, the agent captures the task requirements and the corresponding resources required; therefore, the system state in time slot t is expressed as:
[0056]
[0057] in,
[0058] (2) Action space: The action space contains the actions of allocating communication and computing resources to users; therefore, the action at time slot t is expressed as:
[0059] a t = {b t ,f t} (15)
[0060] in,
[0061] (3) Reward function: Problem P2 aims to maximize the short-term benefit of ESP in each time slot; therefore, the reward function is defined as the benefit of ESP at time slot t, which is expressed as:
[0062] r t =R t -C t (16)
[0063] The implementation method of the collaborative offloading and resource allocation method based on the improved DRL is:
[0064] First, the main network Q1, Q2 and μ and the target network Q'1, Q'2 and μ' are initialized; two Q-value networks with the same structure are introduced, and the minimum value is used to estimate the value of the next state-action pair when calculating the target value; then, the experience replay buffer RB, the number of slice windows W and the number of time slots T in each slice window are initialized; in order to reduce the correlation between experience data and improve the overall training efficiency, the experience replay mechanism is introduced; at the beginning of the training cycle, the system environment is initialized and its initial state is obtained; at the beginning of each slice window, the user traffic in the future time slot is predicted and the slice resource partitioning is performed by the Transformer-based slice resource partitioning method; at time slot t, an action is selected according to the policy network and the exploration noise; then, after executing the action and obtaining the immediate reward and next state of the environment feedback; then, the training samples (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer RB; then, K samples are randomly taken from RB to update the network parameters; based on the target strategy smoothing regularization, the target action is obtained using the target strategy network The process is expressed as:
[0065]
[0066] The noise follows a normal distribution and the sampled noise is clipped to reduce the variance of the target estimate. Then, the target Q value y target Based on the minimum value of the two Q-value networks, the update process is expressed as:
[0067]
[0068] Finally, the gradient descent algorithm is used to minimize the error between the evaluation value and the target value, and then the Q-value network parameters are updated; when the Q-value network is updated a certain number of times, the strategy network parameters and target network parameters are updated using gradient ascent and soft update respectively.
[0069] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a novel slice partitioning and collaborative offloading method for an integrated air-ground-space network to solve the problems existing in the prior art; within the service scope of SAGIN, users can access the customized services provided by ESP and transmit them to available edge servers; based on the analysis of historical user traffic change trends, ESP reasonably divides slice resources; in order to achieve collaborative offloading and optimization of resource allocation strategies, DRL is introduced to interact with the dynamic SAGIN, and decisions are made with the optimization goal of maximizing the long-term cumulative benefits of ESP, thereby improving the system resource utilization efficiency and task completion rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 is a schematic diagram of an air-space-ground integrated network according to an embodiment of the present invention;
[0071] Figure 2 is a schematic diagram of a network slicing model in an embodiment of the present invention;
[0072] Figure 3 is a schematic diagram of collaborative offloading and resource allocation based on improved DRL in an embodiment of the present invention;
[0073] Figure 4 is the prediction performance of the method for user traffic in the embodiment of the present invention;
[0074] Figure 5 It is a comparison of the convergence of the method in the embodiment of the present invention and other methods;
[0075] Figure 6 is a comparison of ESP benefits and costs of different methods in the embodiments of the present invention;
[0076] Figure 7 is a comparison of task failure rates of different methods in the embodiments of the present invention;
[0077] Figure 8 It is a comparison of resource utilization rates of different methods in the embodiments of the present invention. DETAILED DESCRIPTION
[0078] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0079] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0080] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0081] This embodiment provides a slice partitioning and collaborative offloading method (Slice Partitioning and Collaborative Offloading, SPCO) for an air-ground integrated network, including the following steps:
[0082] (1) Collect historical user traffic of ESP, predict future user traffic, and calculate the resource requirements required by ESP. ESP performs network slicing at the beginning of each slicing window based on the resource requirements.
[0083] (2) The infrastructure provider responds to the ESP's resource allocation request and allocates network slices for it;
[0084] (3) The user accesses and uploads the computation offloading request to ESP through SAGIN;
[0085] (4) ESP assigns the appropriate edge server to the user based on the user's access location and request type, and allocates communication and computing resources based on task attributes (such as task data size, computing requirements, maximum tolerable delay, etc.) and user priority;
[0086] (5) During the collaborative offloading process, the actions taken, the rewards obtained, and the new states entered are recorded in each time slot, and based on the above information, the system continuously optimizes its own performance.
[0087] 1. System model and problem definition
[0088] like Figure 1 As shown in the figure, the SAGIN considered in this method is composed of a ground base station (BS), a drone and a satellite. Users access the computing offloading service provided by ESP through SAGIN. The entire area covered by SAGIN is divided into three independent areas: the area covered by the BS, the area covered by the drone, and the area not covered by the BS and the drone. The BS and the drone provide services to users in different areas respectively, while the satellite can provide services to users in areas not covered by the BS and the drone. The satellite communicates directly with the BS and the drone, while the BS and the drone communicate indirectly through the satellite forwarding. Specifically, all users are recorded as a set U = {u1,u2,...,u N The servers equipped by satellite, BS and UAV are defined as L sat , L uav and L bs , all servers are recorded as set L = L sat ∪L uav ∪L bs ={L1,L2,...,L M Considering the differences in offloading requests in different regions, server L j The number of offload requests received within the coverage area is denoted as N j Therefore, the total number of offload requests that ESP needs to process is equal to the sum of the number of requests in each area, that is, Slice Model
[0089] like Figure 2 As shown in Figure 1, according to the characteristics of each SAGIN platform, the physical resources of satellites, BSs, and drones are arranged into resource-customized network slices. j , whose bandwidth and computing resources are recorded as and The system time slot is divided into W slice windows, each slice window w contains a number of time slots t. The bandwidth and computing resources allocated to ESP in the slice window w are respectively and Specifically, at the beginning of each slice window, ESP analyzes the historical requests received by each area to predict future demand and then divides the slice resources appropriately.
[0090] communication model
[0091] When user u i When a computation offloading request is initiated to ESP, the process mainly includes four stages, namely, request access, platform collaboration, task execution, and result return. The task attribute of ui is recorded as a five-tuple {a i ,c i ,o i ,ρ i ,l i}, where each element represents the amount of data, required computing power, type, priority and u of the task. i The server to which you are connected. The type of task is determined by the amount of task data and the computing power required. i It is divided into communication-intensive (CMI) and computation-intensive (CPI). In addition, the priority of the task ρ i Indicates the level of reward that can be obtained by completing the task.
[0092] When ESP processes a computation offload request, it needs to consider u i The area where you are located. i If the BS or UAV is within its coverage, it will be given priority to access the computing offloading service through the BS or UAV; otherwise, u i Computing offload services can only be accessed via satellite. ESP is assigned to u i The bandwidth is denoted as b i , then u i The task access L j The uplink transmission rate is:
[0093]
[0094] Among them, p i is u iThe upload power, g0 and σ 2 u i With L j The channel gain and Gaussian white noise power between u i Task request access L j The time required is:
[0095]
[0096] When the task is uploaded to L j After that, ESP will i The access location and task type are assigned to the corresponding server L j' Compared with satellites and drones, the communication and computing resources provided by BS have higher cost performance and stability. i When accessing the computing offloading service through the BS, both CMI and CPI tasks will be executed on the BS. Compared with the BS, drones and satellites can provide a wider coverage and more flexible communication capabilities, but they are limited by their resource capacity and cannot meet all computing offloading service requirements. i When accessing computing offloading services through drones or satellites, CMI tasks will be executed directly on the drones or satellites, while CPI tasks will be forwarded to BSs with more abundant computing resources. i The task data from L j Forward to L j' The required transfer time is:
[0097]
[0098] Among them, R u2s represents the data transmission rate between the UAV and the satellite, R s2b Indicates the data transmission rate between BS and satellite.
[0099] Computational Model
[0100] When the user task is offloaded to the corresponding server, ESP will allocate computing resources to it. i The CPU power allocated to the task is recorded as f i , then the time required to execute the task is:
[0101]
[0102] After the task is completed, the user will receive the result feedback. Usually, the amount of data returned is small, and its time cost is far less than other stages of the offloading process, so the time cost of returning the result can be ignored. i The total time for the task is:
[0103]
[0104] Revenue Model
[0105] If the total time to complete the task is less than its maximum tolerable delay, the ESP will receive a unit reward Φ; otherwise, the ESP will not receive any reward. i The rewards obtained are:
[0106]
[0107] At the same time, considering the differences in priorities of different tasks, the reward for completing high-priority tasks will be relatively higher. Therefore, at time slot t, the total reward obtained by ESP for completing tasks is:
[0108]
[0109] In addition, ESP provides computing offloading services at a certain cost, which is related to the number of resources actually used by the task. Therefore, in time slot t, the total cost of ESP's resource rental is:
[0110]
[0111] in, and Represent the unit rental price of bandwidth and computing resources respectively. Therefore, in time slot t, the revenue obtained by ESP can be expressed as R t -C t .
[0112] Based on the above model, the present invention considers dividing the slice resources at the beginning of each slice window, and executing the corresponding cooperative offloading strategy and communication and computing resource allocation decision in each time slot to maximize the cumulative total benefit of ESP. Therefore, the optimization problem can be formally defined as:
[0113]
[0114] Among them, the constraints C1 and C2 represent L j The communication and computing resources rented by the servers in the slice window w do not exceed L j The total resources owned; Constraints C3 and C4 respectively indicate that the communication and computing resources allocated to users in each time slot do not exceed the total resources rented in the slice window. In the original optimization problem, there are decision variables belonging to different time scales, so it is difficult to solve them uniformly to obtain the optimal solution. In order to better solve the problem, the present invention decomposes it into two sub-problems on the corresponding time slots: slice partitioning and collaborative unloading:
[0115] P1: Determine the appropriate slice resource allocation strategy to maximize the long-term benefits of ESP. This sub-problem is formally defined as:
[0116]
[0117] The present invention solves P1 by deciding resource partitioning in each slice window, so as to maximize the long-term benefits of ESP. Considering that it is difficult for ESP to accurately obtain the actual resource requirements of users in the real edge environment, the present invention analyzes the historical user request traffic and then converts it into ESP resource requirements.
[0118] P2: Perform cooperative offloading and resource allocation to maximize the short-term benefits of ESP. This sub-problem is formally defined as:
[0119]
[0120] The goal of P2 is to transfer different types of tasks to appropriate servers through cooperative offloading in each time slot t, and then maximize the benefits of ESP in the current time slot through appropriate resource allocation. Since the tasks and the number of requests to be processed in each time slot are different, ESP needs to adjust and optimize cooperative offloading and resource allocation decisions to meet the resource requirements of user tasks.
[0121] SPCO framework proposed in this invention
[0122] In view of the above system model and problem definition, this paper proposes a novel SAGIN-oriented Slice Partitioning and Collaborative Offloading (SPCO) method to solve optimization problems P1 and P2, which effectively integrates the Transformer-based slice resource partitioning method and the improved DRL-based collaborative offloading and resource allocation method. First, Transformer is used to predict user traffic in future time slots, thereby guiding ESP to divide slice resources; then, by solving the problems of overestimation of Q value and high variance, the optimization of collaborative offloading and resource allocation is achieved to maximize the benefits of ESP.
[0123] Transformer-based slice resource division
[0124] In SAGIN, user traffic usually changes dynamically over time. If resource allocation is performed only under fixed slice division, slice resources will not be able to adapt well to traffic fluctuations. Based on the user's historical traffic data, the resource requirements of future time slots are predicted to reasonably utilize slice resources to process the user's computing offloading requests. Therefore, the present invention will perform reasonable slice resource division by predicting future resource requirements to solve the optimization problem P1. However, the actual resource requirements of user tasks cannot be accurately measured. To address this problem, the present invention uses Transformer to effectively perceive the changing trend of future user traffic, and combines the historical usage of resources to predict the demand for communication and computing resources in future time slots, and then divides slice resources. The key steps of the Transformer-based slice resource division method proposed in the present invention are shown in Algorithm 1.
[0125]
[0126]
[0127] First, the input of the encoder and decoder is constructed through the user's historical request traffic (line 1), where X his Indicates historical traffic, X cur represents the traffic sequence of the current slice window, X 0 Represents the time sampling of the traffic sequence to be predicted. The classic self-attention mechanism needs to calculate the attention weights of all historical time slots, which will lead to high computational complexity. In order to alleviate this problem, the present invention adopts sparse probabilistic self-attention when constructing the encoder and uses self-attention distillation between layers to reduce the computational overhead (line 2). Specifically, the feature extraction process from the jth layer to the j+1th layer is expressed as:
[0128]
[0129]
[0130] in,[·] attention represents sparse self-attention, d is , Conv1d represents one-dimensional convolution, ELU is the activation function, and MaxPool is the maximum pooling.
[0131] Next, the output of the encoder is fed into the decoder (line 3), which consists of a layer of multi-head sparse probabilistic self-attention mechanism and a layer of multi-head attention mechanism. Subsequently, the output of the decoder is fed into the MLP, and after inference, the predicted sequence of future traffic can be obtained (line 4). Next, the maximum value of the time slot predicted traffic is taken as the predicted traffic in the next slice window, and it is classified according to the task type and access platform (line 5). The communication resource demand of each platform is the product of the unit communication resource demand of all types of tasks within its coverage area and the corresponding type of request traffic, that is, where b CMI and b CPI Respectively represent the average communication bandwidth required for CMI and CPI tasks, and They represent the predicted CMI and CPI task traffic respectively. Since the computing loads of different platforms are different, it is necessary to configure computing resources in a targeted manner. Specifically, the computing resources required for the CMI task within the BS coverage area are The CPI tasks in other regions are where f CMI and f CPI They represent the average computing resource requirements for CMI and CPI tasks respectively. Satellites and drones will forward CPI tasks within their coverage to BS for execution, so the computing resource requirements of BS are (Row 6). UAVs and satellites only need to process CMI tasks within their coverage area, so their computing resource requirements are and (Lines 7-8).
[0132] Collaborative offloading and resource allocation based on improved DRL
[0133] After completing the slice resource division within the slice window, the unloading process between different slice windows is relatively independent. Therefore, the long-term optimization problem of the maximum ESP cumulative benefit in P2 can be transformed into a short-term optimization problem within a single slice window, and the problem can be constructed as a Markov decision process. Specifically, the considered SAGIN is regarded as an environment, and ESP is abstracted as a DRL agent, which makes decisions on collaborative unloading and resource allocation by interacting with the environment, and then updates the strategy through the reward signal fed back by the environment. Figure 3 The collaborative offloading and resource allocation method based on improved DRL proposed in the present invention is shown. Accordingly, the state space, action space and reward function are defined as follows.
[0134] (1) State space: In time slot t, by sensing the available communication and computing resources and task attributes, the agent can capture the task requirements and the corresponding resources. Therefore, the system state in time slot t can be expressed as:
[0135]
[0136] in,
[0137] (2) Action space: The action space contains the actions of allocating communication and computing resources to users. Therefore, the action in time slot t can be expressed as:
[0138] a t = {b t ,f t} (15)
[0139] in,
[0140] (3) Reward function: Problem P2 aims to maximize the short-term benefit of ESP in each time slot. Therefore, the reward function is defined as the ESP benefit in time slot t, which can be expressed as:
[0141] r t =R t -C t (16)
[0142]
[0143] The key steps of the collaborative unloading and resource allocation method based on improved DRL proposed in the present invention are shown in Algorithm 2. First, initialize the main network Q1, Q2 and μ and the target network Q'1, Q'2 and μ' (lines 1-2). Classical DRL adopts the strategy of maximizing the Q value, which will lead to the problem of non-uniform overestimation of the Q value. In order to solve this problem, this method introduces two Q value networks with the same structure. When calculating the target value, the minimum value is used to estimate the value of the next state-action pair. Next, initialize the experience replay cache RB, the number of slice windows W, and the number of time slots T in each slice window (line 3). In order to reduce the correlation between experience data and improve the overall training efficiency, the experience replay mechanism is introduced. At the beginning of the training cycle, the system environment is initialized and its initial state is obtained (line 5). At the beginning of each slice window, call Algorithm 1 to predict the user traffic in the future time slot and perform slice resource division (line 7). In time slot t, select an action based on the policy network and exploration noise (line 9). Next, after executing the action, the immediate reward and next state of the environment feedback are obtained (line 10). t ,a t ,r t ,s t+1) is stored in the experience replay buffer RB (line 11). Then, K samples are randomly taken from RB to update the network parameters (line 12). Based on the target policy smoothing regularization, the target action is obtained using the target policy network (Line 13), the process can be expressed as:
[0144]
[0145] The noise follows a normal distribution and the sampled noise is clipped to reduce the variance of the target estimate. Next, the target Q value y target Based on the minimum value of the two Q-value networks, the update (line 14) can be expressed as:
[0146]
[0147] Finally, the gradient descent algorithm is used to minimize the error between the evaluation value and the target value, and then the Q-value network parameters are updated (line 15). In addition, the delayed update method is used to improve the training stability of the agent. When the Q-value network is updated a certain number of times, the policy network parameters and target network parameters are updated using the gradient ascent and soft update methods respectively (lines 16-18).
[0148] Performance Evaluation
[0149] Based on a workstation equipped with an 8-core CPU @ 3.2GHz and 32GB RAM, this embodiment constructs the SAGIN environment and implements the proposed SPCO framework using PyTorch. The cellular network traffic dataset of Milan, Italy is used to simulate the user traffic change trend. The dataset records the changes in the communication traffic of users in the city within 2 months with a sampling frequency of 10 minutes, covering three real network traffic data: calls, text messages, and the Internet. In the experiment, the Internet communication traffic sampled in the dataset is regarded as the number of user requests in each time slot, the call and text message services in the dataset are regarded as CMI-type tasks, and the Internet services are regarded as CPI-type tasks. One training round contains 24 slice windows, and each slice window contains 6 scheduling time slots. The constructed SAGIN environment contains 1 satellite, 1 drone, and 1 BS. The main parameter settings are shown in Table 1, where the parameters of the satellite, drone, and BS are expressed in triple form, and the parameters of the CMI and CPI tasks are expressed in binary form.
[0150] In order to evaluate the superiority of the SPCO framework proposed in this invention, this example comprehensively compares it with the following benchmark methods:
[0151] (1) TD3_RA: The Twin Delayed Deep Deterministic policy gradient algorithm (TD3) is used for resource allocation, but adaptive slicing is not considered.
[0152] (2) DDPG_RA: The Deep Deterministic Policy Gradient (DDPG) algorithm is used for resource allocation, but adaptive slicing is not considered.
[0153] (3) HPFS: Tasks are executed according to the High Priority First Service (HPFS) strategy, and communication and computing resources are allocated according to the average demand of the tasks.
[0154] (4) FCFS: Tasks are executed according to the First Come First Service (FCFS) policy. Communication and computing resources are allocated in the same way as HPFS.
[0155] Table 1 Main parameter settings
[0156]
[0157] First, the user traffic prediction performance of this method is evaluated in multiple slice windows. In the experiment, the user traffic of the next time slot is predicted based on the user traffic of the first 6 slice windows, and the actual value and predicted value of user traffic on different servers are compared. Figure 4 As shown in Figure 1, user request traffic shows a trend of periodic changes. This method can effectively capture this trend and accurately predict future user traffic. Next, the user traffic prediction results will be used as the basis for slice resource division. This method will further combine task attributes and environmental conditions to perform reasonable resource allocation to improve ESP benefits.
[0158] Then, the convergence of different methods is analyzed. Figure 5As shown in the figure, the FCFS and HPFS methods are limited by their single, agent-free decision framework, which causes their reward curves to remain unchanged during training. Compared with other DRL-based methods, the performance of the FCFS and HPFS methods is poor. This is because the strategies adopted by the FCFS and HPFS methods when allocating resources to tasks are too rigid, lacking consideration of the system dynamics and task personalization requirements, resulting in the resource requirements of many tasks being unmet, making it impossible for ESP to obtain the expected benefits of completing these tasks. The HPFS method preferentially allocates resources to tasks with high priority, so it can obtain higher benefits. The SPCO and TD3_RA methods both show better performance than the DDPG_RA method in terms of training stability and convergence rate. This is because the SPCO and TD3_RA methods use dual-strategy networks to estimate Q values, which effectively alleviates the problem of overestimation of Q values, thereby improving the stability and performance of the algorithm. At the same time, the SPCO and TD3_RA methods use delayed learning to update the policy network and introduce target policy smoothing, which avoids frequent updates of the network and effectively alleviates the high variance problem in the DDPG_RA method, so that the model training can converge more stably. Compared with the TD3_RA method, the SPCO proposed in the present invention has achieved further improvement in performance. This is because compared with the TD3_RA method that uses static slice resource division, the proposed SPCO can accurately predict the user request traffic in the future time slot and then dynamically divide the slice resources, so as to better meet the resource requests of user services.
[0159] Then, the benefits of ESP and the cost of resource rental under different methods are compared. Figure 6 As shown in the figure, by classifying the task types, the proposed SPCO can better identify the communication and computing resource requirements of tasks in SAGIN. Compared with other methods, SPCO can rent communication and computing resources more reasonably while improving the revenue of ESP. This is because the strategy of SPCO to adjust slice resources according to the predicted user traffic can effectively adapt to the resource requirements of user tasks and better support edge nodes to provide services to users. The results show that the proposed slice partitioning strategy based on traffic prediction can utilize resources more reasonably and efficiently to improve the revenue of ESP.
[0160] Then, the task completion status of different methods under different maximum tolerable delays was compared. Figure 7As shown in the figure, as the maximum tolerable delay increases, the task failure rate decreases. This is because ESP has more time to process tasks, which reduces the number of tasks that fail due to exceeding the maximum tolerable delay. Compared with the DRL-based method, the FCFS and HPFS methods show higher task failure rates under different maximum tolerable delays. This is because the FCFS and HPFS methods only allocate resources based on the average resource requirements of the tasks, resulting in unreasonable resource allocation, so some tasks cannot obtain sufficient resources. Compared with other methods, the proposed SPCO can complete more tasks under different maximum tolerable delays and achieve the lowest task failure rate. This is because the dual Q value network architecture and delayed update mechanism in SPCO improve the performance of decision-making. Therefore, SPCO can make more reasonable resource allocation decisions in different scenarios to complete more tasks.
[0161] Finally, the slice resource utilization of different methods under different user traffic multipliers is compared. Figure 8 As shown in the figure, when the user traffic is 0.5 times the initial setting, the resource utilization rate is significantly reduced. This is because as the number of user requests decreases, the proportion of resources allocated to the task also decreases, resulting in many resources being idle. As the user traffic increases to 1.5 times the initial setting, the resource utilization rate increases slightly. This is because the increase in user requests causes more resources to be allocated and used, and the idle resources decrease accordingly, thus showing a higher resource utilization rate. Compared with other methods, the proposed SPCO can achieve higher resource utilization, which also shows that adjusting the slice division strategy based on traffic prediction helps to make more reasonable and efficient use of limited slice resources.
[0162] The present invention proposes a novel collaborative unloading method for slicing partitioning of an integrated air-space-ground network. First, a traffic prediction method is designed to perceive the changing trend of user traffic through a probabilistic sparse self-attention mechanism, and a network slicing partitioning method is developed based on this. Then, an improved DRL collaborative unloading and resource allocation method is designed. By addressing problems such as difficulty in convergence caused by overestimation of Q values and high variance, the reasonable allocation of communication and computing resources and efficient collaborative unloading of heterogeneous platforms are achieved. The effectiveness of this method has been verified through the above experiments based on real traffic data sets. Compared with other methods, this method can effectively increase the benefits of ESP and show higher performance in terms of task completion rate and resource utilization.
[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0167] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. A slicing division and collaborative offloading method for air-ground integrated network, It is characterized in that The following steps are involved: (1) Collect historical user traffic of ESP, predict future user traffic, and calculate the resource requirements required by ESP. ESP performs network slicing at the beginning of each slicing window based on the resource requirements. (2) The infrastructure provider responds to the ESP's resource allocation request and allocates network slices for it; (3) The user accesses and uploads the computation offloading request to ESP through SAGIN; (4) ESP assigns the appropriate edge server to the user based on the user's access location and request type, and allocates communication and computing resources based on task attributes and user priority; (5) During the collaborative offloading process, the actions taken, the rewards obtained, and the new states entered are recorded in each time slot, and based on the above information, the system continuously optimizes its own performance.
2. The slicing division and collaborative offloading method for an air-ground integrated network according to claim 1, It is characterized in that The SAGIN is composed of a ground base station BS, a drone and a satellite. Users access the computing offloading service provided by ESP through SAGIN. The entire area covered by SAGIN is divided into three independent areas: the area covered by the BS, the area covered by the drone and the area not covered by the BS and the drone. The BS and the drone provide services to users in different areas respectively, while the satellite can provide services to users in the area not covered by the BS and the drone. The satellite communicates directly with the BS and the drone, while the BS and the drone communicate indirectly through the satellite forwarding. All users are denoted as a set U = {u 1 ,u 2 ,...,u N }, the servers equipped by satellite, BS and UAV are defined as L sat , L uav and L bs , all servers are recorded as set L = L sat ∪L uav ∪L bs ={L 1 ,L 2 ,…,L M }; Considering the differences in offloading requests in different regions, server L will be j The number of offload requests received within the coverage area is denoted as N j Therefore, the total number of offload requests that ESP needs to process is equal to the sum of the number of requests in each area, that is, The slicing model of SAGIN is: The physical resources of satellites, BSs, and drones are arranged into resource-customized network slices; for server L j , whose bandwidth and computing resources are recorded as and The system time slot is divided into W slice windows, each slice window w contains a number of time slots t; the bandwidth and computing resources allocated to ESP in the slice window w are respectively denoted as and At the beginning of each slice window, ESP analyzes the historical requests received by each area to predict future demand and then divides the slice resources appropriately; The communication model of the SAGIN is: When user u i When initiating a computing offloading request to ESP, the process mainly includes four stages, namely, request access, platform collaboration, task execution and result return; the task attribute of ui is recorded as a five-tuple {a i ,c i ,o i ,ρ i ,l i }, where each element represents the amount of data, required computing power, type, priority and u of the task. i The server to which you connect; the type of task, based on the amount of task data and the computing power required i Divided into communication-intensive CMI and computation-intensive CPI; the priority of the task ρ i Indicates the level of reward that can be obtained by completing the task; When ESP processes a computation offload request, it needs to consider u i The area where you are located; if u i If the BS or UAV is within its coverage, it will be given priority to access the computing offloading service through the BS or UAV; otherwise, u i Computing offload services can only be accessed via satellite; ESP is allocated to u i The bandwidth is denoted as b i , then u i The task access L j The uplink transmission rate is: Among them, p i is u i Upload power, g 0 With σ 2 u i With L j The channel gain and Gaussian white noise power between; will u i Task request access L j The time required is: When the task is uploaded to L j After that, ESP will i The access location and task type are assigned to the corresponding server L j' ; when u i When computing offloading services are accessed through BS, both CMI and CPI tasks are executed on BS; i When computing offloading services are accessed through drones or satellites, CMI tasks are directly executed on the drones or satellites, while CPI tasks are forwarded to the BS with more abundant computing resources for execution; accordingly, u i The task data from L j Forward to L j' The required transfer time is: Among them, R u2s represents the data transmission rate between the UAV and the satellite, R s2b Indicates the data transmission rate between BS and satellite; The calculation model of SAGIN is: When the user task is offloaded to the corresponding server, ESP allocates computing resources to it; user u i The CPU power allocated to the task is recorded as f i , then the time required to execute the task is: After the task is completed, the user will receive feedback on the results; Complete i The total time for the task is: T i total =T i up +T i tran +T i exec (5) The revenue model of SAGIN is: If the total time to complete the task is less than its maximum tolerable delay, the ESP will receive a unit reward Φ; otherwise, the ESP will not receive any reward; therefore, at time t, the ESP will receive a unit reward Φ from u i The rewards obtained are: At the same time, considering the differences in priorities of different tasks, the reward for completing high-priority tasks will be relatively higher; therefore, at time slot t, the total reward obtained by ESP for completing tasks is: In addition, ESP provides computing offloading services at a certain cost, which is related to the number of resources actually used by the task; therefore, in time slot t, the total resource rental cost of ESP is: in, and denote the unit rental price of bandwidth and computing resources respectively; therefore, at time slot t, the revenue obtained by ESP is expressed as R t -C t .
3. The slicing division and collaborative offloading method for an air-ground integrated network according to claim 2, It is characterized in that Based on the slicing model, communication model, computing model and revenue model of SAGIN, it is considered to divide the slice resources at the beginning of each slice window, and execute the corresponding cooperative offloading strategy and communication and computing resource allocation decision in each time slot to maximize the cumulative total revenue of ESP; therefore, the optimization problem is defined as: Among them, the constraints C1 and C2 represent L j The communication and computing resources rented by the servers in the slice window w do not exceed L j The total resources owned; Constraints C3 and C4 respectively indicate that the communication and computing resources allocated to users in each time slot do not exceed the total resources rented in the slice window; The optimization problem is decomposed into two sub-problems on the corresponding time slots: slice partitioning and collaborative offloading: P1: Determine the appropriate slice resource allocation strategy to maximize the long-term benefits of ESP. This sub-problem is defined as: By deciding on resource allocation in each slice window to solve P1, the long-term benefits of ESP are maximized. Considering that in the real edge environment, it is difficult for ESP to accurately obtain the actual resource requirements of users, historical user request traffic is analyzed and converted into ESP resource requirements. P2: Perform cooperative offloading and resource allocation to maximize the short-term benefits of ESP. This sub-problem is defined as: The goal of P2 is to transfer different types of tasks to appropriate servers through cooperative offloading in each time slot t, and then maximize the benefits of ESP in the current time slot through appropriate resource allocation; since the tasks and the number of requests to be processed in each time slot are different, ESP needs to adjust and optimize cooperative offloading and resource allocation decisions to meet the resource requirements of user tasks; The slice partitioning and collaborative offloading method combines the Transformer-based slice resource partitioning method and the improved DRL-based collaborative offloading and resource allocation method. First, the Transformer is used to predict the user traffic in the future time slot, and then guide the ESP to partition the slice resources. Then, by solving the Q value overestimation and high variance problems, the collaborative offloading and resource allocation are optimized to maximize the benefits of the ESP.
4. The slicing division and collaborative offloading method for an air-ground integrated network according to claim 3, It is characterized in that The implementation method of the Transformer-based slice resource partitioning method is: First, the inputs of the encoder and decoder are constructed through the user's historical request traffic. Sparse probabilistic self-attention is used when constructing the encoder, and self-attention distillation is used between layers to reduce computational overhead. The feature extraction process from the jth layer to the j+1th layer is expressed as: in,[·] attention represents sparse self-attention, and d is Dimension, Conv1d represents one-dimensional convolution, ELU is the activation function, and MaxPool is the maximum pooling; Next, the output of the encoder is sent to the decoder, which consists of a layer of multi-head sparse probabilistic self-attention mechanism and a layer of multi-head attention mechanism; then, the output of the decoder is sent to the MLP, and the predicted sequence of future traffic is obtained through reasoning; then, the maximum value of the time slot predicted traffic is taken as the predicted traffic in the next slice window, and it is classified according to the task type and access platform; the communication resource demand of each platform is the product of the unit communication resource demand of all types of tasks within its coverage range and the corresponding type of request traffic, that is, where b CMI and b CPI Respectively represent the average communication bandwidth required for CMI and CPI tasks, and They represent the predicted CMI and CPI task traffic respectively; the computing resources required for the CMI task within the BS coverage area are The CPI tasks in other regions are where f CMI and f CPI They represent the average computing resource requirements for CMI and CPI tasks respectively; satellites and UAVs forward CPI tasks within their coverage to BS for execution, so the computing resource requirements of BS are UAVs and satellites only need to process CMI tasks within their coverage area, so their computing resource requirements are and 5. The slicing division and collaborative offloading method for an air-ground integrated network according to claim 3, It is characterized in that After completing the division of slice resources within the slice window, the unloading process between different slice windows is relatively independent; Therefore, the long-term optimization problem of the maximum ESP cumulative benefit in P2 is transformed into a short-term optimization problem within a single slice window, and the problem is constructed as a Markov decision process; SAGIN is regarded as the environment, and ESP is abstracted as a DRL agent, which makes decisions on collaborative unloading and resource allocation by interacting with the environment, and then updates the strategy through the reward signal fed back by the environment; the state space, action space and reward function are defined as follows; (1) State space: In time slot t, by sensing the available communication and computing resources and task attribute information, the agent captures the task requirements and the corresponding resources required; therefore, the system state in time slot t is expressed as: in, (2) Action space: The action space contains the actions of allocating communication and computing resources to users; therefore, the action at time slot t is expressed as: a t ={b t ,f t } (15) in, (3) Reward function: Problem P2 aims to maximize the short-term benefit of ESP in each time slot; therefore, the reward function is defined as the benefit of ESP at time slot t, which is expressed as: r t =R t -C t (16) The implementation method of the collaborative offloading and resource allocation method based on the improved DRL is: First, initialize the main network Q 1 , Q 2 and μ and the target network Q' 1 , Q' 2 and μ'; introduce two Q-value networks with the same structure, and use the minimum value to estimate the value of the next state-action pair when calculating the target value; then, initialize the experience replay buffer RB, the number of slice windows W, and the number of time slots T in each slice window; in order to reduce the correlation between experience data and improve the overall training efficiency, introduce the experience replay mechanism; at the beginning of the training cycle, initialize the system environment and obtain its initial state; at the beginning of each slice window, predict the user traffic in the future time slot and perform slice resource partitioning through the Transformer-based slice resource partitioning method; at time slot t, select an action based on the policy network and exploration noise; then, after executing the action and obtaining the immediate reward and next state of the environment feedback; then, the training samples (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer RB; then, K samples are randomly taken from RB to update the network parameters; based on the target strategy smoothing regularization, the target action is obtained using the target strategy network The process is expressed as: The noise follows a normal distribution and the sampled noise is clipped to reduce the variance of the target estimate. Then, the target Q value y target Based on the minimum value of the two Q-value networks, the update process is expressed as: Finally, the gradient descent algorithm is used to minimize the error between the evaluation value and the target value, and then the Q-value network parameters are updated; when the Q-value network is updated a certain number of times, the strategy network parameters and target network parameters are updated using gradient ascent and soft update respectively.
Citation Information
Patent Citations
Deep learning-based intelligent identification method for color steel tile building along railway
CN114239755A
Computing unloading method for 5G network slice in MEC environment
CN117202264A
MEC-based air-space-ground integrated network task segmentation and resource allocation method
CN118075772A
Computing resource configuration methods and apparatuses
US20240054020A1
Cited By
Wireless communication system of digital ward
CN119922616A
Edge computing network-oriented energy efficiency service deployment and delivery method and system
CN120751406A
Real-time deviation rectifying method, device, equipment, medium and product for end slope mining cave track
CN121596754A
