Comprehensive optimization method and system for base station dormancy control and calculation task distribution based on deep reinforcement learning

Abstract

The invention discloses a comprehensive optimization scheme and system for base station dormancy control and calculation task distribution based on deep reinforcement learning, and belongs to the field of mobile communication, and the scheme comprises the steps: constructing a multi-user edge calculation system composed of a cloud server and a plurality of edge nodes, and determining the initial dormancy state of a base station; then, according to the dormant state of the base station, the system determines the incidence relation between the user and the base station, reasonably divides calculation tasks according to equipment and calculation resource conditions, and determines the task distribution proportion of each calculation node; and after task allocation is completed, the system calculates average time delay of the users and total energy consumption of the base station, calculates a cost function by using the results and generates a reward value which is used for updating parameters of the deep reinforcement learning model so as to adjust the dormant state of the base station. The process is continuously adjusted through multi-round iterative optimization, and finally a comprehensive optimization result close to global optimum is achieved. According to the method, the energy consumption of the base station is effectively reduced on the premise of not obviously influencing the time delay performance of the user.

Description

Technical Field

[0001] The present invention belongs to the technical field related to mobile communications, and more specifically, to a comprehensive optimization method and system for base station sleep control and computing task distribution based on deep reinforcement learning. Background Art

[0002] In recent years, with the vigorous development of mobile communication networks, it is becoming a trend to use mobile edge computing (MEC) to support highly complex and delay-sensitive applications such as autonomous driving and virtual reality. Through MEC servers deployed at the edge of the network (such as base stations), mobile users can offload their computing tasks to nearby MEC servers for fast processing. At the same time, in order to support these data-intensive and delay-sensitive applications, it has become a major development trend of mobile communications to shorten the distance between base stations and users by densely deploying a large number of base stations, thereby improving users' performance in terms of transmission rate, delay and reliability. In this context, implementing a reasonable base station sleep strategy is conducive to reducing base station energy consumption and reducing operators' operating costs. However, base station sleep affects initial access, transmission and computing delays. Therefore, users need to adjust their computing task distribution strategy according to the base station sleep situation to minimize delays.

[0003] Existing research on base station sleep in edge computing systems is mainly aimed at low-frequency mobile communication networks, and does not consider the characteristics of high-frequency mobile communication networks. In low-frequency bands, the channel state changes relatively slowly, so traditional base station sleep control methods usually make decisions based on historical channel states. However, in high-frequency band networks, the channel state fluctuates greatly, resulting in significant changes in transmission rate and delay, especially when the user side faces multiple delays (such as initial access delay, data transmission delay, computing delay, etc.), how to optimize these delays is still an urgent problem to be solved. Therefore, in mobile edge computing (MEC) systems, the comprehensive optimization of base station sleep control and computing task distribution has both important theoretical research significance and broad application prospects. Summary of the invention

[0004] In response to the above defects or improvement needs of the prior art, the present invention provides a comprehensive optimization method and system for base station sleep control and computing task distribution based on deep reinforcement learning, thereby solving the technical problem of joint optimization of base station sleep and computing task distribution in mobile edge computing scenarios.

[0005] To achieve the above object, according to one aspect of the present invention, the present invention provides a comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning, including:

[0006] S1. Build a multi-user edge computing system consisting of a cloud server and multiple edge nodes, where the edge node (EN) consists of an edge server and a base station, and determine the initial sleep state of the base station.

[0007] S2. According to the sleep state of the current base station, the system first determines the association between the user and the base station, and then reasonably divides the computing tasks according to the resource conditions of the user device, edge computing server and cloud computing server, and determines the proportion of subtasks assigned to each computing node.

[0008] S3. After the task allocation is completed, the system calculates the average delay of the user and the total energy consumption of the base station, uses the user delay and the total energy consumption of the base station to calculate the cost function, and generates the corresponding reward value. The system updates the parameters of the deep reinforcement learning model according to the reward value and further adjusts the dormant state of the base station.

[0009] S4. The system repeats the process of task allocation and base station sleep state adjustment, and gradually optimizes through multiple rounds of iterations until the solution converges, and finally obtains a comprehensive optimization result close to the global optimal.

[0010] Preferably, the association relationship between the user and the base station includes:

[0011] The relationship between UE and EN is determined by In order to ensure that each UE is associated with at most one EN, x k,j Need to satisfy constraints

[0012] Preferably, the task division includes:

[0013] For UEs associated with EN j, the tasks generated by them can be assigned to EN j, cloud servers, or processed by the UE itself. Each task is divided into several subtasks, each corresponding to a different data size. The proportion of task allocation is α k , β k,j and γ k,j In which, α k represents the proportion of tasks handled by UE k itself, β k,j represents the proportion of tasks handled by EN j, γ k,j represents the proportion of tasks handled by the cloud server. Assume x k,j =1, the division of tasks needs to satisfy the constraints: ①β k,j ≤x k,j ,γ k,j ≤x k,j , ② When x k,j =0, β k,j = 0 and γ k,j=0, indicating that the task is completely handled by the user terminal.

[0014] Preferably, modeling of user mobility includes:

[0015] To capture the mobility of UE, it is assumed that its position may remain unchanged or move randomly in each time slot. Therefore, the system needs to re-optimize the user association (x k,j ) and task allocation (α k ,β k,j ,γ k,j ). Specifically, the optimization problem of task allocation and user association needs to be solved based on the location information of the UE in the current time slot. When the location of the UE changes, the system will re-optimize based on the updated location in the next time slot.

[0016] Preferably, the total computing power required to complete the task includes:

[0017] The input data size of the task (such as the file size of a video clip) is s k , in bits, the total number of CPU cycles required to complete the task generated by each UE k is η k .

[0018] Preferably, the user calculates the delay including:

[0019] The time required to execute the assigned task locally (in seconds) is Among them, α k η k Indicates the number of CPU cycles required to complete the assigned task, f k (U) represents the computing capability of UE k.

[0020] Assume that each EN is capable of executing subtasks received from multiple UEs simultaneously. For fairness, the computing power of the EN is equally distributed to all UEs associated with the EN in each time slot. Once the input data of the subtasks is uploaded to the EN, the EN will immediately execute these subtasks. In this way, each EN will not incur waiting time.

[0021] The time required to execute the subtask of UE k on EN j is given by the subtask ratio β k,j , input data size s k And the computational complexity z k Decision, that is in represents the total number of associated users of EN j; is the computing power of each EN j, which is evenly distributed among all associated users. Therefore, the computing power allocated to UE k is

[0022] The cloud server provides fixed computing power for each UE k If there is a ratio of γ k,j The subtask is offloaded to EN j, which then forwards it to the cloud server for execution (x k,j =1), then the execution time of the cloud server is

[0023] Preferably, the user transmission delay includes:

[0024] Each edge node (EN) can measure the uplink signal to interference and noise ratio (SINR) of the user equipment (UE) associated with it, denoted as SINR k,j The specific process is that at the beginning of each time slot, the UE sends a pilot or beacon signal to the EN; the EN uses the received signal to estimate the channel state information (CSI) and calculate the corresponding SINR value.

[0025] Assume W is the access channel bandwidth of each EN, then the uplink data rate when UE k is associated with ENj is k∈K,j∈J, where is the load of EN j, i.e. the number of UEs associated with it. UE k is only k,j ≥ SINR th (preset threshold) can be associated with EN j. The optional EN set of UE k is π k ={j|SINR k,j ≥ SINR th}, like Then x k,j = 0. In addition, the number of UEs that each EN j can serve has an upper limit S j , usually determined by the number of physical channels of the EN.

[0026] For the subtask of UE k, if the allocation ratio is β k,j and γ k,j , then the time it takes to transmit to EN is k∈K,j∈J.

[0027] The backhaul link between EN j and the cloud server is a wired connection with a bandwidth of M j , and the backhaul bandwidth is evenly distributed to all UEs associated with EN j. Therefore, the total time required to upload the subtask of UE k to the cloud server through EN j is k∈K,j∈J, where is the return link transmission time.

[0028] It is assumed that the amount of output data from task execution is small, so the delay in returning task results from the cloud server or EN is negligible.

[0029] Preferably, the total user delay includes:

[0030] Consider a case of a divisible computing task, where the task can be divided into multiple independent subtasks and distributed to local devices, edge nodes (ENs), and cloud servers for parallel processing. For example, in a video object recognition task, the video is segmented into several segments and processed locally, on ENs, and on cloud servers.

[0031] The task is completed on the local device in

[0032] The completion time of the task after it is transmitted from UE k to EN j is in is the transmission time for uploading to EN, EN execution time.

[0033] The task is uploaded to EN by UE k, and then forwarded to the cloud server by EN. The total time to return the result after completion is in Including the transmission time of upload and return links, is the computing time of the cloud server.

[0034] Since the subtasks are independent and can be executed in parallel, the time it takes to complete the entire task depends on the last completed part in the local device, EN, and cloud server. k,j =1 means UE k selects EN j as the associated node, then the total task completion delay T k is the maximum of the following three:

[0035] Preferably, the base station energy consumption includes:

[0036] The total power consumption of the base station P j It includes two parts: static power consumption and dynamic power consumption, namely Indicates the static power consumption when the base station is turned on, which is related to the fixed configuration of the base station's hardware equipment, cooling system, etc. This part of the power consumption has nothing to do with the working state of the base station and is the basic energy consumption of the base station operation. Represents dynamic power consumption, which is determined by the number of user equipment (UE) served by the base station and the power consumption of the active radio frequency chain (RF chain). Indicates the power consumption of a single RF chain. It represents the number of UEs served by base station j, which is equal to the number of active RF chains of the base station.

[0037] Preferably, the system cost function includes:

[0038] The system cost function is the weighted sum of the total user delay and the total energy consumption of the base station accumulated at multiple times, specifically: Among them, ω represents the weight, y j represents the sleep state variable of base station j (0 means sleep, 1 means not sleep), and there is a constraint x i,j ≤y j ,k∈K,j∈J, that is, the UE can only associate with the base station j that is not in the sleep state. In the base station sleep strategy based on deep reinforcement learning, the agent's reward replaces the negative value of the valuation function.

[0039] Optionally, the minimization of the average user delay accumulated at multiple times (equivalent to minimizing the total user delay) and the total energy consumption of the base station, the optimization objectives and constraints are specifically as follows:

[0040]

[0041] According to another aspect of the present invention, a comprehensive optimization system for base station sleep control and computing task distribution based on deep reinforcement learning is provided, comprising:

[0042] The network controller is used to collect various status information of other devices in the network, such as user devices, edge nodes, and cloud servers, and solve the optimal strategy for base station sleep and computing task distribution based on this information and preset optimization goals, and issue the strategy to guide network devices to adjust their behavior in response to new instructions. In the sub-problem of base station sleep strategy based on deep reinforcement learning, the network controller is used as an agent to solve the sleep strategy. In the sub-problem of computing task allocation, the network controller is also a solving unit for user association and computing task distribution.

[0043] User devices are used to generate computing tasks and communicate with edge nodes or cloud servers.

[0044] The edge node, which is integrated by the edge server and the base station, is used to receive task requests from user devices and perform task calculations or forward them to the cloud server.

[0045] Cloud servers are used to receive tasks forwarded by edge nodes and complete highly complex computing tasks.

[0046] According to another aspect of the present invention, a computer readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0047] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0048] The comprehensive optimization method of base station sleep control and computing task distribution based on deep reinforcement learning proposed in the present invention decomposes the joint optimization problem into two sub-problems of base station sleep and computing tasks based on deep reinforcement learning, and realizes the search for the optimal solution through continuous iteration of the two until convergence. The system first constructs a multi-user edge computing system consisting of a cloud server and multiple edge nodes, the edge nodes include edge servers and base stations, and determines the initial sleep state of the base station. Then, according to the sleep state of the base station, the system determines the association between the user and the base station, and reasonably divides the computing tasks according to the equipment and computing resources, and determines the task allocation ratio of each computing node. After the task allocation is completed, the system calculates the average delay of the user and the total energy consumption of the base station, and uses these results to calculate the cost function and generate a reward value, which is used to update the parameters of the deep reinforcement learning model, and then adjust the sleep state of the base station. This process is optimized through multiple rounds of iterations, and is continuously adjusted until the solution converges, and finally achieves a comprehensive optimization result close to the global optimum. Based on the above method, the present invention effectively reduces the energy consumption of the base station without significantly affecting the user's delay performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning in one embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of a comprehensive optimization system for base station sleep control and computing task distribution based on deep reinforcement learning in one embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of the timeline under the optimal subtask division scheme in one embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] Example 1

[0054] like Figure 1 As shown, a flowchart of the steps of comprehensive optimization of base station sleep control and computing task distribution based on deep reinforcement learning in one embodiment of the present invention is provided, and the steps are introduced in detail below.

[0055] S1, build a multi-user edge computing system consisting of a cloud server and multiple edge nodes, where the edge node (EN) consists of an edge server and a base station, and determine the initial sleep state of the base station.

[0056] Specifically, Figure 2 The figure shows a schematic diagram of a comprehensive optimization system for base station sleep control and computing task distribution based on deep reinforcement learning in an embodiment. Consider a multi-user mobile edge computing system, which consists of a cloud server and multiple mobile edge computing servers, which are placed next to or integrated in the base station of the wireless cellular network. The combination of the base station and the mobile edge server is regarded as an edge node (EN), and each EN communicates with the cloud server through a backhaul connection. The whole system includes J ENs, indexed as These ENs collectively serve K mobile user equipments (UEs), indexed as The base station is initially set to be fully non-dormant.

[0057] S2. According to the sleep state of the current base station, the system first determines the association between the user and the base station, and then reasonably divides the computing tasks according to the resource conditions of the user device, edge computing server and cloud computing server, and determines the proportion of subtasks assigned to each computing node.

[0058] Specifically, in order to solve the above optimization problem, the present invention adopts a hierarchical strategy to decompose the problem into two levels. The lower layer focuses on determining the optimal computing task division ratio based on a given user association scheme; the higher layer is committed to optimizing the user association scheme. In the lower-level problem, the goal of the present invention is to find a computing task division method that minimizes the total delay of the user. Since the last completed part of the user device, edge computing server and cloud server determines the overall delay, and the completion time of each part directly depends on the task ratio assigned to it (the sum of the three ratios is 1), the key to minimizing the total delay is to ensure that the three parts can complete their respective tasks as simultaneously as possible. Therefore, the present invention achieves the goal of minimizing user delay by finding an optimal task division ratio so that the subtasks of the three parts of local processing, edge computing and cloud computing are completed almost at the same time. Based on the optimal task division scheme obtained by the lower layer, the user association problem of the higher layer will be converted into a linear integer programming problem. At this level, the present invention uses a method based on dual decomposition to solve it. This method can not only ensure that each user is associated with the base station with the most favorable channel conditions, but also promote load balancing between different base stations, thereby improving the efficiency and performance of the entire system. Through such a hierarchical optimization strategy, the present invention can effectively solve the comprehensive optimization challenges of base station sleep control and computing task distribution, and achieve maximization of system energy efficiency and optimization of user experience.

[0059] Specifically, the present invention is Indicates the association between UE and EN. To ensure that each UE is associated with at most one EN, the constraint For UEs associated with EN j, the tasks generated by them can be assigned to EN j, cloud servers, or processed by the UE itself. Each task is divided into several subtasks, each corresponding to a different data size. The proportion of task allocation is α k , β k,j and γ k,j In which, α k represents the proportion of tasks handled by UE k itself, β k,j represents the proportion of tasks handled by EN j, γ k,j represents the proportion of tasks handled by the cloud server. Assume x k,j =1, the division of tasks needs to satisfy two constraints. Constraint 1 is β k,j ≤x k,j ,γ k,j ≤x k,j , It means that only users associated with the base station can offload tasks to edge nodes or cloud servers for computing; constraint 2 is It means that the sum of the proportions of the task being divided into three parts is 1. k,j =0, β k,j = 0 and γ k,j =0, indicating that the task is completely handled by the user terminal.

[0060] To further illustrate, in order to capture the mobility of the UE, it is assumed that its position may remain unchanged or move randomly in each time slot. Therefore, the system needs to re-optimize the user association (x k,j ) and task allocation (α k ,β k,j ,γ k,j ). Specifically, the optimization problem of task allocation and user association needs to be solved based on the location information of the UE in the current time slot. When the location of the UE changes, the system will re-optimize based on the updated location in the next time slot.

[0061] To further explain, the input data size of the task (such as the file size of the video clip) is s k , in bits, the total number of CPU cycles required to complete the task generated by each UE k is η k The time required to execute the assigned task locally (in seconds) is Among them, α k η k Indicates the number of CPU cycles required to complete the assigned task. represents the computational capacity of UE k. It is assumed that each EN is capable of executing subtasks received from multiple UEs simultaneously. For fairness, the computational capacity of EN is equally distributed to all UEs associated with the EN in each time slot. Once the input data of the subtasks are uploaded to the EN, the EN will immediately execute these subtasks. In this way, each EN will not incur waiting time.

[0062] The time required to execute the subtask of UE k on EN j is given by the subtask ratio β k,j , the number of CPU cycles required to complete the total task η k Decision, that is in represents the total number of associated users of EN j; is the computing power of each EN j, which is evenly distributed among all associated users. Therefore, the computing power allocated to UE k is

[0063] The cloud server provides fixed computing power for each UE k If there is a ratio of γ k,j The subtask is offloaded to EN j, which then forwards it to the cloud server for execution (x k,j =1), then the execution time of the cloud server is

[0064] To further illustrate, each edge node (EN) can measure the uplink signal to interference and noise ratio (SINR) of its associated user equipment (UE), denoted as θ k,j The specific process is that at the beginning of each time slot, the UE sends a pilot or beacon signal to the EN; the EN uses the received signal to estimate the channel state information (CSI) and calculate the corresponding SINR value.

[0065] To further illustrate, let W be the access channel bandwidth of each EN, then the uplink data rate when UE k is associated with EN j is in is the load of EN j, i.e., the number of UEs associated with it. k,j ≥θ th (preset threshold) can be associated with EN j. The optional EN set of UE k is π k ={j|θ k,j ≥θ th}, like Then x k,j = 0. In addition, the number of UEs that each EN j can serve has an upper limit S j , usually determined by the number of physical channels of the EN.

[0066] Based on the above, for the subtask of UE k, if the allocation ratio is β k,j and γ k,j , then the time it takes to transmit to EN is k∈K,j∈J.

[0067] To further illustrate, assume that the backhaul link between EN j and the cloud server is a wired connection with a bandwidth of M j , and the backhaul bandwidth is evenly distributed to all UEs associated with EN j. Therefore, the total time required to upload the subtask of UE k to the cloud server through EN j is k∈K,j∈J, where is the return link transmission time.

[0068] To further illustrate, it is assumed that the output data volume of the task execution is small, so the delay of returning the task results from the cloud server or EN is negligible.

[0069] To further illustrate, consider a case of a divisible computing task, where the task can be divided into multiple independent subtasks and distributed to local devices, edge nodes (ENs), and cloud servers for parallel processing. For example, in a video object recognition task, the video is segmented into several segments and processed locally, on ENs, and on cloud servers.

[0070] Based on the above assumptions, the task completion time on the local device is The completion time of the task after it is transmitted from UE k to EN j is in is the transmission time for uploading to EN, is the execution time of EN. The task is uploaded to EN by UE k, and then forwarded to the cloud server by EN. The total time to return the result after completion is in Including the transmission time of upload and return links, is the computing time of the cloud server.

[0071] To further explain, because independent subtasks are used, the total latency of a task is equal to the latency of the last set of subtasks completed in the local device, edge node (EN), and cloud server. Since the sum of the proportions of the three parts of the computing task is 1, reducing the proportion of one part will inevitably lead to an increase in the proportion of at least one other part. Therefore, Figure 3 As shown in , the optimal task division can be achieved when the subtasks on the local device, edge node, and cloud server are completed almost simultaneously. Based on this, we can get the equation To indicate that the completion time of each part is equal, where represents the time required for UE k to execute the task assigned to it locally, represents the time required for UE k’s subtask j to execute on the edge node, represents the time required for UE k’s subtask j to execute on the cloud server, x k,j represents the ratio coefficient assigned to each subtask. According to the derivation of the delay in the previous article, the optimal task division ratio of UE k can be solved. and Combined with the task division ratio being 1, the optimal computing task division ratio can be determined, that is, the lower-level problem is solved.

[0072] In the case of user association, the total task completion delay Γ k is the maximum of the following three: When users are not associated, the delay is Therefore, the goal of the high-level question is Equivalent to Since the high-level problem is an integer programming problem, the present invention relaxes it and uses the dual decomposition method to obtain an approximate optimal solution for the relaxed problem. After partially relaxing the constraints of the high-level problem, the present invention obtains the Lagrangian function where λ = {λ 1 ,...,λ J} is the Lagrange multiplier of the constraint. The high-level problem can be solved by To get the approximate optimal solution. The specific solution process is as follows:

[0073] The first step is to set reasonable initial Q and λ.

[0074] In the second step, the UE selects EN j with the best load and signal quality.

[0075] In the third step, after receiving the selection from each UE, EN j determines the user association, calculates the gradient, and adjusts λ by the gradient method. j , so that the system gradually approaches the optimal state, and finally updates the flow load Q j .

[0076] In the fourth step, after updating the Lagrange multiplier and traffic load, the EN broadcasts the updated information to nearby UEs.

[0077] In the fifth step, the UE initiates the next round of EN selection process according to the broadcast update information until a global approximate optimal solution is reached.

[0078] S3. After the task allocation is completed, the system calculates the average delay of the user and the total energy consumption of the base station, uses the user delay and the total energy consumption of the base station to calculate the cost function, and generates the corresponding reward value. The system updates the parameters of the deep reinforcement learning model according to the reward value and further adjusts the dormant state of the base station.

[0079] Specifically, the total power consumption of the base station P j It includes two parts: static power consumption and dynamic power consumption, namely Indicates the static power consumption when the base station is turned on, which is related to the fixed configuration of the base station's hardware equipment, cooling system, etc. This part of the power consumption has nothing to do with the working state of the base station and is the basic energy consumption of the base station operation. Represents dynamic power consumption, which is determined by the number of user equipment (UE) served by the base station and the power consumption of the active radio frequency chain (RF chain). Indicates the power consumption of a single RF chain. It represents the number of UEs served by base station j, which is equal to the number of active RF chains of the base station.

[0080] To further explain, the system cost function is the sum of the total user delay and the total energy consumption of the base station accumulated at multiple times, that is, Among them, ω represents the weight, y j represents the sleep state variable of base station j (0 means sleep, 1 means not sleep), and there is a constraint x i,j ≤y, j k∈K,j∈J, that is, the UE can only associate with the base station j which is not in the sleep state.

[0081] To further explain, the base station sleep control problem is modeled as a Markov Decision Process (MDP), in which the network controller acts as an agent responsible for deciding the sleep control strategy of the base stations in the area. Specifically, the system state in the MDP consists of the following information: the load of the edge computing server and the cloud server at the current moment, the channel state of the user, the transmission rate of the user, and the distribution of the user's requested tasks. The agent selects the action according to the current system state, that is, the working or sleeping decision of all base stations. The reward at each moment is defined as the negative value of the current cost function. The goal of the agent is to maximize the sum of the discounted rewards at the current moment and all future moments through continuous interaction.

[0082] For solving MDP, a method based on deep reinforcement learning can be used. This method builds an intelligent agent that can interact with the environment and continuously optimizes its decision-making strategy during the training process to achieve comprehensive optimization of latency and energy consumption. Specifically, the agent first perceives the current system state, including the load of edge computing servers and cloud servers, the channel state of users, the transmission rate of users, and task distribution information. Based on the current state, the agent outputs a base station sleep action through the Deep Q-Network (DQN). After executing the sleep action, the environment calculates the user delay and base station energy consumption based on the sleep state of the base station, the task allocation of users, and the transmission performance, and generates an immediate reward value based on the cost function and feeds it back to the agent. The agent uses the reward signal to evaluate the pros and cons of the action and updates the parameters of its deep Q network, thereby gradually optimizing the decision-making strategy. Through multiple rounds of state updates and action optimization, the agent can learn how to make the best base station sleep decision under different network conditions, and finally achieve comprehensive optimization of latency and energy consumption.

[0083] S4. The system repeats the process of task allocation and base station sleep state adjustment, and gradually optimizes through multiple rounds of iterations until the solution converges, and finally obtains a comprehensive optimization result close to the global optimal.

[0084] Example 2

[0085] The present invention also provides a comprehensive optimization system for base station sleep control and computing task distribution based on deep reinforcement learning, including:

[0086] The network controller is used to collect various status information of other devices in the network, such as user devices, edge nodes, and cloud servers, and solve the optimal strategy for base station sleep and computing task distribution based on this information and preset optimization goals, and issue the strategy to guide network devices to adjust their behavior in response to new instructions. In the sub-problem of base station sleep strategy based on deep reinforcement learning, the network controller is used as an agent to solve the sleep strategy. In the sub-problem of computing task allocation, the network controller is also a solving unit for user association and computing task distribution.

[0087] User devices are used to generate computing tasks and communicate with edge nodes or cloud servers.

[0088] The edge node, which is integrated by the edge server and the base station, is used to receive task requests from user devices and perform task calculations or forward them to the cloud server.

[0089] Cloud servers are used to receive tasks forwarded by edge nodes and complete highly complex computing tasks.

[0090] Example 3

[0091] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0092] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0093] Example 4

[0094] The embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the method of the above embodiment of the present invention.

[0095] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning, characterized in that: The method comprises the following steps: S1, constructing a multi-user edge computing system consisting of a cloud server and multiple edge nodes, wherein the edge nodes (EN) include edge servers and base stations; S2, setting the initial sleep state of the base station, and assigning computing tasks to the user equipment, edge server and cloud server respectively according to the current sleep state of the base station and determining the proportion of subtasks assigned to each computing node; S3, after the task allocation is completed, the average delay of the user and the total energy consumption of the base station are calculated, and an immediate reward value is generated based on the cost function. The parameters of the deep reinforcement learning model are updated according to the reward value, and the sleep state of the base station is further adjusted; S4, repeatedly performing task allocation and base station sleep state adjustment, and gradually optimizing through multiple rounds of iterations until a comprehensive optimization result close to the global optimum is obtained.

2. According to claim 1, the comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning is characterized in that: In step S2, the association relationship between the user and the base station is also determined, which specifically includes: The association relationship between the user (UE) and the edge node (EN) is given by In order to ensure that each user (UE) is associated with at most one base station (EN), x k,j Need to satisfy constraints 3. The comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning according to claim 2 is characterized in that: The calculation task allocation in step S2 includes the following steps: for a user (UE) associated with EN j, the generated task can be allocated to EN j, the cloud server, or processed by the user (UE) itself. Each task is divided into several subtasks, each of which corresponds to a different data size; the ratio of task allocation is expressed by α k , β k,j and γ k,j In which, α k represents the proportion of tasks handled by UE k itself, β k,j represents the proportion of tasks handled by EN j, γ k,j represents the proportion of tasks processed by the cloud server; assuming x k,j =1, then the division of tasks needs to satisfy the constraints: ① ② When x k,j =0, β k,j = 0 and γ k,j =0, indicating that the task is completely handled by the user terminal.

4. The comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning according to claim 3 is characterized in that: The time required to execute the assigned task locally is Among them, α k η k Indicates the number of CPU cycles required to complete the assigned task. represents the computing capability of UE k; Assuming that each EN is capable of executing subtasks received from multiple UEs simultaneously, for fairness, the computing power of the EN is equally distributed to all UEs associated with the EN in each time slot. Once the input data of the subtasks are uploaded to the EN, the EN will immediately execute these subtasks; The time required to execute the subtask of UE k on EN j is given by the subtask ratio β k,j , the number of CPU cycles required to complete the total task η k Decision, that is in represents the total number of associated users of EN j; is the computing power of each EN j, which is evenly distributed among all associated users. Therefore, the computing power allocated to UE k is The cloud server provides fixed computing power for each UE k If there is a ratio of γ k,j The subtask is offloaded to EN j, which then forwards it to the cloud server for execution (x k,j =1), then the execution time of the cloud server is 5. According to claim 4, the comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning is characterized in that: User transmission delay includes: Each edge node (EN) can measure the uplink signal to interference and noise ratio (SINR) of the user equipment (UE) associated with it, denoted as SINR k,j ,The specific process is that at the beginning of each time slot, the UE sends a pilot or beacon signal to the EN; the EN uses the received signal to perform channel state information (CSI) estimation and calculate the corresponding SINR value; Assume W is the access channel bandwidth of each EN, then the uplink data rate when UE k is associated with EN j is in is the load of EN j, i.e., the number of UEs associated with it; UE k is only k,j ≥ SINR th (preset threshold) can be associated with EN j; the optional EN set of UE k is like Then x k,j =0; In addition, the number of UEs that each EN j can serve has an upper limit S j , usually determined by the number of physical channels of the EN; For the subtask of UE k, if the allocation ratio is β k,j and γ k,j , the task size is s k , then the time it takes to transmit to EN is The backhaul link between EN j and the cloud server is a wired connection with a bandwidth of M j , and the backhaul bandwidth is evenly distributed to all UEs associated with ENj; therefore, the total time required to upload the subtask of UE k to the cloud server through EN j is in is the return link transmission time.

6. The comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning according to claim 5 is characterized in that: The total user delay includes: Consider a case of divisible computing tasks, where the task can be divided into multiple independent subtasks and assigned to local devices, edge nodes (ENs), and cloud servers for parallel processing; in the video object recognition task, the video is segmented into several segments and processed locally, on ENs, and on cloud servers respectively. The task is completed on the local device in The completion time of the task after it is transmitted from UE k to EN j is in is the transmission time for uploading to EN, is the execution time of EN; The task is uploaded to EN by UE k, and then forwarded to the cloud server by EN. The total time to return the result after completion is in Including the transmission time of upload and return links, is the computing time of the cloud server; Since the subtasks are independent and can be executed in parallel, the completion time of the entire task depends on the last completed part in the local device, EN, and cloud server; let x k,j =1 means UE k selects EN j as the associated node, then the total task completion delay T k is the maximum of the following three:

7. The comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning according to claim 6 is characterized in that: In step S3, the base station energy consumption includes: The total power consumption of the base station P j It includes two parts: static power consumption and dynamic power consumption, namely Indicates the static power consumption when the base station is turned on, which is related to the fixed configuration of the base station such as hardware equipment and cooling system; Represents dynamic power consumption, which is determined by the number of user equipment (UE) served by the base station and the power consumption of the active radio frequency chain (RF chain); Indicates the power consumption of a single RF chain; It represents the number of UEs served by base station j, which is equal to the number of active RF chains of the base station.

8. The comprehensive optimization method for base station sleep control and computing task distribution based on deep reinforcement learning according to claim 7 is characterized in that: In step S3, the cost function is the weighted sum of the total user delay and the total energy consumption of the base station accumulated at multiple times, specifically: Among them, ω represents the weight, y j represents the sleep state variable of base station j (0 means sleep, 1 means not sleep), and there is a constraint That is, the UE can only associate with the base station j that is not in the dormant state.

9. A comprehensive optimization system for base station sleep control and computing task distribution based on deep reinforcement learning, characterized in that: The system includes: The network controller is used to collect various status information of user devices, edge nodes and cloud servers in the network, and solve the optimal strategy for base station sleep and computing task distribution based on the status information and preset optimization goals, and issue the strategy to guide network devices to adjust their behavior in response to new instructions; it is also used to solve the base station sleep strategy based on deep reinforcement learning; it is also used to solve user association and computing task distribution in computing task allocation; User devices, used to generate computing tasks and communicate with edge nodes or cloud servers; The edge node is an integration of the edge server and the base station, which is used to receive task requests from user devices and perform task calculations or forward them to the cloud server; Cloud servers are used to receive tasks forwarded by edge nodes and complete computing tasks.

10. A computer-readable medium storing a computer program, wherein the computer-readable medium implements the steps of the method according to any one of claims 1 to 8 when the computer program is executed by a processor.

Images