Edge computing resource allocation optimization method and system based on reinforcement learning

Abstract

The invention discloses an edge computing resource allocation optimization method and system based on reinforcement learning, and relates to the technical field of edge computing resource management.The method comprises the steps that computing load data and network state data are collected, a historical load sequence is formed, a state space matrix is constructed, the historical load sequence is mapped into a discrete state value, and the discrete state value is obtained; establishing a deep neural network model; inputting the state space matrix into a deep neural network model, and generating a resource allocation strategy set according to the priority and time delay requirements of the calculation tasks; iteratively training the resource allocation strategy set by adopting a double Q learning algorithm, setting a reward function based on task completion time and a resource utilization rate, and selecting an optimal resource allocation strategy; and adjusting computing resource allocation of the edge nodes according to the optimal resource allocation strategy, and meanwhile, feeding back an execution result to the deep neural network model for online learning updating. According to the method, intelligent optimization of edge computing resource allocation is realized by introducing a deep reinforcement learning method.

Description

Technical Field

[0001] The present invention relates to the field of edge computing resource management technology, and in particular to an edge computing resource allocation optimization method and system based on reinforcement learning. Background Art

[0002] Edge computing reduces data transmission delays and improves system response speed and computing efficiency by moving data processing and storage capabilities to the edge of the network. However, traditional edge computing resource allocation methods often use static strategies or simple load balancing mechanisms, which are difficult to dynamically adapt to complex and changing computing environments and business needs, resulting in low computing resource utilization and large fluctuations in task processing delays.

[0003] The existing edge computing resource allocation technology has the following key limitations: First, traditional methods lack the ability to accurately perceive and dynamically model the computing load and network status of edge nodes, and resource allocation strategies are often too rigid; second, when dealing with resource allocation of heterogeneous edge devices, existing technologies find it difficult to simultaneously take into account task priorities, latency requirements, and resource utilization, and often result in unbalanced resource allocation or computing performance bottlenecks; third, there is a lack of effective adaptive learning mechanisms, and resource allocation strategies cannot be adjusted in a timely manner according to real-time load changes, resulting in difficulty in continuous optimization of the overall system performance.

[0004] In view of the above-mentioned shortcomings of the existing edge computing resource allocation technology, the present invention proposes an edge computing resource allocation optimization method based on reinforcement learning, which solves the key technical problems of dynamic resource allocation and continuous performance optimization in the edge computing environment. By introducing deep neural networks and double Q-learning algorithms, the present invention aims to achieve intelligent, efficient and adaptive allocation of edge computing resources. Summary of the invention

[0005] In view of the problems existing in the existing edge computing resource allocation technology, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to achieve intelligent, dynamic and efficient allocation of edge computing resources, accurately perceive and predict the computing load of edge nodes through deep reinforcement learning methods, and adaptively optimize resource allocation strategies according to task priority and latency requirements, so as to improve the overall performance and resource utilization of the edge computing system.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In the first aspect, an embodiment of the present invention provides an edge computing resource allocation optimization method based on reinforcement learning, which includes collecting computing load data and network status data of edge devices, recording CPU utilization, memory occupancy and bandwidth usage through a resource monitoring module of the edge node to form a historical load sequence; constructing a state space matrix according to the historical load sequence, and mapping the historical load sequence into discrete state values ​​to establish a deep neural network model; inputting the state space matrix into the deep neural network model, and generating a resource allocation strategy set according to the priority and delay requirements of the computing task; using a double Q learning algorithm to iteratively train the resource allocation strategy set, setting a reward function based on task completion time and resource utilization, and selecting an optimal resource allocation strategy; based on the optimal resource allocation strategy, adjusting the computing resource allocation of the edge node, and at the same time feeding back the execution results to the deep neural network model for online learning and updating.

[0009] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, wherein: based on the optimal resource allocation strategy, the computing resource allocation of the edge node is adjusted, and the execution result is fed back to the deep neural network model for online learning and updating, including: converting the resource allocation ratio in the optimal resource allocation strategy into resource configuration parameters, and issuing the resource configuration parameters through the resource management interface of the edge node, wherein the resource configuration parameters include CPU core quantity allocation, memory space allocation and network bandwidth allocation; adjusting the optimal resource allocation strategy in the order of task priority, and using a smooth transition strategy to switch resources for the optimal resource allocation strategy; based on the effect of resource switching, The current system status is monitored and recorded in real time, and the recorded system status is compared with the expected target, and the performance difference value and resource utilization deviation are calculated, wherein the system status includes the actual resource occupancy of each task, the task execution progress and the performance index; a learning sample set of the system status is established by a sliding window mechanism, and the deep neural network model is incrementally updated by mini-batch gradient descent, wherein the learning sample set includes the initial state, the resource allocation strategy and the execution effect; the performance difference value and the resource utilization deviation are counted, and when the resource utilization deviation is greater than the second preset threshold value and the performance difference value is greater than the third preset threshold value, the structural adaptive adjustment mechanism of the deep neural network model is triggered to optimize the resource allocation strategy.

[0010] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, wherein: the method for obtaining the optimal resource allocation strategy is to construct a dual Q learning algorithm with the same network structure, wherein the dual Q learning algorithm includes a main Q network and a target Q network, and the network structure includes an input layer, a hidden layer and an output layer; set a reward function according to the task completion time and resource utilization, and balance the task processing efficiency and resource utilization efficiency through this reward function, wherein the reward function includes a task completion time reward and a resource utilization reward; use the ε-greedy method to select an execution action from the resource allocation strategy set, and execute the selected resource allocation strategy to obtain an actual reward and a next state; store the resource state, the execution of the selected resource allocation strategy, the actual reward and the next state in an experience replay pool D, and randomly sample B experience samples from the experience replay pool D for training; for the experience sample, calculate the Q value of the current state through the main Q network, and calculate the target Q value of the next state through the target Q network, and update the main Q network parameters according to the temporal difference error; repeat the above operation steps until the training converges, and select the resource allocation strategy with the largest Q value in the current state as the optimal resource allocation strategy.

[0011] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, the specific formula of the reward function R is as follows:

[0012]

[0013] Among them, ω i is the weight coefficient of the i-th task, P i is the performance index of the i-th task, Λ ∈ (T c ) is the normalization factor of the task completion time T, ε is the control normalization degree, U r is the actual resource utilization, U t is the target resource utilization, σ is the smoothing parameter of resource utilization deviation, and n is the number of tasks.

[0014] When the performance index of the task P i is greater than the first preset threshold, and the actual resource utilization U r Equal to the target resource utilization U t When , the reward function R will give a positive reward; when the task completion time T is greater than 50% of the expected task completion time and the resource utilization fluctuation U is greater than 15% of the expected resource utilization fluctuation, the reward function R will give a negative reward and punish inefficient resource allocation strategies; when the task priority weight ω iIf a task is not completed on time, the reward function R will give a negative penalty; if the system has several parallel tasks and there is resource contention, the reward function R will dynamically adjust the reward value according to the importance of the task and the urgency of execution.

[0015] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, wherein: the method for generating the resource allocation strategy set is to establish a feature vector for the computing task according to the historical load sequence, and use the priority scoring function to calculate the task priority, wherein the feature vector includes the task urgency, task importance and task resource requirement; sort the computing tasks according to the priority score, generate a task priority queue, and convert the delay requirement of the computing task into a constraint condition for resource allocation, wherein the constraint condition includes a minimum resource guarantee constraint and a maximum resource upper limit constraint; based on the task priority queue and the constraint condition, input the state space matrix into the trained deep neural network model, obtain the evaluation value of each action in the current state, and use the Softmax function to convert the evaluation value into an action probability distribution; generate a resource allocation strategy set according to the action probability distribution sampling, and calculate the evaluation index of each resource allocation strategy, wherein each resource allocation strategy includes the computing resource allocation ratio of the edge node; the evaluation index includes a resource utilization efficiency index and a task completion time index; mechanically standardize the evaluation index, and obtain the comprehensive score of the resource allocation strategy by linear weighting, and retain the k resource allocation strategy sets with the highest scores {A1, A2, ..., A k1}.

[0016] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, wherein: a state space matrix is ​​constructed according to the historical load sequence, and the historical load sequence is mapped into discrete state values, and a deep neural network model is established, including: the CPU utilization, memory occupancy and bandwidth usage in the historical load sequence are divided into several discrete state intervals through the K-means clustering algorithm, and the discrete state intervals are assigned corresponding numerical identifiers to form a discretized feature vector; the discretized feature vector is organized into an M×N state space matrix according to the time dimension, and a time sliding window mechanism is introduced to capture state transition features, where M represents the number of monitoring indicators and N represents the length of the time series; based on the state transition features, a multi-layer perceptron structure is designed, and a ReLU activation function is used in the hidden layer to construct a deep neural network model; the Adam optimizer is used, and the parameters of the deep neural network model are adjusted and optimized in combination with the learning rate decay strategy, and the difference between the predicted result and the actual state is measured by the cross entropy loss function to complete the training of the deep neural network model.

[0017] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, wherein: the method for forming the historical load sequence is to sample the system status of the edge device based on the resource monitoring module of the edge node; obtain the CPU utilization by reading the processor usage percentage, wherein the CPU utilization includes idle time, user time and system time; obtain the memory occupancy rate by counting the ratio of the used space of the physical memory to the total space, and record the memory page replacement frequency at the same time; obtain the bandwidth usage by monitoring the data packet sending and receiving rate of the network interface, wherein the bandwidth usage includes the real-time value of the uplink bandwidth and the real-time value of the downlink bandwidth; mark and store the CPU utilization, the memory occupancy rate and the bandwidth usage according to the timestamp, and use the sliding time window method to form a historical load sequence from the monitoring data at several consecutive time points.

[0018] In the second aspect, an embodiment of the present invention provides an edge computing resource allocation optimization system based on reinforcement learning, which includes: a collection module, which is used to collect computing load data and network status data of edge devices, and record CPU utilization, memory occupancy and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence; an establishment module, which constructs a state space matrix according to the historical load sequence, and maps the historical load sequence into discrete state values ​​to establish a deep neural network model; a generation module, which is used to input the state space matrix into the deep neural network model, and generate a resource allocation strategy set according to the priority and delay requirements of the computing task; a selection module, which is used to iteratively train the resource allocation strategy set using a double Q learning algorithm, set a reward function based on task completion time and resource utilization, and select the optimal resource allocation strategy; an adjustment and update module, which adjusts the computing resource allocation of the edge node based on the optimal resource allocation strategy, and at the same time feeds back the execution results to the deep neural network model for online learning and updating.

[0019] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the edge computing resource allocation optimization method based on reinforcement learning as described in the first aspect of the present invention are implemented.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the edge computing resource allocation optimization method based on reinforcement learning as described in the first aspect of the present invention are implemented.

[0021] The beneficial effects of the present invention are as follows: the present invention realizes dynamic and accurate perception of edge computing resources through refined resource monitoring and state space matrix construction; based on deep neural network and dual Q learning algorithm, it can adaptively generate optimal resource allocation strategy according to task priority and delay requirements; establishes a closed-loop system of resource allocation-execution-feedback-learning, so that edge computing resource allocation has continuous learning and self-optimization capabilities, effectively improves the resource utilization and computing efficiency of the system, and solves the key technical problems of unbalanced resource allocation and poor adaptability of traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0023] Figure 1 This is a flow chart of the edge computing resource allocation optimization method based on reinforcement learning in Example 1.

[0024] Figure 2 This is a system block diagram of the edge computing resource allocation optimization method based on reinforcement learning in Example 1. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0027] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0028] Example 1

[0029] Reference Figure 1-2 , which is the first embodiment of the present invention, and provides an edge computing resource allocation optimization method based on reinforcement learning, including:

[0030] S1: Collect computing load data and network status data of edge devices, record CPU utilization, memory occupancy, and bandwidth usage through the resource monitoring module of the edge node, and form a historical load sequence.

[0031] Specifically, the method for forming the historical load sequence is to sample the system status of the edge device based on the resource monitoring module of the edge node; obtain the CPU utilization by reading the percentage of processor usage; obtain the memory occupancy rate by counting the ratio of the used space of the physical memory to the total space, and record the memory page replacement frequency; obtain the bandwidth usage by monitoring the data packet sending and receiving rate of the network interface

[0032] It should be noted that CPU utilization includes idle time, user time, and system time; bandwidth usage includes real-time values ​​of uplink bandwidth and downlink bandwidth.

[0033] Furthermore, the CPU utilization, memory occupancy and bandwidth usage are marked and stored according to timestamps, and the monitoring data of several consecutive time points are formed into a historical load sequence using a sliding time window method.

[0034] S2: construct a state space matrix according to the historical load sequence, map the historical load sequence into discrete state values, and establish a deep neural network model.

[0035] Specifically, the CPU utilization, memory occupancy, and bandwidth usage in the historical load sequence are divided into several discrete state intervals through the K-means clustering algorithm, and the discrete state intervals are assigned corresponding numerical labels to form a discretized feature vector.

[0036] It should be noted that the CPU utilization is divided into: when the CPU utilization is between 0-30%, it is judged as a low load state; when the CPU utilization is between 30%-70%, it is judged as a medium load state; when the CPU utilization is between 70%-100%, it is judged as a high load state. The memory usage is divided into: when the memory utilization is between 0-20%, it is judged as an idle state; when the memory utilization is between 20%-60%, it is judged as a normal state; when the memory utilization is between 60%-80%, it is judged as a tense state; when the memory utilization is between 80%-100%, it is judged as a saturated state. The bandwidth usage is divided into: when the bandwidth utilization is between 0-40%, it is judged as a low speed state; when the bandwidth utilization is between 40%-80%, it is judged as a medium speed state; when the bandwidth utilization is between 80%-100%, it is judged as a high speed state.

[0037] Furthermore, the discretized feature vectors are organized into an M×N state space matrix according to the time dimension, and a time sliding window mechanism is introduced to capture the state transition characteristics, where M represents the number of monitoring indicators and N represents the length of the time series. Based on the state transition characteristics, a multi-layer perceptron structure is designed, and the ReLU activation function is used in the hidden layer to construct a deep neural network model.

[0038] Furthermore, the Adam optimizer is used in combination with the learning rate decay strategy to adjust and optimize the parameters of the deep neural network model. At the same time, the difference between the predicted results and the actual state is measured through the cross entropy loss function to complete the training of the deep neural network model.

[0039] S3: Input the state space matrix into the deep neural network model, and generate a resource allocation strategy set according to the priority and latency requirements of the computing task.

[0040] Specifically, the method for generating the resource allocation strategy set is to establish a feature vector for the computing task according to the historical load sequence, and use the priority scoring function to calculate the task priority, where the feature vector includes the task urgency, task importance and task resource demand.

[0041] It should be noted that when the task urgency in the feature vector exceeds 0.8, the task importance is greater than 0.7, and the resource demand is less than 20% of the system available resources, it is an urgent task; if it is an urgent task, the task delay requirement is a strict delay-tolerance of 5ms; when the task urgency in the feature vector is between 0.4-0.8, the task importance is in the range of 0.4-0.7, and the resource demand accounts for 20%-50% of the system available resources, it is an ordinary task; if it is an ordinary task, the task delay requirement is an ordinary delay-tolerance of 20ms; when the task urgency in the feature vector is less than 0.4, the task importance is less than 0.4, and the resource demand exceeds 50% of the system available resources, it is a low-level task; if it is a low-level task, the task delay requirement is a loose delay-tolerance of 50ms.

[0042] Furthermore, computing tasks are sorted according to priority scores to generate a task priority queue, and the latency requirements of computing tasks are converted into resource allocation constraints, where the constraints include minimum resource guarantee constraints and maximum resource upper limit constraints. Based on the task priority queue and constraints, the state space matrix is ​​input into the trained deep neural network model to obtain the evaluation value of each action in the current state, and the Softmax function is used to convert the evaluation value into action probability distribution.

[0043] Furthermore, the evaluation value is composed of three key dimensions; the first dimension is the task priority score, which is adjusted by the weight coefficient to adjust its influence on the overall evaluation value; the second dimension is the resource utilization rate, which is also adjusted by the weight coefficient to reflect the influence of the current resource status of the system on the action selection; the third dimension is the resource allocation constraint, which is mainly reflected by the ratio of the current resource allocation to the maximum resource upper limit, ensuring that the resource allocation does not exceed the system carrying capacity and smoothing the impact of the excess part; these three dimensions are comprehensively generated into the final evaluation value through weighted summation.

[0044] It should be noted that the Softmax function is a standardized exponential function that converts a set of raw evaluation values ​​into a probability distribution. Its core function is to map any real value to the interval (0,1) and ensure that the sum of all probability values ​​is always 1. The specific conversion process is to take the exponent of each evaluation value and then divide it by the sum of all evaluation value exponents to obtain the probability distribution of each action.

[0045] Specifically, a resource allocation strategy set is generated according to action probability distribution sampling, and the evaluation index of each resource allocation strategy is calculated.

[0046] It should be noted that each resource allocation strategy includes the computing resource allocation ratio of the edge nodes; the evaluation indicators include resource utilization efficiency indicators and task completion time indicators.

[0047] Furthermore, the evaluation indicators are mechanically standardized, and the comprehensive scores of resource allocation strategies are obtained through linear weighting, and the k resource allocation strategy sets with the highest scores {A1, A2, ..., A k1}.

[0048] S4: Adopting the double Q learning algorithm to iteratively train the resource allocation strategy set, setting the reward function based on the task completion time and resource utilization, and selecting the optimal resource allocation strategy.

[0049] Specifically, the method for obtaining the optimal resource allocation strategy is to construct a dual Q-learning algorithm with the same network structure; set a reward function according to the task completion time and resource utilization, and balance the task processing efficiency and resource utilization efficiency through this reward function.

[0050] It should be noted that the dual Q learning algorithm includes a main Q network and a target Q network; the network structure includes an input layer, a hidden layer and an output layer; the input layer receives the state space matrix, and the output layer generates a resource allocation strategy set; the reward function includes a task completion time reward and a resource utilization reward.

[0051] Furthermore, the specific formula of the reward function R is as follows:

[0052]

[0053] Among them, ω i is the weight coefficient of the i-th task, P i is the performance index of the i-th task, Λ ∈ (T c ) is the normalization factor of the task completion time T, ε is the control normalization degree, U r is the actual resource utilization, U t is the target resource utilization, σ is the smoothing parameter of resource utilization deviation, and n is the number of tasks.

[0054] Furthermore, when the performance index P of the task i is greater than the first preset threshold, and the actual resource utilization U r Equal to the target resource utilization U t When , the reward function R will give a positive reward; when the task completion time T is greater than 50% of the expected task completion time and the resource utilization fluctuation U is greater than 15% of the expected resource utilization fluctuation, the reward function R will give a negative reward and punish inefficient resource allocation strategies; when the task priority weight ω i If a task is not completed on time, the reward function R will give a negative penalty; if the system has several parallel tasks and there is resource contention, the reward function R will dynamically adjust the reward value according to the importance of the task and the urgency of execution.

[0055] Specifically, the ε-greedy method is used to select execution actions from the resource allocation strategy set, and the selected resource allocation strategy is executed to obtain the actual reward and the next state; the resource state, the execution of the selected resource allocation strategy, the actual reward and the next state are stored in the experience replay pool D, and B experience samples are randomly sampled from the experience replay pool D for training.

[0056] For further empirical samples, the Q value of the current state is calculated through the main Q network, and the target Q value of the next state is calculated through the target Q network, and the main Q network parameters are updated according to the timing difference error; every C training steps, the main Q network parameters are copied to the target Q network to maintain the delay of the target network parameter update; the above steps are repeated until the training converges, and the resource allocation strategy with the largest Q value of the current state is selected as the optimal resource allocation strategy.

[0057] S5: Based on the optimal resource allocation strategy, the computing resource allocation of the edge nodes is adjusted, and the execution results are fed back to the deep neural network model for online learning and updating.

[0058] Specifically, the resource allocation ratio in the optimal resource allocation strategy is converted into resource configuration parameters, and the resource configuration parameters are issued through the resource management interface of the edge node; the optimal resource allocation strategy is adjusted according to the order of task priority, and a smooth transition strategy is used to switch the resources of the optimal resource allocation strategy; based on the effect of resource switching, the current system status is monitored and recorded in real time, and the recorded system status is compared with the expected target, and the performance difference value and resource utilization deviation are calculated.

[0059] It should be noted that resource configuration parameters include CPU core number allocation, memory space allocation, and network bandwidth allocation; system status includes the actual resource usage of each task, task execution progress, and performance indicators.

[0060] Furthermore, the specific formula for the performance difference is as follows:

[0061]

[0062] Among them, δ performance is the performance difference index, E i (t) is the actual execution efficiency of the i-th task, E ideal (t) is the target execution efficiency, ω is the weight coefficient, and n is the number of tasks.

[0063] Furthermore, the specific formula for resource utilization deviation is as follows:

[0064]

[0065] Among them, Ω utilization is the resource utilization deviation index, ρ j is the actual resource utilization of the jth resource dimension, ρ j,target is the target resource utilization, and m is the number of resource dimensions.

[0066] Specifically, a sliding window mechanism is used to establish a learning sample set of the system state, and the deep neural network model is incrementally updated through small-batch gradient descent, where the learning sample set includes the initial state, resource allocation strategy and execution effect; the performance difference value and the resource utilization deviation are counted, and when the resource utilization deviation is greater than the second preset threshold and the performance difference value is greater than the third preset threshold, the structural adaptive adjustment mechanism of the deep neural network model is triggered to optimize the resource allocation strategy.

[0067] It should be noted that the second preset threshold is based on the statistical fluctuation range of the resource utilization of edge nodes and the fluctuation range of the historical load series. By conducting long-term monitoring of the CPU utilization, memory occupancy and network bandwidth usage of multiple edge nodes, the standard deviation and peak deviation of their fluctuations are calculated, thereby determining a reasonable upper limit of resource utilization fluctuations; the third preset threshold is based on the discrete degree of performance indicators of different computing tasks and the correlation between tasks, and comprehensively considers multi-dimensional factors such as task execution delay, resource consumption differences, priority differences, etc. Through cluster analysis and performance dispersion evaluation of historical task execution data, a threshold that reflects the overall performance fluctuation of the system is set.

[0068] Further, such as Figure 2 As shown, this embodiment also provides an edge computing resource allocation optimization system based on reinforcement learning, including: a collection module, which is used to collect computing load data and network status data of edge devices, and record CPU utilization, memory occupancy and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence; an establishment module, which constructs a state space matrix according to the historical load sequence, and maps the historical load sequence into discrete state values ​​to establish a deep neural network model; a generation module, which is used to input the state space matrix into the deep neural network model, and generate a resource allocation strategy set according to the priority and delay requirements of the computing task; a selection module, which is used to iteratively train the resource allocation strategy set using a double Q learning algorithm, set a reward function based on task completion time and resource utilization, and select the optimal resource allocation strategy; an adjustment and update module, which adjusts the computing resource allocation of the edge node based on the optimal resource allocation strategy, and at the same time feeds back the execution results to the deep neural network model for online learning and updating.

[0069] This embodiment also provides a computer device, which is suitable for the edge computing resource allocation optimization method based on reinforcement learning, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the edge computing resource allocation optimization method based on reinforcement learning as proposed in the above embodiment.

[0070] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0071] This embodiment also provides a storage medium having a computer program stored thereon, which implements the following steps when executed by a processor: collecting computing load data and network status data of edge devices, recording CPU utilization, memory occupancy, and bandwidth usage through a resource monitoring module of an edge node, and forming a historical load sequence; constructing a state space matrix according to the historical load sequence, and mapping the historical load sequence into discrete state values, and establishing a deep neural network model; inputting the state space matrix into the deep neural network model, and generating a resource allocation strategy set according to the priority and latency requirements of the computing task; iteratively training the resource allocation strategy set using a double Q learning algorithm, setting a reward function based on task completion time and resource utilization, and selecting an optimal resource allocation strategy; adjusting the computing resource allocation of edge nodes based on the optimal resource allocation strategy, and feeding back the execution results to the deep neural network model for online learning updates.

[0072] In summary, the present invention realizes dynamic and accurate perception of edge computing resources through refined resource monitoring and state space matrix construction; based on deep neural network and dual Q learning algorithm, it can adaptively generate optimal resource allocation strategy according to task priority and delay requirements; establishes a closed-loop system of resource allocation-execution-feedback-learning, so that edge computing resource allocation has continuous learning and self-optimization capabilities, effectively improves the resource utilization and computing efficiency of the system, and solves the key technical problems of unbalanced resource allocation and poor adaptability of traditional methods.

[0073] Example 2

[0074] Referring to Table 1, which is the second embodiment of the present invention, this embodiment provides an edge computing resource allocation optimization method based on reinforcement learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0075] Specifically, 10 edge computing nodes were selected as test objects, and each edge node was equipped with an Intel i7-9700 processor (8 cores and 16 threads), 32GB DDR4 memory, and 1Gbps network bandwidth. In order to comprehensively evaluate the system performance, three typical application scenarios were designed: video stream analysis, sensor data processing, and mobile device task offloading. The experiment first deployed a resource monitoring module based on Prometheus, set the sampling interval to 100ms, and established a 15-minute sliding time window. The node_exporter was used to collect the CPU time proportion of various types, including idle time, user time, and system time. At the same time, memory_exporter was used to record the memory usage status, including physical memory occupancy, page replacement frequency, and other indicators. Network status monitoring uses network_exporter to record the real-time usage of upstream and downstream bandwidths respectively.

[0076] Furthermore, in the data preprocessing stage, an improved K-means clustering algorithm was used to verify the best clustering effect through the silhouette coefficient. After multiple rounds of testing, the optimal partitioning intervals of CPU utilization, memory occupancy, and bandwidth utilization were determined. In order to improve the accuracy of feature extraction, a data preprocessing method based on exponential smoothing was introduced to effectively reduce the impact of data noise on the clustering effect.

[0077] Furthermore, the deep neural network model adopts a 4-layer structure, including 1 input layer (the input dimension is M×N, M=3 represents the number of monitoring indicators, and N=900 represents the number of sampling points within 15 minutes), 2 hidden layers (the number of neurons is 256 and 128 respectively) and 1 output layer. The hidden layer uses LeakyReLU as the activation function to solve the problem of neuron death that may be caused by traditional ReLU. The initial value of the learning rate is set to 0.001, and the cosine annealing strategy is used for dynamic adjustment. During the training process, the improved Adam optimizer is used, and the Warmup mechanism is introduced. The learning rate is gradually increased in the early stage of training, which effectively improves the convergence stability of the model. To prevent overfitting, a Dropout layer is added between the two hidden layers, and the loss rate is set to 0.3. At the same time, batch normalization technology is used to accelerate the training process, and the batch size is set to 64.

[0078] Specifically, as shown in Table 1, from the perspective of resource utilization, the average CPU utilization of the 8 test nodes is distributed between 28.3% and 75.2%, indicating that the system dynamically adjusts resource allocation according to the actual load situation to avoid over-allocation or under-allocation of resources. In particular, although the CPU utilization of the Edge-03 node is low (28.3%), its task completion rate reaches 99.2%, and the average response time is only 8.9ms, indicating that the system successfully identifies the light load scenario and appropriately reduces resource allocation, thereby achieving a higher resource utilization efficiency (92.1%). In terms of memory management, this solution maintains the memory occupancy rate in a reasonable range through accurate load prediction and dynamic adjustment strategies. The data shows that even under high load conditions (such as the memory occupancy rate of the Edge-04 node is 82.4%), the system can still maintain a task completion rate of 95.6%, thanks to the accurate division of memory states by the improved K-means clustering algorithm and the accurate prediction ability of the deep neural network model. The utilization of bandwidth resources also shows good adaptability. The bandwidth utilization rate of each node is positively correlated with the business load, but none of them has reached saturation. For example, the Edge-07 node still maintains a 96.4% task completion rate and 85.3% resource utilization efficiency at a high bandwidth utilization rate (58.9%), verifying the optimization effect of the system in network resource scheduling.

[0079] Table 1 Edge computing resource allocation performance evaluation data

[0080] Node ID Average CPU utilization Memory usage Bandwidth usage Task completion rate Average response time Resource efficiency Edge-01 45.6％ 58.3％ 35.2％ 98.5％ 12.3ms 89.4％ Edge-02 62.8％ 72.1％ 45.6％ 97.8％ 15.7ms 86.2％ Edge-03 28.3％ 45.6％ 28.9％ 99.2％ 8.9ms 92.1％ Edge-04 75.2％ 82.4％ 68.7％ 95.6％ 18.4ms 83.5％ Edge-05 52.1％ 63.5％ 42.3％ 98.1％ 13.8ms 88.7％ Edge-06 38.7％ 51.2％ 32.8％ 98.9％ 10.5ms 90.8％ Edge-07 68.4％ 77.8％ 58.9％ 96.4％ 16.9ms 85.3％ Edge-08 42.9％ 56.7％ 38.4％ 98.7％ 11.8ms 89.9％

[0081] Furthermore, from the performance indicators, the average response time of all test nodes is controlled within 20ms, with the best value being 8.9ms for Edge-03 node and the maximum value being 18.4ms for Edge-04 node. This result is significantly better than the traditional fixed allocation scheme (usually the response time is between 30-50ms). At the same time, the task completion rate is generally maintained above 95%, which fully proves the effectiveness of the dual Q learning algorithm in optimizing resource allocation strategies.

[0082] Furthermore, the resource utilization efficiency index of the eight nodes averaged 88.2%, with a maximum of 92.1% and a minimum of 83.5%, which is about 20-30% higher than the traditional solution. This improvement is mainly due to the adaptive learning mechanism and precise state space modeling method introduced in this solution.

[0083] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for optimizing edge computing resource allocation based on reinforcement learning, characterized in that: include, Collect computing load data and network status data of edge devices, and record CPU utilization, memory occupancy, and bandwidth usage through the resource monitoring module of edge nodes to form a historical load sequence; Constructing a state space matrix according to the historical load sequence, mapping the historical load sequence into discrete state values, and establishing a deep neural network model; Inputting the state space matrix into the deep neural network model, and generating a resource allocation strategy set according to the priority and latency requirements of the computing task; Iteratively train the resource allocation strategy set using a double Q learning algorithm, set a reward function based on task completion time and resource utilization, and select the optimal resource allocation strategy; Based on the optimal resource allocation strategy, the computing resource allocation of the edge nodes is adjusted, and the execution results are fed back to the deep neural network model for online learning and updating.

2. The edge computing resource allocation optimization method based on reinforcement learning according to claim 1, characterized in that: Based on the optimal resource allocation strategy, the computing resource allocation of the edge nodes is adjusted, and the execution results are fed back to the deep neural network model for online learning and updating, including: Convert the resource allocation ratio in the optimal resource allocation strategy into resource configuration parameters, and issue the resource configuration parameters through the resource management interface of the edge node, wherein the resource configuration parameters include CPU core quantity allocation, memory space allocation and network bandwidth allocation; Adjusting the optimal resource allocation strategy according to the order of task priority, and using a smooth transition strategy to switch resources for the optimal resource allocation strategy; Based on the effect of resource switching, the current system status is monitored and recorded in real time, and the recorded system status is compared with the expected target to calculate the performance difference value and resource utilization deviation, where the system status includes the actual resource occupancy of each task, task execution progress and performance indicators; A sliding window mechanism is used to establish a learning sample set of system status, and the deep neural network model is incrementally updated through mini-batch gradient descent. The learning sample set includes the initial state, resource allocation strategy, and execution effect. The performance difference value and the resource utilization deviation are counted. When the resource utilization deviation is greater than the second preset threshold and the performance difference value is greater than the third preset threshold, the structural adaptive adjustment mechanism of the deep neural network model is triggered to optimize the resource allocation strategy.

3. The edge computing resource allocation optimization method based on reinforcement learning as claimed in claim 2, characterized in that: The method for obtaining the optimal resource allocation strategy is: Constructing a dual Q learning algorithm with the same network structure, wherein the dual Q learning algorithm comprises a main Q network and a target Q network, and the network structure comprises an input layer, a hidden layer and an output layer; A reward function is set according to the task completion time and resource utilization rate, and the task processing efficiency and resource utilization efficiency are balanced through the reward function, wherein the reward function includes a task completion time reward and a resource utilization rate reward; An ε-greedy method is used to select an execution action from the resource allocation strategy set, and the selected resource allocation strategy is executed to obtain an actual reward and a next state; The resource state, the selected resource allocation strategy, the actual reward, and the next state are stored in the experience replay pool D, and B experience samples are randomly sampled from the experience replay pool D for training; For the experience sample, the Q value of the current state is calculated by the main Q network, and the target Q value of the next state is calculated by the target Q network, and the main Q network parameters are updated according to the timing difference error; Repeat the above steps until the training converges, and select the resource allocation strategy with the largest Q value in the current state as the optimal resource allocation strategy.

4. The edge computing resource allocation optimization method based on reinforcement learning as claimed in claim 3, characterized in that: The specific formula of the reward function R is as follows: Among them, ω i is the weight coefficient of the i-th task, P i is the performance index of the i-th task, Λ ∈ (T c ) is the normalization factor of the task completion time T, ε is the control normalization degree, U r is the actual resource utilization, U t is the target resource utilization, σ is the smoothing parameter of resource utilization deviation, and n is the number of tasks; When the performance index of the task P i is greater than the first preset threshold, and the actual resource utilization U r Equal to the target resource utilization U t When , the reward function R will give a positive reward; When the task completion time T is greater than 50% of the expected task completion time and the resource utilization fluctuation U is greater than 15% of the expected resource utilization fluctuation, the reward function R will give a negative reward and punish inefficient resource allocation strategies; When the task priority weight ω i If the task is not completed on time, the reward function R will give a negative penalty; If the system has several parallel tasks and there is resource contention, the reward function R will dynamically adjust the reward value according to the importance and execution urgency of the task.

5. The edge computing resource allocation optimization method based on reinforcement learning as claimed in claim 3, characterized in that: The method for generating the resource allocation strategy set is: Establishing a feature vector for the computing task according to the historical load sequence, and calculating the task priority using a priority scoring function, wherein the feature vector includes task urgency, task importance, and task resource requirement; Sorting the computing tasks according to the priority scores, generating a task priority queue, and converting the latency requirements of the computing tasks into resource allocation constraints, wherein the constraints include a minimum resource guarantee constraint and a maximum resource upper limit constraint; Based on the task priority queue and the constraint conditions, the state space matrix is ​​input into the trained deep neural network model to obtain the evaluation value of each action in the current state, and the evaluation value is converted into the action probability distribution by using the Softmax function; Generate a resource allocation strategy set according to the action probability distribution sampling, and calculate the evaluation index of each resource allocation strategy, wherein each resource allocation strategy includes the computing resource allocation ratio of the edge node; the evaluation index includes a resource utilization efficiency index and a task completion time index; The evaluation indicators are mechanically standardized, and the comprehensive scores of resource allocation strategies are obtained by linear weighting, and the k resource allocation strategy sets with the highest scores {A1, A2, ..., A k1 }.

6. The edge computing resource allocation optimization method based on reinforcement learning as claimed in claim 5, characterized in that: A state space matrix is ​​constructed according to the historical load sequence, and the historical load sequence is mapped into discrete state values, and a deep neural network model is established, including: The CPU utilization, memory occupancy and bandwidth usage in the historical load sequence are divided into several discrete state intervals by using the K-means clustering algorithm, and the discrete state intervals are assigned corresponding numerical identifiers to form a discretized feature vector; The discretized feature vectors are organized into an M×N state space matrix according to the time dimension, and a time sliding window mechanism is introduced to capture the state transition characteristics, where M represents the number of monitoring indicators and N represents the length of the time series. Based on the state transition characteristics, a multi-layer perceptron structure is designed, and a ReLU activation function is used in the hidden layer to construct a deep neural network model; The Adam optimizer is used in combination with the learning rate decay strategy to adjust and optimize the parameters of the deep neural network model. At the same time, the difference between the predicted result and the actual state is measured by the cross entropy loss function to complete the training of the deep neural network model.

7. The edge computing resource allocation optimization method based on reinforcement learning according to claim 6, characterized in that: The method for forming the historical load sequence is: Based on the resource monitoring module of edge nodes, the system status of edge devices is sampled; Obtaining CPU utilization by reading the usage percentage of the processor, wherein the CPU utilization includes idle time, user time, and system time; The memory usage rate is obtained by counting the ratio of the used space to the total space of the physical memory, and the memory page replacement frequency is recorded at the same time; Obtaining bandwidth usage by monitoring the data packet receiving and sending rate of the network interface, wherein the bandwidth usage includes a real-time value of the uplink bandwidth and a real-time value of the downlink bandwidth; The CPU utilization, the memory occupancy and the bandwidth usage are marked and stored according to timestamps, and a sliding time window method is used to form a historical load sequence from the monitoring data of several consecutive time points.

8. A reinforcement learning-based edge computing resource allocation optimization system, based on the reinforcement learning-based edge computing resource allocation optimization method according to any one of claims 1 to 7, characterized in that: Also includes, The collection module is used to collect computing load data and network status data of edge devices, and record CPU utilization, memory occupancy, and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence; Establishing a module, constructing a state space matrix according to the historical load sequence, mapping the historical load sequence into discrete state values, and establishing a deep neural network model; A generation module, used to input the state space matrix into the deep neural network model, and generate a resource allocation strategy set according to the priority and latency requirements of the computing task; A selection module is used to iteratively train the resource allocation strategy set using a double Q learning algorithm, set a reward function based on task completion time and resource utilization, and select an optimal resource allocation strategy; The adjustment and update module adjusts the computing resource allocation of the edge nodes based on the optimal resource allocation strategy, and feeds back the execution results to the deep neural network model for online learning and updating.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the edge computing resource allocation optimization method based on reinforcement learning are implemented in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the edge computing resource allocation optimization method based on reinforcement learning according to any one of claims 1 to 7 are implemented.

Images