A reinforcement learning-based edge computing resource allocation optimization method and system

Abstract

The present invention discloses a method and system for optimizing edge computing resource allocation based on reinforcement learning, which relates to the field of edge computing resource management technology. The method comprises collecting computing load data and network status data, forming a historical load sequence, constructing a state space matrix, and mapping the historical load sequence into discrete state values to establish a deep neural network model; inputting the state space matrix into the deep neural network model, generating a resource allocation strategy set according to the priority and delay requirements of the computing task; iteratively training the resource allocation strategy set using a double Q learning algorithm, setting a reward function based on task completion time and resource utilization, and selecting the optimal resource allocation strategy; adjusting the computing resource allocation of edge nodes according to the optimal resource allocation strategy, and feeding back the execution results to the deep neural network model for online learning and updating. The present invention realizes intelligent optimization of edge computing resource allocation by introducing a deep reinforcement learning method.

Description

Technical Field

[0001] The present invention relates to the field of edge computing resource management technology, and in particular to a reinforcement learning-based edge computing resource allocation optimization method and system. Background Art

[0002] Edge computing reduces data transmission latency and improves system response speed and computing efficiency by moving data processing and storage capabilities to the edge of the network. However, traditional edge computing resource allocation methods often use static strategies or simple load balancing mechanisms, which are difficult to dynamically adapt to complex and changing computing environments and business needs. This leads to problems such as low computing resource utilization and large fluctuations in task processing latency.

[0003] Existing edge computing resource allocation technologies have the following key limitations: First, traditional methods lack the ability to accurately perceive and dynamically model the computing load and network status of edge nodes, and resource allocation strategies are often too rigid; second, when dealing with resource allocation for heterogeneous edge devices, existing technologies have difficulty balancing task priorities, latency requirements, and resource utilization, often resulting in unbalanced resource allocation or computing performance bottlenecks; third, there is a lack of effective adaptive learning mechanisms, and resource allocation strategies cannot be adjusted in a timely manner according to real-time load changes, resulting in difficulty in continuously optimizing the overall system performance.

[0004] In view of the above-mentioned shortcomings of existing edge computing resource allocation technology, the present invention proposes an edge computing resource allocation optimization method based on reinforcement learning, which solves the key technical problems of dynamic resource allocation and continuous performance optimization in edge computing environments. By introducing deep neural networks and dual Q-learning algorithms, the present invention aims to achieve intelligent, efficient and adaptive allocation of edge computing resources. Summary of the Invention

[0005] In view of the problems existing in the existing edge computing resource allocation technology, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to achieve intelligent, dynamic and efficient allocation of edge computing resources, accurately perceive and predict the computing load of edge nodes through deep reinforcement learning methods, and adaptively optimize resource allocation strategies according to task priority and latency requirements, thereby improving the overall performance and resource utilization of the edge computing system.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In the first aspect, an embodiment of the present invention provides an edge computing resource allocation optimization method based on reinforcement learning, which includes collecting computing load data and network status data of edge devices, recording CPU utilization, memory occupancy and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence; constructing a state space matrix according to the historical load sequence, and mapping the historical load sequence into discrete state values to establish a deep neural network model; inputting the state space matrix into the deep neural network model, and generating a resource allocation strategy set according to the priority and delay requirements of the computing task; using a double Q learning algorithm to iteratively train the resource allocation strategy set, setting a reward function based on task completion time and resource utilization, and selecting the optimal resource allocation strategy; based on the optimal resource allocation strategy, adjusting the computing resource allocation of the edge node, and at the same time feeding back the execution results to the deep neural network model for online learning and updating.

[0009] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, wherein: based on the optimal resource allocation strategy, the computing resource allocation of the edge node is adjusted, and the execution result is fed back to the deep neural network model for online learning and updating, including: converting the resource allocation ratio in the optimal resource allocation strategy into resource configuration parameters, and issuing the resource configuration parameters through the resource management interface of the edge node, wherein the resource configuration parameters include CPU core quantity allocation, memory space allocation and network bandwidth allocation; adjusting the optimal resource allocation strategy in the order of task priority, and using a smooth transition strategy to switch resources for the optimal resource allocation strategy; based on the effect of resource switching, The current system status is monitored and recorded in real time, and the recorded system status is compared with the expected target to calculate the performance difference value and resource utilization deviation, where the system status includes the actual resource occupancy of each task, task execution progress and performance indicators; a sliding window mechanism is used to establish a learning sample set of the system status, and the deep neural network model is incrementally updated through small-batch gradient descent, where the learning sample set includes the initial state, resource allocation strategy and execution effect; the performance difference value and resource utilization deviation are statistically analyzed, and when the resource utilization deviation is greater than a second preset threshold and the performance difference value is greater than a third preset threshold, the structural adaptive adjustment mechanism of the deep neural network model is triggered to optimize the resource allocation strategy.

[0010] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, the method for obtaining the optimal resource allocation strategy is to construct a dual Q learning algorithm with the same network structure, wherein the dual Q learning algorithm includes a main Q network and a target Q network, and the network structure includes an input layer, a hidden layer, and an output layer; set a reward function according to the task completion time and resource utilization, and balance the task processing efficiency and resource utilization efficiency through this reward function, wherein the reward function includes a task completion time reward and a resource utilization reward; use the ε-greedy method to select an execution action from the resource allocation strategy set, and execute the selected resource allocation strategy to obtain an actual reward and a next state; store the resource state, the execution of the selected resource allocation strategy, the actual reward, and the next state in an experience replay pool D, and randomly sample B experience samples from the experience replay pool D for training; for the experience samples, calculate the Q value of the current state through the main Q network, and calculate the target Q value of the next state through the target Q network, and update the main Q network parameters according to the temporal difference error; repeat the above operation steps until the training converges, and select the resource allocation strategy with the largest Q value in the current state as the optimal resource allocation strategy.

[0011] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, the specific formula of the reward function R is as follows:

[0012]

[0013] Among them, ω i is the weight coefficient of the i-th task, P i is the performance index of the i-th task, Λ ∈ (T c ) is the normalization factor of the task completion time T, ε is the control normalization degree, U r is the actual resource utilization, U t is the target resource utilization, σ is the smoothing parameter of resource utilization deviation, and n is the number of tasks.

[0014] When the performance index of the task P i is greater than the first preset threshold, and the actual resource utilization U r Equal to the target resource utilization U t When , the reward function R will give positive rewards; when the task completion time T is greater than 50% of the expected task completion time and the resource utilization fluctuation U is greater than 15% of the expected resource utilization fluctuation, the reward function R will give negative rewards and punish inefficient resource allocation strategies; when the task priority weight ω iIf a task is not completed on time, the reward function R will give a negative penalty; if the system has several parallel tasks and there is resource contention, the reward function R will dynamically adjust the reward value according to the importance of the task and the urgency of execution.

[0015] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, the method for generating the resource allocation strategy set is as follows: establishing a feature vector for the computing task according to the historical load sequence, and calculating the task priority using a priority scoring function, wherein the feature vector includes task urgency, task importance, and task resource requirement; sorting the computing tasks according to the priority score to generate a task priority queue, and converting the delay requirement of the computing task into a resource allocation constraint, wherein the constraint includes a minimum resource guarantee constraint and a maximum resource upper limit constraint; based on the task priority queue and the constraint, inputting the state space matrix into the trained deep neural network model to obtain the evaluation value of each action in the current state, and using the Softmax function to convert the evaluation value into an action probability distribution; generating a resource allocation strategy set based on the action probability distribution sampling, and calculating the evaluation index of each resource allocation strategy, wherein each resource allocation strategy includes the computing resource allocation ratio of the edge node; the evaluation index includes a resource utilization efficiency index and a task completion time index; mechanically standardizing the evaluation index, and obtaining a comprehensive score of the resource allocation strategy by linear weighting, retaining the k resource allocation strategy sets with the highest scores {A1, A2, ..., A k1}.

[0016] As an optimal solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, wherein: a state space matrix is constructed according to the historical load sequence, and the historical load sequence is mapped into discrete state values, and a deep neural network model is established, including: the CPU utilization, memory occupancy and bandwidth usage in the historical load sequence are divided into several discrete state intervals through the K-means clustering algorithm, and the discrete state intervals are assigned corresponding numerical identifiers to form discretized feature vectors; the discretized feature vectors are organized into an M×N state space matrix according to the time dimension, and a time sliding window mechanism is introduced to capture state transition features, where M represents the number of monitoring indicators and N represents the length of the time series; based on the state transition features, a multi-layer perceptron structure is designed, and a ReLU activation function is used in the hidden layer to construct a deep neural network model; the Adam optimizer is used, and the parameters of the deep neural network model are adjusted and optimized in combination with the learning rate decay strategy, and the difference between the predicted results and the actual state is measured by the cross entropy loss function to complete the training of the deep neural network model.

[0017] As a preferred solution of the edge computing resource allocation optimization method based on reinforcement learning described in the present invention, the method for forming the historical load sequence is as follows: based on the resource monitoring module of the edge node, the system status of the edge device is sampled; the CPU utilization is obtained by reading the processor usage percentage, wherein the CPU utilization includes idle time, user time and system time; the memory occupancy rate is obtained by counting the ratio of the used space of the physical memory to the total space, and the memory page replacement frequency is recorded at the same time; the bandwidth usage is obtained by monitoring the data packet sending and receiving rate of the network interface, wherein the bandwidth usage includes the real-time value of the uplink bandwidth and the real-time value of the downlink bandwidth; the CPU utilization rate, the memory occupancy rate and the bandwidth usage are marked and stored according to the timestamp, and the monitoring data of several consecutive time points are formed into a historical load sequence using a sliding time window method.

[0018] In the second aspect, an embodiment of the present invention provides an edge computing resource allocation optimization system based on reinforcement learning, which includes: an acquisition module for collecting computing load data and network status data of edge devices, and recording CPU utilization, memory occupancy and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence; an establishment module for constructing a state space matrix according to the historical load sequence, and mapping the historical load sequence into discrete state values to establish a deep neural network model; a generation module for inputting the state space matrix into the deep neural network model, and generating a resource allocation strategy set according to the priority and delay requirements of the computing task; a selection module for iteratively training the resource allocation strategy set using a double Q learning algorithm, setting a reward function based on task completion time and resource utilization, and selecting the optimal resource allocation strategy; an adjustment and update module for adjusting the computing resource allocation of the edge node based on the optimal resource allocation strategy, and at the same time feeding back the execution results to the deep neural network model for online learning and updating.

[0019] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the edge computing resource allocation optimization method based on reinforcement learning as described in the first aspect of the present invention are implemented.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, wherein: when the computer program instructions are executed by a processor, the steps of the edge computing resource allocation optimization method based on reinforcement learning as described in the first aspect of the present invention are implemented.

[0021] The beneficial effects of the present invention are as follows: the present invention realizes dynamic and accurate perception of edge computing resources through refined resource monitoring and state space matrix construction; based on deep neural networks and dual Q learning algorithms, it can adaptively generate optimal resource allocation strategies according to task priorities and latency requirements; establishes a closed-loop system of resource allocation-execution-feedback-learning, so that edge computing resource allocation has continuous learning and self-optimization capabilities, effectively improves the system's resource utilization and computing efficiency, and solves the key technical problems of unbalanced resource allocation and poor adaptability of traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0023] Figure 1 This is a flowchart of the edge computing resource allocation optimization method based on reinforcement learning in Example 1.

[0024] Figure 2 This is a system block diagram of the edge computing resource allocation optimization method based on reinforcement learning in Example 1. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0027] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0028] Example 1

[0029] Reference Figures 1 and 2 , which is the first embodiment of the present invention, provides an edge computing resource allocation optimization method based on reinforcement learning, including:

[0030] S1: Collect computing load data and network status data of edge devices, and record CPU utilization, memory occupancy, and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence.

[0031] Specifically, the method for forming the historical load sequence is to sample the system status of the edge device based on the resource monitoring module of the edge node; obtain the CPU utilization by reading the percentage of processor usage; obtain the memory occupancy rate by counting the ratio of the used space to the total space of the physical memory, and record the memory page replacement frequency; obtain the bandwidth usage by monitoring the data packet sending and receiving rate of the network interface

[0032] It should be noted that CPU utilization includes idle time, user time, and system time; bandwidth usage includes the real-time values of uplink bandwidth and downlink bandwidth.

[0033] Furthermore, the CPU utilization, memory occupancy, and bandwidth usage are marked and stored according to timestamps, and a sliding time window method is used to form a historical load sequence from the monitoring data of several consecutive time points.

[0034] S2: Construct a state space matrix according to the historical load sequence, map the historical load sequence into discrete state values, and establish a deep neural network model.

[0035] Specifically, the CPU utilization, memory occupancy, and bandwidth usage in the historical load sequence are divided into several discrete state intervals using the K-means clustering algorithm, and corresponding numerical labels are assigned to the discrete state intervals to form a discretized feature vector.

[0036] It should be noted that the CPU utilization is categorized as follows: when the CPU utilization is between 0-30%, it is considered a low load state; when the CPU utilization is between 30%-70%, it is considered a medium load state; and when the CPU utilization is between 70%-100%, it is considered a high load state. Memory usage is categorized as follows: when the memory utilization is between 0-20%, it is considered an idle state; when the memory utilization is between 20%-60%, it is considered a normal state; when the memory utilization is between 60%-80%, it is considered a stressed state; and when the memory utilization is between 80%-100%, it is considered a saturated state. Bandwidth usage is categorized as follows: when the bandwidth utilization is between 0-40%, it is considered a low speed state; when the bandwidth utilization is between 40%-80%, it is considered a medium speed state; and when the bandwidth utilization is between 80%-100%, it is considered a high speed state.

[0037] Furthermore, the discretized feature vectors are organized into an M×N state space matrix according to the time dimension, and a time sliding window mechanism is introduced to capture the state transition features, where M represents the number of monitoring indicators and N represents the length of the time series. Based on the state transition features, a multi-layer perceptron structure is designed, and the ReLU activation function is used in the hidden layer to construct a deep neural network model.

[0038] Furthermore, the Adam optimizer is used in combination with the learning rate decay strategy to adjust and optimize the parameters of the deep neural network model. At the same time, the cross entropy loss function is used to measure the difference between the predicted results and the actual state to complete the training of the deep neural network model.

[0039] S3: Input the state space matrix into the deep neural network model, and generate a resource allocation strategy set based on the priority and latency requirements of the computing task.

[0040] Specifically, the method for generating the resource allocation strategy set is to establish a feature vector for the computing task based on the historical load sequence and calculate the task priority using a priority scoring function, where the feature vector includes task urgency, task importance, and task resource demand.

[0041] It should be noted that when the task urgency in the feature vector exceeds 0.8, the task importance is greater than 0.7, and the resource demand is less than 20% of the system's available resources, it is an urgent task; if it is an urgent task, the task delay requirement is a strict delay-tolerance of 5ms; when the task urgency in the feature vector is between 0.4-0.8, the task importance is in the range of 0.4-0.7, and the resource demand accounts for 20%-50% of the system's available resources, it is an ordinary task; if it is an ordinary task, the task delay requirement is an ordinary delay-tolerance of 20ms; when the task urgency in the feature vector is less than 0.4, the task importance is less than 0.4, and the resource demand exceeds 50% of the system's available resources, it is a low-level task; if it is a low-level task, the task delay requirement is a loose delay-tolerance of 50ms.

[0042] Furthermore, computing tasks are sorted according to their priority scores to generate a task priority queue, and the latency requirements of the computing tasks are converted into resource allocation constraints, where the constraints include minimum resource guarantee constraints and maximum resource upper limit constraints. Based on the task priority queue and constraints, the state space matrix is input into the trained deep neural network model to obtain the evaluation values of each action in the current state, and the Softmax function is used to convert the evaluation values into action probability distribution.

[0043] Furthermore, the evaluation value is composed of three key dimensions; the first dimension is the task priority score, which is adjusted by the weight coefficient to adjust its influence on the overall evaluation value; the second dimension is the resource utilization rate, which is also adjusted by the weight coefficient to reflect the influence of the current resource status of the system on the action selection; the third dimension is the resource allocation constraint, which is mainly reflected by the ratio of the current resource allocation to the maximum resource upper limit, ensuring that the resource allocation does not exceed the system carrying capacity and smoothing the impact of the excess part; these three dimensions are combined through weighted summation to generate the final evaluation value.

[0044] It should be noted that the Softmax function is a normalized exponential function that converts a set of raw evaluation values into a probability distribution. Its core function is to map any real value to the interval (0, 1) and ensure that the sum of all probability values is always 1. The specific conversion process is to take the exponential of each evaluation value and then divide it by the sum of all the evaluation value exponentials to obtain the probability distribution of each action.

[0045] Specifically, a set of resource allocation strategies is generated based on action probability distribution sampling, and the evaluation index of each resource allocation strategy is calculated.

[0046] It should be noted that each resource allocation strategy includes the computing resource allocation ratio of edge nodes; the evaluation indicators include resource utilization efficiency indicators and task completion time indicators.

[0047] Furthermore, the evaluation indicators are mechanically standardized, and the comprehensive scores of resource allocation strategies are obtained through linear weighting, and the k resource allocation strategy sets with the highest scores {A1, A2, ..., A k1}.

[0048] S4: Using a double Q learning algorithm to iteratively train the resource allocation strategy set, setting a reward function based on task completion time and resource utilization, and selecting the optimal resource allocation strategy.

[0049] Specifically, the method for obtaining the optimal resource allocation strategy is to construct a double Q-learning algorithm with the same network structure; set a reward function based on task completion time and resource utilization, and balance task processing efficiency and resource utilization efficiency through this reward function.

[0050] It should be noted that the dual Q-learning algorithm includes a main Q-network and a target Q-network; the network structure includes an input layer, a hidden layer, and an output layer; the input layer receives the state space matrix, and the output layer generates a resource allocation strategy set; the reward function includes a task completion time reward and a resource utilization reward.

[0051] Furthermore, the specific formula of the reward function R is as follows:

[0052]

[0053] Among them, ω i is the weight coefficient of the i-th task, P i is the performance index of the i-th task, Λ ∈ (T c ) is the normalization factor of the task completion time T, ε is the control normalization degree, U r is the actual resource utilization, U t is the target resource utilization, σ is the smoothing parameter of resource utilization deviation, and n is the number of tasks.

[0054] Furthermore, when the performance index P i is greater than the first preset threshold, and the actual resource utilization U r Equal to the target resource utilization U t When , the reward function R will give positive rewards; when the task completion time T is greater than 50% of the expected task completion time and the resource utilization fluctuation U is greater than 15% of the expected resource utilization fluctuation, the reward function R will give negative rewards and punish inefficient resource allocation strategies; when the task priority weight ω i If a task is not completed on time, the reward function R will give a negative penalty; if the system has several parallel tasks and there is resource contention, the reward function R will dynamically adjust the reward value according to the importance of the task and the urgency of execution.

[0055] Specifically, the ε-greedy method is used to select an execution action from the resource allocation strategy set, and the selected resource allocation strategy is executed to obtain the actual reward and the next state; the resource state, the execution of the selected resource allocation strategy, the actual reward, and the next state are stored in the experience replay pool D, and B experience samples are randomly sampled from the experience replay pool D for training.

[0056] For the empirical samples, the Q value of the current state is calculated through the main Q network, and the target Q value of the next state is calculated through the target Q network. The main Q network parameters are updated according to the timing difference error. After every C training steps, the main Q network parameters are copied to the target Q network to maintain the delay of the target network parameter update. The above steps are repeated until the training converges, and the resource allocation strategy with the largest Q value of the current state is selected as the optimal resource allocation strategy.

[0057] S5: Based on the optimal resource allocation strategy, adjust the computing resource allocation of the edge nodes, and feed back the execution results to the deep neural network model for online learning and updating.

[0058] Specifically, the resource allocation ratio in the optimal resource allocation strategy is converted into resource configuration parameters, and these resource configuration parameters are issued through the resource management interface of the edge node; the optimal resource allocation strategy is adjusted according to the order of task priority, and a smooth transition strategy is used to switch resources for the optimal resource allocation strategy; based on the effect of resource switching, the current system status is monitored and recorded in real time, and the recorded system status is compared with the expected target, and the performance difference value and resource utilization deviation are calculated.

[0059] It should be noted that resource configuration parameters include CPU core number allocation, memory space allocation, and network bandwidth allocation; system status includes the actual resource usage of each task, task execution progress, and performance indicators.

[0060] Furthermore, the specific formula for performance difference is as follows:

[0061]

[0062] Among them, δ performance is the performance difference index, E i (t) is the actual execution efficiency of the i-th task, E ideal (t) is the target execution efficiency, ω is the weight coefficient, and n is the number of tasks.

[0063] Furthermore, the specific formula for resource utilization deviation is as follows:

[0064]

[0065] Among them, Ω utilization is the resource utilization deviation index, ρ j is the actual resource utilization of the jth resource dimension, ρ j,target is the target resource utilization, and m is the number of resource dimensions.

[0066] Specifically, a sliding window mechanism is used to establish a learning sample set of the system state, and the deep neural network model is incrementally updated through small-batch gradient descent, where the learning sample set includes the initial state, resource allocation strategy and execution effect; the performance difference value and resource utilization deviation are counted, and when the resource utilization deviation is greater than the second preset threshold and the performance difference value is greater than the third preset threshold, the structural adaptive adjustment mechanism of the deep neural network model is triggered to optimize the resource allocation strategy.

[0067] It should be noted that the second preset threshold is based on the statistical fluctuation range of the resource utilization of edge nodes and the fluctuation range of the historical load series. By conducting long-term monitoring of the CPU utilization, memory occupancy and network bandwidth usage of multiple edge nodes, the standard deviation and peak deviation of their fluctuations are calculated to determine a reasonable upper limit of resource utilization fluctuations; the third preset threshold is based on the discrete degree of performance indicators of different computing tasks and the correlation between tasks, and comprehensively considers multi-dimensional factors such as task execution delay, resource consumption differences, priority differences, etc. Through cluster analysis and performance dispersion evaluation of historical task execution data, a threshold that reflects the overall performance fluctuation of the system is set.

[0068] Further, such as Figure 2 As shown, this embodiment also provides an edge computing resource allocation optimization system based on reinforcement learning, including: an acquisition module, which is used to collect computing load data and network status data of edge devices, and record CPU utilization, memory occupancy and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence; an establishment module, which constructs a state space matrix according to the historical load sequence, and maps the historical load sequence into discrete state values to establish a deep neural network model; a generation module, which is used to input the state space matrix into the deep neural network model, and generate a resource allocation strategy set according to the priority and delay requirements of the computing task; a selection module, which is used to iteratively train the resource allocation strategy set using a double Q learning algorithm, set a reward function based on task completion time and resource utilization, and select the optimal resource allocation strategy; an adjustment and update module, which adjusts the computing resource allocation of the edge node based on the optimal resource allocation strategy, and at the same time feeds back the execution results to the deep neural network model for online learning and updating.

[0069] This embodiment also provides a computer device, which is suitable for the edge computing resource allocation optimization method based on reinforcement learning, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the edge computing resource allocation optimization method based on reinforcement learning proposed in the above embodiment.

[0070] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0071] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps: collecting computing load data and network status data of edge devices, recording CPU utilization, memory occupancy, and bandwidth usage through a resource monitoring module of the edge node to form a historical load sequence; constructing a state space matrix based on the historical load sequence, and mapping the historical load sequence into discrete state values to establish a deep neural network model; inputting the state space matrix into the deep neural network model, and generating a resource allocation strategy set based on the priority and latency requirements of the computing task; iteratively training the resource allocation strategy set using a double Q learning algorithm, setting a reward function based on task completion time and resource utilization, and selecting an optimal resource allocation strategy; adjusting the computing resource allocation of the edge node based on the optimal resource allocation strategy, and feeding back the execution results to the deep neural network model for online learning and updating.

[0072] In summary, the present invention achieves dynamic and accurate perception of edge computing resources through refined resource monitoring and state space matrix construction; based on deep neural networks and dual Q learning algorithms, it can adaptively generate optimal resource allocation strategies according to task priorities and latency requirements; establishes a closed-loop system of resource allocation-execution-feedback-learning, so that edge computing resource allocation has continuous learning and self-optimization capabilities, effectively improving the system's resource utilization and computing efficiency, and solving the key technical problems of unbalanced resource allocation and poor adaptability of traditional methods.

[0073] Example 2

[0074] Referring to Table 1, which is the second embodiment of the present invention, this embodiment provides an edge computing resource allocation optimization method based on reinforcement learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0075] Specifically, 10 edge computing nodes were selected as test objects. Each edge node was equipped with an Intel i7-9700 processor (8 cores and 16 threads), 32GB of DDR4 memory, and 1Gbps of network bandwidth. To comprehensively evaluate system performance, three typical application scenarios were designed: video stream analysis, sensor data processing, and mobile device task offloading. The experiment first deployed a resource monitoring module based on Prometheus, set the sampling interval to 100ms, and established a 15-minute sliding time window. The node_exporter function collected the CPU time percentages of various types, including idle time, user time, and system time. Furthermore, the memory_exporter function was used to record memory usage, including physical memory usage and page fault rate. The network_exporter function was used to monitor network status, recording the real-time usage of both upstream and downstream bandwidth.

[0076] Furthermore, during the data preprocessing phase, an improved K-means clustering algorithm was used, with the Silhouette Coefficient used to verify the optimal clustering results. After multiple rounds of testing, the optimal partitioning intervals for CPU utilization, memory usage, and bandwidth usage were determined. To improve feature extraction accuracy, a data preprocessing method based on exponential smoothing was introduced, effectively reducing the impact of data noise on clustering results.

[0077] Furthermore, the deep neural network model adopts a four-layer structure, consisting of one input layer (the input dimension is M×N, where M=3 represents the number of monitoring indicators and N=900 represents the number of sampling points within 15 minutes), two hidden layers (with 256 and 128 neurons, respectively), and one output layer. The hidden layers use LeakyReLU as the activation function to address the problem of neuron death that may be caused by traditional ReLU. The initial learning rate is set to 0.001 and is dynamically adjusted using a cosine annealing strategy. During training, an improved Adam optimizer is used, and a warmup mechanism is introduced to gradually increase the learning rate at the beginning of training, effectively improving the convergence stability of the model. To prevent overfitting, a dropout layer is added between the two hidden layers, with a dropout rate set to 0.3. At the same time, batch normalization technology is used to accelerate the training process, with a batch size of 64.

[0078] Specifically, as shown in Table 1, from a resource utilization perspective, the average CPU utilization of the eight test nodes ranged from 28.3% to 75.2%, demonstrating that the system dynamically adjusts resource allocation based on actual load conditions, avoiding over- or under-allocation. In particular, despite a low CPU utilization of 28.3% for the Edge-03 node, its task completion rate reached 99.2%, with an average response time of only 8.9ms. This demonstrates that the system successfully identified light-load scenarios and appropriately reduced resource allocation, resulting in high resource utilization efficiency (92.1%). Regarding memory management, this solution maintains memory utilization within a reasonable range through precise load prediction and dynamic adjustment strategies. Data shows that even under high load conditions (e.g., 82.4% memory utilization for the Edge-04 node), the system maintains a 95.6% task completion rate, thanks to the precise partitioning of memory states by the improved K-means clustering algorithm and the accurate prediction capabilities of the deep neural network model. Bandwidth resource utilization also demonstrated good adaptability, with each node's bandwidth utilization showing a positive correlation with service load, though none reached saturation. For example, the Edge-07 node maintained a 96.4% task completion rate and 85.3% resource utilization efficiency despite a relatively high bandwidth utilization rate (58.9%), validating the system's effectiveness in optimizing network resource scheduling.

[0079] Table 1 Edge computing resource allocation performance evaluation data

[0080] Node ID Average CPU utilization Memory usage Bandwidth usage Task completion rate Average response time Resource utilization efficiency Edge-01 45.6％ 58.3％ 35.2％ 98.5％ 12.3ms 89.4％ Edge-02 62.8％ 72.1％ 45.6％ 97.8％ 15.7ms 86.2％ Edge-03 28.3％ 45.6％ 28.9％ 99.2％ 8.9ms 92.1％ Edge-04 75.2％ 82.4％ 68.7％ 95.6％ 18.4ms 83.5％ Edge-05 52.1％ 63.5％ 42.3％ 98.1％ 13.8ms 88.7％ Edge-06 38.7％ 51.2％ 32.8％ 98.9％ 10.5ms 90.8％ Edge-07 68.4％ 77.8％ 58.9％ 96.4％ 16.9ms 85.3％ Edge-08 42.9％ 56.7％ 38.4％ 98.7％ 11.8ms 89.9％

[0081] Furthermore, performance metrics showed that the average response time of all tested nodes remained below 20ms, with the optimal response time being 8.9ms for Edge-03 and 18.4ms for Edge-04. This significantly outperformed traditional fixed allocation schemes (typically with response times between 30-50ms). Furthermore, task completion rates generally remained above 95%, fully demonstrating the effectiveness of the dual Q-learning algorithm in optimizing resource allocation strategies.

[0082] Furthermore, the resource utilization efficiency index across the eight nodes reached an average of 88.2%, with a peak of 92.1% and a minimum of 83.5%, representing a 20-30% improvement over the traditional solution. This improvement is primarily due to the adaptive learning mechanism and precise state-space modeling method introduced in this solution.

[0083] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A reinforcement learning-based edge computing resource allocation optimization method, characterized by: include, Collect computing load data and network status data of edge devices, and record CPU utilization, memory occupancy, and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence; Constructing a state space matrix based on the historical load sequence, mapping the historical load sequence into discrete state values, and establishing a deep neural network model; Inputting the state space matrix into the deep neural network model, and generating a resource allocation strategy set based on the priority and latency requirements of the computing task; The resource allocation strategy set is iteratively trained using a double Q-learning algorithm, a reward function is set based on task completion time and resource utilization, and the optimal resource allocation strategy is selected; Based on the optimal resource allocation strategy, the computing resource allocation of the edge nodes is adjusted, and the execution results are fed back to the deep neural network model for online learning and updating; Build a deep neural network model, including: The CPU utilization, memory occupancy, and bandwidth usage in the historical load sequence are divided into several discrete state intervals using the K-means clustering algorithm, and corresponding numerical identifiers are assigned to the discrete state intervals to form a discretized feature vector; The discretized feature vectors are organized into an M×N state space matrix according to the time dimension, and a time sliding window mechanism is introduced to capture state transition features, where M represents the number of monitoring indicators and N represents the length of the time series. Based on the state transition characteristics, a multi-layer perceptron structure is designed, and the ReLU activation function is used in the hidden layer to construct a deep neural network model; The Adam optimizer is used in combination with the learning rate decay strategy to adjust and optimize the parameters of the deep neural network model. At the same time, the cross entropy loss function is used to measure the difference between the predicted result and the actual state to complete the training of the deep neural network model.

2. The edge computing resource allocation optimization method based on reinforcement learning according to claim 1, characterized in that: Based on the optimal resource allocation strategy, the computing resource allocation of the edge nodes is adjusted, and the execution results are fed back to the deep neural network model for online learning and updating, including: Convert the resource allocation ratio in the optimal resource allocation strategy into resource configuration parameters, and issue these resource configuration parameters through the resource management interface of the edge node, where the resource configuration parameters include CPU core number allocation, memory space allocation, and network bandwidth allocation; Adjusting the optimal resource allocation strategy according to the order of task priority, and using a smooth transition strategy to switch resources to the optimal resource allocation strategy; Based on the effect of resource switching, the current system status is monitored and recorded in real time. The recorded system status is compared with the expected target to calculate the performance difference value and resource utilization deviation. The system status includes the actual resource usage of each task, task execution progress and performance indicators. A sliding window mechanism is used to establish a learning sample set of system states, and the deep neural network model is incrementally updated through mini-batch gradient descent. The learning sample set includes the initial state, resource allocation strategy, and execution effect. The performance difference value and resource utilization deviation are counted. When the resource utilization deviation is greater than the second preset threshold and the performance difference value is greater than the third preset threshold, the structural adaptive adjustment mechanism of the deep neural network model is triggered to optimize the resource allocation strategy.

3. The edge computing resource allocation optimization method based on reinforcement learning according to claim 2, characterized in that: The method for obtaining the optimal resource allocation strategy is: Constructing a dual Q learning algorithm with the same network structure, wherein the dual Q learning algorithm includes a main Q network and a target Q network, and the network structure includes an input layer, a hidden layer, and an output layer; Setting a reward function based on task completion time and resource utilization, and balancing task processing efficiency and resource utilization efficiency through this reward function, wherein the reward function includes a task completion time reward and a resource utilization reward; An ε-greedy method is used to select an execution action from the resource allocation strategy set, and the selected resource allocation strategy is executed to obtain an actual reward and a next state; The resource state, the selected resource allocation strategy, the actual reward, and the next state are stored in the experience replay pool D, and B experience samples are randomly sampled from the experience replay pool D for training; For the empirical sample, the Q value of the current state is calculated through the main Q network, and the target Q value of the next state is calculated through the target Q network, and the main Q network parameters are updated according to the timing difference error; Repeat the above steps until the training converges, and select the resource allocation strategy with the largest Q value in the current state as the optimal resource allocation strategy.

4. The edge computing resource allocation optimization method based on reinforcement learning according to claim 3, characterized in that: The reward function The specific formula is as follows: ； in, is the weight coefficient of the i-th task, is the performance index of the i-th task, Task completion time The normalization factor of To control the degree of normalization, is the actual resource utilization rate, is the target resource utilization, is the smoothing parameter of resource utilization deviation, n is the number of tasks; When the performance index of the task is greater than the first preset threshold, and the actual resource utilization Equal to target resource utilization When , the reward function R will give a positive reward; When the task is completed More than 50% of the expected task completion time and fluctuating resource utilization When the fluctuation is greater than 15% of the expected resource utilization, the reward function R will give a negative reward and punish inefficient resource allocation strategies; When the task priority weight If the task is not completed on time, the reward function R will give a negative penalty; If the system has several parallel tasks and there is resource contention, the reward function R will dynamically adjust the reward value according to the importance and execution urgency of the task.

5. The edge computing resource allocation optimization method based on reinforcement learning according to claim 4, characterized in that: The method for generating the resource allocation strategy set is: Establishing a feature vector for the computing task according to the historical load sequence and calculating the task priority using a priority scoring function, wherein the feature vector includes task urgency, task importance and task resource requirement; Sort the computing tasks according to the priority scores, generate a task priority queue, and convert the latency requirements of the computing tasks into resource allocation constraints, wherein the constraints include a minimum resource guarantee constraint and a maximum resource upper limit constraint; Based on the task priority queue and the constraints, the state space matrix is input into the trained deep neural network model to obtain the evaluation value of each action in the current state, and the evaluation value is converted into an action probability distribution using a Softmax function; Generate a resource allocation strategy set based on the action probability distribution sampling, and calculate the evaluation index of each resource allocation strategy, wherein each resource allocation strategy includes the computing resource allocation ratio of the edge node; the evaluation index includes a resource utilization efficiency index and a task completion time index; The evaluation indicators are mechanically standardized and the comprehensive scores of resource allocation strategies are obtained by linear weighting, and the k resource allocation strategy sets with the highest scores are retained. .

6. The edge computing resource allocation optimization method based on reinforcement learning according to claim 5, characterized in that: The method for forming the historical load sequence is: The resource monitoring module based on edge nodes samples the system status of edge devices; Obtaining CPU utilization by reading a percentage of processor usage, wherein the CPU utilization includes idle time, user time, and system time; The memory usage is obtained by counting the ratio of used physical memory space to total space, and the memory page replacement frequency is recorded at the same time; Obtaining bandwidth usage by monitoring the data packet sending and receiving rate of the network interface, wherein the bandwidth usage includes a real-time value of uplink bandwidth and a real-time value of downlink bandwidth; The CPU utilization, the memory occupancy, and the bandwidth usage are marked and stored according to timestamps, and a sliding time window method is used to form a historical load sequence from the monitoring data of several consecutive time points.

7. A reinforcement learning-based edge computing resource allocation optimization system, based on the reinforcement learning-based edge computing resource allocation optimization method according to any one of claims 1 to 6, characterized in that: include, The collection module is used to collect computing load data and network status data of edge devices, and record CPU utilization, memory occupancy, and bandwidth usage through the resource monitoring module of the edge node to form a historical load sequence; Establishing a module to construct a state space matrix based on the historical load sequence, and mapping the historical load sequence into discrete state values to establish a deep neural network model; A generation module, configured to input the state space matrix into the deep neural network model and generate a resource allocation strategy set based on the priority and latency requirements of the computing task; A selection module is used to iteratively train the resource allocation strategy set using a double Q learning algorithm, set a reward function based on task completion time and resource utilization, and select the optimal resource allocation strategy; An adjustment and update module adjusts the computing resource allocation of edge nodes based on the optimal resource allocation strategy, and feeds back the execution results to the deep neural network model for online learning and updating.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the edge computing resource allocation optimization method based on reinforcement learning are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the edge computing resource allocation optimization method based on reinforcement learning are implemented.

Images