An edge computing server information processing method based on deep reinforcement learning

Abstract

The application is suitable for the field of edge computing, and provides an edge computing server information processing method based on deep reinforcement learning, which comprises the following steps: configuring an independent deep reinforcement learning model for each edge server, and collecting state information of the server and multiple adjacent servers in real time; based on the collected state information, outputting a decision action for obtaining a to-be-processed task from a target adjacent server through the deep reinforcement learning model; requesting the target adjacent server to obtain the to-be-processed task according to the decision action, and adding the obtained task to a local to-be-processed queue; calculating a reward value according to system performance feedback after the decision action is executed, and storing task transfer data into an experience pool. The application greatly reduces the communication overhead required for global state synchronization while improving the overall resource utilization rate and task processing efficiency of the edge server cluster.

Description

Technical Field

[0001] This invention relates to the field of edge computing, specifically to an edge computing server information processing method based on deep reinforcement learning. Background Technology

[0002] In edge computing networks, multiple geographically distributed edge servers collaboratively process computing tasks from terminal devices. Due to factors such as user mobility and uneven spatial and temporal distribution of task arrival, the workload among edge servers often experiences severe imbalances: some servers remain continuously overloaded, with task queuing delays increasing dramatically, while adjacent servers remain idle or lightly loaded for extended periods, resulting in low overall system resource utilization and degraded service quality. How to dynamically adjust the task distribution among servers is a critical problem that urgently needs to be solved in the field of edge computing information processing.

[0003] In existing technologies, load balancing methods mostly adopt a centralized scheduling architecture, where a central controller collects global status information and then actively distributes newly arriving tasks to relatively idle servers. While this model alleviates load unevenness to some extent, it has significant technical drawbacks. The centralized scheduler is prone to becoming a performance bottleneck and a single point of failure; frequent synchronization of global status information leads to significant communication overhead; and in large-scale, highly dynamic edge network environments, the scalability of centralized scheduling is severely limited.

[0004] To overcome the shortcomings of centralized scheduling, the field of distributed computing has developed a work-stealing mechanism, in which idle nodes actively steal tasks from the task queues of busy nodes for execution. However, traditional work-stealing methods rely on fixed heuristic rules (such as randomly selecting victim nodes or always stealing tasks from the head of the queue), failing to perceive the heterogeneity of tasks, differences in network latency between nodes, and dynamic trends in load changes in edge computing environments. Blindly stealing tasks may not only increase unnecessary task migration overhead but also lead to decreased execution efficiency due to stealing tasks that are incompatible with local resource types. Therefore, this paper proposes an edge computing server information processing method based on deep reinforcement learning to address the aforementioned problems. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this invention is to provide an edge computing server information processing method based on deep reinforcement learning, so as to solve the problems existing in the above-mentioned background technology.

[0006] This invention is implemented as follows: an edge computing server information processing method based on deep reinforcement learning, the method comprising the following steps: Each edge server is configured with an independent deep reinforcement learning model, and the status information of the server and multiple neighboring servers is collected in real time. The status information includes the queue length of the local server and its neighbors, the CPU utilization of the local server and its neighbors, and the network communication latency of the local server and its neighbors. Based on the collected state information, a deep reinforcement learning model is used to output decision actions for obtaining the task to be processed from the target's neighboring servers; According to the decision action, request the task to be processed from the target adjacent server, and add the obtained task to the local task queue; The reward value is calculated based on the system performance feedback after the decision action is executed, and the data of task transfer is stored in the experience pool to update the deep reinforcement learning model.

[0007] As a further aspect of the present invention: the step of collecting the status information of the current server and multiple adjacent servers in real time specifically includes: Maintain a binary feedback table locally on this server. This feedback table records the historical success rate of each neighboring server's interaction with this server, and initializes all historical success rates to the same value. The historical success rate is updated by passively listening for task retrieval requests from adjacent servers. When it is necessary to collect status information of neighboring servers, read the historical success rate of each neighboring server from the binary feedback table. The historical success rate of each adjacent server is converted into a multi-dimensional state vector through a preset mapping function. The multi-dimensional state vector has the same attributes as the state information, such as queue length, CPU utilization and network communication latency. The multidimensional state vectors of each adjacent server are merged with the local state information of the server collected by the local monitoring module.

[0008] As a further aspect of the present invention: the step of updating the corresponding historical success rate by passively listening to task acquisition requests from adjacent servers specifically includes: Preset a positive step size and a maximum number of listeners within a unit time window; When this server receives a task retrieval request from a neighboring server, if the cumulative number of times that neighboring server has listened within the current time window has not reached the upper limit, then the historical success rate of that neighboring server is increased by a positive step, and the cumulative number of listening is incremented by one. If the cumulative number of times the adjacent server has listened within the current time window has reached the upper limit, then this listening event will be ignored and its historical success rate will not be updated. At the end of each unit time window, the cumulative listening count of all adjacent servers is reset to zero.

[0009] As a further aspect of the present invention: the step of converting the historical success rate of each adjacent server into a multi-dimensional state vector through a preset mapping function specifically includes: A multi-dimensional baseline vector is preset, and each dimension of the vector corresponds to the default baseline values ​​of queue length, CPU utilization, and network communication latency, respectively. For each neighboring server, its current historical success rate is read, and its complementary value is calculated. The complementary value represents the estimated load level of the neighboring server, and the sum of the complementary value and the current historical success rate is 1. Multiply each dimension of the multidimensional baseline vector by its complementary value to obtain a new multidimensional vector, which is then used as the multidimensional state information of the neighboring server. If this server does not receive any task retrieval request from a neighboring server within a preset long time window, the complementary value of that neighboring server will be set to 0.5, and will remain so until the next task retrieval request from that neighboring server is received.

[0010] As a further aspect of the present invention: the step of outputting decision actions for obtaining the task to be processed from the target neighboring server through a deep reinforcement learning model specifically includes: A confidence coefficient is attached to the status information of each neighboring server. The confidence coefficient is inversely proportional to the time difference between the last time the neighboring server was passively monitored and the current time. Based on a deep reinforcement learning model, the basic Q value of each candidate action is calculated according to the current state information. The candidate action is to select a neighboring server to obtain the task to be processed. The final Q-value is obtained by multiplying the base Q-value of each candidate action by the corresponding confidence coefficient. The candidate action with the largest final Q value is selected as the final output decision action.

[0011] As a further aspect of the present invention: the step of requesting the task to be processed from the target neighboring server and adding the obtained task to the local task queue specifically includes: This server first sends a probe request to the target's neighboring server. The probe request only requests to obtain a minimum unit of data packet and does not carry the complete content. The estimated transmission cost for this acquisition is calculated based on the response time of the probe request and the queue status returned by the target neighboring server. If the estimated transmission cost exceeds the preset threshold, the acquisition task is canceled and a new target neighboring server is selected. If the estimated transmission cost does not exceed the preset threshold, a formal request is made to the target server to obtain the task to be processed, and only one task to be processed is obtained in one request. After adding the task to the local pending queue, a preset short timer is started, so that the server will not send a new task retrieval request to the same target server before the timeout.

[0012] As a further aspect of the present invention: the estimated transmission cost is calculated by multiplying the response time of the probe request by the square of the current queue length of the target adjacent server plus one.

[0013] As a further aspect of the present invention: the step of calculating the reward value based on the system performance feedback after the execution of the decision action specifically includes: Before executing this decision action, record the average waiting time of the local pending queue on this server as the first time value; After executing this decision action, the average waiting time of the local pending queue on this server is remeasured as the second time value; The first component of the reward value is obtained by subtracting the second time value from the first time value. The first component represents the amount of reduction in the average waiting time. Record the queue length of the target's adjacent servers and compare it with the queue length before the current decision action. If the queue length has not increased, generate a preset normal number as the second component of the reward value; otherwise, the second component is zero. The first component is added to the second component to obtain the final reward value.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This invention configures an independent deep reinforcement learning model for each edge server. Each server only needs to collect state information from its own machine and neighboring servers to make intelligent decisions locally, without relying on any centralized scheduler. This completely eliminates the single point of failure and performance bottlenecks inherent in centralized architectures, significantly improving the scalability and robustness of large-scale dynamic edge networks. Simultaneously, it can autonomously output optimal task acquisition decisions based on real-time perception of neighbor load and network status, effectively avoiding the blind migration and resource mismatch phenomena commonly found in traditional static rules. This not only directly reduces the average queuing time for tasks but also enables continuous online optimization of model strategies through a reward-guided experience replay mechanism, allowing the system to dynamically adapt to constantly changing load patterns and network environments. In summary, this invention significantly reduces the communication overhead required for global state synchronization while improving the overall resource utilization and task processing efficiency of edge server clusters, making it particularly suitable for large-scale edge computing deployment scenarios with limited bandwidth or sensitive communication costs. Attached Figure Description

[0015] Figure 1 This is a flowchart of an edge computing server information processing method based on deep reinforcement learning.

[0016] Figure 2 This is a flowchart illustrating the process of collecting state information of the local server and multiple neighboring servers in an edge computing server information processing method based on deep reinforcement learning.

[0017] Figure 3 This is a flowchart illustrating the updating of historical success rates in an edge computing server information processing method based on deep reinforcement learning.

[0018] Figure 4 This is a flowchart illustrating the conversion of historical success rates into multidimensional state vectors in an edge computing server information processing method based on deep reinforcement learning.

[0019] Figure 5 This is a flowchart of the output decision action in an edge computing server information processing method based on deep reinforcement learning.

[0020] Figure 6 This is a flowchart illustrating how an edge computing server information processing method based on deep reinforcement learning requests tasks from a target neighboring server and adds the retrieved tasks to a local processing queue.

[0021] Figure 7 This is a flowchart illustrating the calculation of reward values ​​in an edge computing server information processing method based on deep reinforcement learning. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0024] like Figure 1 As shown, this embodiment of the invention provides an edge computing server information processing method based on deep reinforcement learning, the method comprising the following steps: S100: Configure an independent deep reinforcement learning model for each edge server and collect the status information of the server and multiple neighboring servers in real time. The status information includes the queue length of the local server and its neighbors, the CPU utilization of the local server and its neighbors, and the network communication latency of the local server and its neighbors. S200, based on the collected state information, outputs decision actions for obtaining the task to be processed from the target neighboring server through a deep reinforcement learning model; S300, according to the decision action, request the task to be processed from the target adjacent server, and add the obtained task to the local task queue; S400: Calculate the reward value based on the system performance feedback after the decision action is executed, and store the task transfer data into the experience pool to update the deep reinforcement learning model.

[0025] It should be noted that the data for task transfer refers to a set of experience tuples generated during the execution of this decision action and used for training the deep reinforcement learning model. Specifically, this includes: state information before executing the decision action (i.e., queue lengths, CPU utilization, network communication latency, etc. of the local server and neighboring servers), the decision action itself (identifier of the target neighboring server), the reward value calculated after executing the action, and the state information newly collected at the next moment after executing the action. These four elements are encapsulated into an experience sample in the format of "state-action-reward-next state" and stored in the experience pool, which is then added to the end of the experience pool. When the number of samples stored in the experience pool exceeds the preset capacity limit, the oldest sample is removed according to the first-in, first-out principle to maintain a fixed pool size. During the update process of the deep reinforcement learning model, a number of experience samples are randomly sampled from the experience pool for mini-batch gradient descent training.

[0026] In this embodiment of the invention, each edge server is configured with an independent deep reinforcement learning model. Each server only needs to collect the state information of its own machine and neighboring servers to complete intelligent decision-making locally, without relying on any centralized scheduler. This completely eliminates the single point of failure and performance bottleneck problems inherent in centralized architectures, significantly improving the scalability and robustness of large-scale dynamic edge networks. Simultaneously, based on real-time perception of neighbor load and network status, it can autonomously output the optimal task acquisition decision, effectively avoiding the blind migration and resource mismatch phenomena commonly found in traditional static rules. This not only directly reduces the average queuing time for tasks but also achieves continuous online optimization of model strategies through a reward-guided experience replay mechanism, enabling the system to dynamically adapt to constantly changing load patterns and network environments. In summary, this invention improves the overall resource utilization and task processing efficiency of edge server clusters while significantly reducing the communication overhead required for global state synchronization, making it particularly suitable for large-scale edge computing deployment scenarios with limited bandwidth or sensitive communication costs.

[0027] like Figure 2 As shown, in a preferred embodiment of the present invention, the step of collecting the status information of the local server and multiple adjacent servers in real time specifically includes: S101. Maintain a binary feedback table locally on this server. This feedback table records the historical success rate of each neighboring server's interaction with this server and initializes all historical success rates to the same value. S102, the corresponding historical success rate is updated by passively listening to task acquisition requests from adjacent servers; S103, When it is necessary to collect the status information of adjacent servers, read the historical success rate of each adjacent server in the binary feedback table; S104, the historical success rate of each adjacent server is converted into a multi-dimensional state vector through a preset mapping function. The multi-dimensional state vector has the same attributes as the state information, such as queue length, CPU utilization and network communication latency. S105, merge the multidimensional state vectors of each adjacent server with the local state information of this server collected by the local monitoring module.

[0028] In this embodiment of the invention, during system initialization, since no interaction has occurred yet, the historical success rates of all neighboring servers are set to the same value, for example, 0.5, indicating a neutral estimate of the neighbor's load status. The key feature is that this server does not actively send any status query messages, but instead updates the feedback table through a passive listening mechanism. Whenever a neighboring server initiates a task acquisition request to this server, this server assumes that the neighbor currently has a low load, and thus adjusts the historical success rate corresponding to that neighboring server. This passive listening and updating method requires no additional communication overhead and completely reuses the existing task-stealing interaction process. When the deep reinforcement learning model needs to collect the state information of neighboring servers to make decisions, this server directly reads the current historical success rate of each neighbor in the feedback table and converts it into a multi-dimensional state vector through a preset mapping function. This multi-dimensional state vector corresponds perfectly to the actual queue length, CPU utilization, and network communication latency collected in reality. Therefore, it can be directly merged with the actual state information of this server collected by the local monitoring module to form the complete input state required by the deep reinforcement learning model. This approach completely avoids the communication overhead and latency caused by actively sending state probe requests. It can maintain the assessment of neighbor load by only using the passive listening events naturally generated during the task stealing process. It also expands the one-dimensional success rate into a multi-dimensional state vector through a simple multiplication mapping, which not only ensures low-overhead acquisition of state information, but also makes the input dimension of the deep reinforcement learning model compatible with traditional methods, and significantly reduces the signaling load of the edge cluster.

[0029] like Figure 3 As shown in the preferred embodiment of the present invention, the step of updating the corresponding historical success rate by passively listening to task acquisition requests from adjacent servers specifically includes: S1021, presets a positive step size value and an upper limit on the number of listeners within a unit time window; S1022, when this server listens to a task acquisition request sent by a neighboring server, if the cumulative number of times the neighboring server listens within the current time window has not reached the upper limit, the historical success rate of the neighboring server is increased by a positive step, and the cumulative number of listens is incremented by one. S1023, If the cumulative number of times the adjacent server has listened within the current time window has reached the upper limit, then ignore this listening event and do not update its historical success rate; S1024, at the end of each unit time window, reset the cumulative listening count of all adjacent servers to zero.

[0030] In this embodiment of the invention, to prevent a neighboring server from excessively inflating its historical success rate due to frequent task acquisition requests to the local server within a short period, a positive step size (e.g., increasing by 0.05 for each successful listener) and a maximum number of listeners within a unit time window (e.g., recording a maximum of 3 valid listeners every 10 seconds) are pre-defined. For example, if a neighboring server sends 5 task acquisition requests within 10 seconds, only the first 3 will be used to positively update its historical success rate, while the last 2 will be ignored. This approach avoids excessive deviation in the local server's load assessment of a neighbor due to sudden behavior from a single neighbor (such as repeated theft within a short period), while also preventing malicious nodes from manipulating the local server's feedback table through high-frequency requests. This ensures the stability and fairness of historical success rate statistics, and the implementation is extremely simple, requiring only the maintenance of an integer counter and resetting logic for each neighbor.

[0031] like Figure 4 As shown in the preferred embodiment of the present invention, the step of converting the historical success rate of each adjacent server into a multi-dimensional state vector through a preset mapping function specifically includes: S1041, a multi-dimensional baseline vector is preset, and each dimension of the vector corresponds to the default baseline values ​​of queue length, CPU utilization and network communication latency, respectively. S1042, For each adjacent server, read its current historical success rate and calculate its complementary value, wherein the complementary value represents the estimated load level of the adjacent server, and the sum of the complementary value and the current historical success rate is 1. S1043, Multiply each dimension of the multidimensional reference vector by the complementary value to obtain a new multidimensional vector, and use it as the multidimensional state information of the adjacent server. S1044, when this server does not detect any task acquisition request from a neighboring server within a preset long time window, the complementary value of that neighboring server is set to 0.5 and remains so until the next task acquisition request from that neighboring server is detected.

[0032] In this embodiment of the invention, it is assumed that in the pre-defined multidimensional baseline vector, the queue length baseline is 100, the CPU utilization baseline is 0.5 (i.e., 50%), and the network communication latency baseline is 10 milliseconds. For a certain neighboring server, its current historical success rate is 0.7, then the complementary value (i.e., the estimated load level) is 1 - 0.7 = 0.3. Multiplying each dimension of the baseline vector by 0.3, the multidimensional state information of the neighboring server is obtained: queue length = 100 × 0.3 = 30, CPU utilization = 0.5 × 0.3 = 0.15 (i.e., 15%), and network latency = 10 × 0.3 = 3 milliseconds. This simulated multidimensional state information is perfectly aligned with the actually collected queue length, CPU utilization, and network latency in terms of attributes. Therefore, it can be directly concatenated with the state information of the local server obtained from local monitoring to form the complete input of the deep reinforcement learning model. Furthermore, to prevent information from becoming outdated due to prolonged lack of interaction, this server maintains a preset long window. For example, if no task acquisition request is received from a neighboring server within 10 minutes, its complementary value is forcibly set to 0.5 (i.e., the historical success rate regression neutral value) and remains unchanged until the next request from that neighbor is received. The effect is that when a neighboring server does not actively steal tasks for a long time, this server cannot obtain any updates through passive listening. In this case, using neutral estimation avoids making extremely erroneous decisions due to blindly trusting outdated information; it avoids overestimating the availability of neighboring servers, leading to ineffective migrations, and underestimating them, resulting in missed idle nodes. The entire mapping process involves only one subtraction, one multiplication, and a timer reset operation, with extremely low computational overhead, making it very suitable for resource-constrained edge devices. The forced neutralization strategy combined with the long window is equivalent to a forgetting mechanism, allowing the mapping results of historical success rates to adaptively reflect the uncertainty of the current environment without increasing any active probing communication.

[0033] like Figure 5 As shown in the preferred embodiment of the present invention, the step of outputting a decision action for obtaining the task to be processed from the target neighboring server through a deep reinforcement learning model specifically includes: S201, attach a confidence coefficient to the status information of each adjacent server, the confidence coefficient being inversely proportional to the time difference between the time when the adjacent server was last passively monitored and the current time; S202, Based on a deep reinforcement learning model, calculate the basic Q value of each candidate action according to the current state information, wherein the candidate action is to select a neighboring server to obtain the task to be processed; S203, multiply the base Q value of each candidate action by the corresponding confidence coefficient to obtain the final Q value; S204: Select the candidate action with the largest final Q value as the final output decision action.

[0034] In this embodiment of the invention, if a neighboring server was detected sending a task retrieval request 0.5 seconds ago, its confidence coefficient can be set to 1.0 (indicating the information is very fresh); if the neighboring server has not been detected sending any requests within 10 seconds, its confidence coefficient can be set to 0.1 (indicating the information is outdated and has low credibility). For example, the confidence coefficient can be defined as... α is a preset time decay factor, and the confidence coefficient is clipped to the [0,1] interval. The smaller Δt (i.e., time difference) is, the closer the confidence coefficient is to 1; the larger Δt is, the closer the confidence coefficient is to 0. After the deep reinforcement learning model calculates the basic Q value of each candidate action (i.e., selecting a neighboring server as the task acquisition target) based on the current state information, the basic Q value of each candidate action is multiplied by the confidence coefficient of its corresponding target neighboring server to obtain the corrected Q value. For example, the basic Q value of neighboring server A is 0.8 but the confidence coefficient is only 0.2 (no interaction for a long time), and after correction, it is only 0.16; the basic Q value of neighboring server B is 0.6 but the confidence coefficient is as high as 0.9 (recent interaction), and after correction, it is 0.54. In this case, the model will select neighboring server B with the largest corrected Q value as the output action. This approach directly embeds the freshness of state information into the decision-making process, making the model naturally inclined to select neighboring servers with recent interactions and relatively certain states as targets for data theft. This avoids making risky decisions based on outdated or ambiguous neighboring server information, thus significantly improving the success rate of task acquisition and system stability while maintaining a fully distributed system without active probing. Furthermore, the confidence coefficient is calculated based on the time difference, making it very simple.

[0035] like Figure 6 As shown, in a preferred embodiment of the present invention, the step of requesting the task to be processed from the target neighboring server and adding the obtained task to the local task queue specifically includes: S301, the server first sends a probe request to the target neighboring server. The probe request only requests to obtain a minimum unit of data packet and does not carry the complete content. S302, calculate the estimated transmission cost for this acquisition based on the response time of the probe request and the queue status returned by the target neighboring server. If the estimated transmission cost exceeds the preset threshold, cancel the acquisition task and reselect the target neighboring server. S303, if the estimated transmission cost does not exceed the preset threshold, then formally request the target server to obtain the task to be processed, and only one task to be processed can be obtained in one request. S304 After adding the task to the local pending queue, a preset short timer is started, so that the server will not send a new task retrieval request to the same target server before the timeout.

[0036] In this embodiment of the invention, after identifying the target neighboring server, instead of directly requesting the task to be processed, a probe request (which can be an empty heartbeat packet or a single byte of test data) is first sent to the target neighboring server. The probe mechanism assesses the migration cost in advance with minimal communication overhead, avoiding high latency or failure caused by blind migration. The server records the response time of the probe request and obtains the current queue length (i.e., the queue status) returned by the target neighboring server. The estimated transmission cost can be obtained by multiplying the response time by the queue length plus one and then squared (aiming to exponentially penalize the behavior of initiating task migration to servers with long queues, thereby forcing the model to prioritize neighboring servers with shorter queue lengths). If the cost exceeds a preset threshold, the task acquisition is canceled, and the decision action is re-executed to select another target neighboring server. If the preset threshold is not exceeded, the server formally requests the task to be processed from the target neighboring server, and only one task is acquired per request, rather than multiple tasks in batches. This prevents the batch migration of incorrect tasks due to outdated status and avoids instantaneous load impact on the target server. Once a task is successfully added to the local processing queue, the server immediately starts a preset short timer (e.g., 500 milliseconds). Before this timer expires, the server will not initiate any new task retrieval requests to adjacent servers targeting the same target. This prevents the server from repeatedly attacking the same target within a short period, allowing other servers opportunities to steal tasks and giving the target server sufficient processing time, thereby improving the success rate of task retrieval and the overall throughput stability of the system.

[0037] like Figure 7 As shown, in a preferred embodiment of the present invention, the step of calculating the reward value based on the system performance feedback after the execution of the decision action specifically includes: S401, Before executing this decision action, record the average waiting time of the local pending queue on this server as the first time value; S402, After executing this decision action, remeasure the average waiting time of the local pending queue of this server as the second time value; S403, the first component of the reward value is obtained by subtracting the second time value from the first time value, and the first component represents the amount of reduction in the average waiting time; S404, record the queue length of the target's adjacent servers, and compare the queue length with the queue length before the current decision action. If the queue length has not increased, generate a preset normal number as the second component of the reward value; otherwise, the second component is zero. S405, add the first component and the second component to get the final reward value.

[0038] In this embodiment of the invention, it is assumed that before the server executes the decision action, the average waiting time of the local queue is 200 milliseconds (first time value). After executing the decision action, the server successfully obtains a task to be processed from the target neighboring server and adds it to the local queue. At this time, due to the addition of the new task, the queue length increases, and the remeasured average waiting time becomes 210 milliseconds (second time value). Then the first component is -10 milliseconds, indicating that the average waiting time increased by 10 milliseconds, which is a negative reward. At the same time, it is recorded that the queue length of the target neighboring server before executing the decision action was 5 tasks, and after the action, the queue length of the target server became 4 tasks (the number of tasks decreased, indicating that its load capacity is sufficient). Therefore, the second component takes a preset positive constant (this positive constant is a small value relative to the first component, mainly serving as a bias), for example, +0.1. The final reward value is -9.9. Conversely, if the average waiting time of the local server decreases to 190 milliseconds after the action is performed (for example, the stolen task happens to be a special task that the local server urgently needs and can accelerate subsequent processing), then the first component is +10 milliseconds; if the target server's queue length does not increase, then the second component is still +0.1, and the final reward value is 10.1. In this reward calculation method, the first component can directly reflect the changing trend of the local server's own performance, enabling the deep reinforcement learning model to learn to select theft actions that can reduce or minimize local queuing latency; while the second component encourages the deep reinforcement learning model to select target servers with sufficient load capacity (i.e., the other party's queue does not increase or decrease after theft), avoiding cascading overload on the target.

[0039] The above description only details the preferred embodiments of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0040] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0041] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. An edge computing server information processing method based on deep reinforcement learning, characterized in that, The method includes the following steps: Each edge server is configured with an independent deep reinforcement learning model, and the status information of the server and multiple neighboring servers is collected in real time. The status information includes the queue length of the local server and its neighbors, the CPU utilization of the local server and its neighbors, and the network communication latency of the local server and its neighbors. Based on the collected state information, a deep reinforcement learning model is used to output decision actions for obtaining the task to be processed from the target's neighboring servers; According to the decision action, request the task to be processed from the target adjacent server, and add the obtained task to the local task queue; The reward value is calculated based on the system performance feedback after the decision action is executed, and the data of task transfer is stored in the experience pool to update the deep reinforcement learning model.

2. The edge computing server information processing method based on deep reinforcement learning according to claim 1, characterized in that, The step of collecting status information of the local server and multiple neighboring servers in real time specifically includes: Maintain a binary feedback table locally on this server. This feedback table records the historical success rate of each neighboring server's interaction with this server, and initializes all historical success rates to the same value. The historical success rate is updated by passively listening for task retrieval requests from adjacent servers. When it is necessary to collect status information of neighboring servers, read the historical success rate of each neighboring server from the binary feedback table. The historical success rate of each adjacent server is converted into a multi-dimensional state vector through a preset mapping function. The multi-dimensional state vector has the same attributes as the state information, such as queue length, CPU utilization and network communication latency. The multidimensional state vectors of each adjacent server are merged with the local state information of the server collected by the local monitoring module.

3. The edge computing server information processing method based on deep reinforcement learning according to claim 2, characterized in that, The step of updating the corresponding historical success rate by passively listening to task acquisition requests from adjacent servers specifically includes: Preset a positive step size and a maximum number of listeners within a unit time window; When this server receives a task retrieval request from a neighboring server, if the cumulative number of times that neighboring server has listened within the current time window has not reached the upper limit, then the historical success rate of that neighboring server is increased by a positive step, and the cumulative number of listening is incremented by one. If the cumulative number of times the adjacent server has listened within the current time window has reached the upper limit, then this listening event will be ignored and its historical success rate will not be updated. At the end of each unit time window, the cumulative listening count of all adjacent servers is reset to zero.

4. The edge computing server information processing method based on deep reinforcement learning according to claim 2, characterized in that, The step of converting the historical success rate of each adjacent server into a multi-dimensional state vector using a preset mapping function specifically includes: A multi-dimensional baseline vector is preset, and each dimension of the vector corresponds to the default baseline values ​​of queue length, CPU utilization, and network communication latency, respectively. For each neighboring server, its current historical success rate is read, and its complementary value is calculated. The complementary value represents the estimated load level of the neighboring server, and the sum of the complementary value and the current historical success rate is 1. Multiply each dimension of the multidimensional baseline vector by its complementary value to obtain a new multidimensional vector, which is then used as the multidimensional state information of the neighboring server. If this server does not receive any task retrieval request from a neighboring server within a preset long time window, the complementary value of that neighboring server will be set to 0.5, and will remain so until the next task retrieval request from that neighboring server is received.

5. The edge computing server information processing method based on deep reinforcement learning according to claim 1, characterized in that, The step of outputting decision actions for obtaining the task to be processed from the target neighboring server through a deep reinforcement learning model specifically includes: A confidence coefficient is attached to the status information of each neighboring server. The confidence coefficient is inversely proportional to the time difference between the last time the neighboring server was passively monitored and the current time. Based on a deep reinforcement learning model, the basic Q value of each candidate action is calculated according to the current state information. The candidate action is to select a neighboring server to obtain the task to be processed. The final Q-value is obtained by multiplying the base Q-value of each candidate action by the corresponding confidence coefficient. The candidate action with the largest final Q value is selected as the final output decision action.

6. The edge computing server information processing method based on deep reinforcement learning according to claim 1, characterized in that, The step of requesting tasks to be processed from adjacent servers and adding the obtained tasks to the local processing queue specifically includes: This server first sends a probe request to the target's neighboring server. The probe request only requests to obtain a minimum unit of data packet and does not carry the complete content. The estimated transmission cost for this acquisition is calculated based on the response time of the probe request and the queue status returned by the target neighboring server. If the estimated transmission cost exceeds the preset threshold, the acquisition task is canceled and a new target neighboring server is selected. If the estimated transmission cost does not exceed the preset threshold, a formal request is made to the target server to obtain the task to be processed, and only one task to be processed is obtained in one request. After adding the task to the local pending queue, a preset short timer is started, so that the server will not send a new task retrieval request to the same target server before the timeout.

7. The edge computing server information processing method based on deep reinforcement learning according to claim 6, characterized in that, The estimated transmission cost is calculated by multiplying the response time of the probe request by the square of the current queue length of the target's adjacent server plus one.

8. The edge computing server information processing method based on deep reinforcement learning according to claim 1, characterized in that, The step of calculating the reward value based on the system performance feedback after the execution of the decision action specifically includes: Before executing this decision action, record the average waiting time of the local pending queue on this server as the first time value; After executing this decision action, the average waiting time of the local pending queue on this server is remeasured as the second time value; The first component of the reward value is obtained by subtracting the second time value from the first time value. The first component represents the amount of reduction in the average waiting time. Record the queue length of the target's adjacent servers and compare it with the queue length before the current decision action. If the queue length has not increased, generate a preset normal number as the second component of the reward value; otherwise, the second component is zero. The first component is added to the second component to obtain the final reward value.

Images