Edge network service state optimization method and system based on reinforcement learning

Abstract

The invention relates to the technical field of edge optimization, in particular to an edge network service state optimization method and system based on reinforcement learning. The method comprises the following steps: acquiring edge computing network operation data; an edge computing network optimization model is constructed, including dynamic environment-oriented Markov decision process MDP modeling and model training and optimization based on a breach deep Q network D3QN; performing data optimization by using the trained edge computing network optimization model; and outputting the optimized data. Compared with a traditional information age (AoI) and query information age (QAoI) method, the TPAoI index provided by the invention depicts the life cycle of the state information more completely, and the accuracy of optimization decision making is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of edge optimization, and in particular, to a method and system for optimizing the edge network service state based on reinforcement learning. Background Art

[0002] In recent years, with the rapid development of 5G communication, Internet of Things (IoT), and artificial intelligence technologies, the computing power network has become an important infrastructure to support key fields such as intelligent transportation, industrial Internet, and smart city. In the computing power network, edge services significantly reduce data transmission latency and improve the response efficiency of real-time sensitive applications by sinking computing power to the network edge. In a dynamic and complex network environment, edge servers need to continuously update and transmit key information such as their load status and resource availability to the access point for users to consume. If the status information is not updated in a timely manner, it may cause users to initiate requests based on outdated information, leading to problems such as service queuing and task failure, seriously affecting system reliability and user experience. Taking autonomous driving as an example, vehicles need to obtain the computing resource status of edge servers in real time to decide whether to offload high-load tasks to edge servers. If the status information is stale, the vehicle may send tasks to an overloaded server, resulting in task processing delays and even endangering driving safety. In view of this series of problems, an edge service state update optimization mechanism is urgently needed to improve the timeliness and accuracy of service state information by finding more efficient information update strategies and intelligent decision-making methods.

[0003] Traditional edge service state update optimization is mainly based on the Age of Information (AoI) or Query Age of Information (QAoI) metrics. AoI reduces the latency from information generation to reception through periodic updates, but ignores the randomness of user access, which is likely to cause resource waste. QAoI focuses on the freshness of information at the user query moment and adopts an on-demand pull method. Although it reduces the update frequency, it does not consider the transmission delay between state updates and query requests. In terms of optimization methods, heuristic algorithms are simple and efficient but lack global optimization capabilities; queuing models have limited modeling of system behavior and are difficult to handle the random changes of tasks and requests; mathematical programming methods are highly dependent on system parameters and have large computational overheads, making them unsuitable for real-time decision-making. Generally speaking, traditional methods are difficult to balance the freshness of service state and system scheduling efficiency when facing complex characteristics such as the coupling of computing and transmission and the dynamic interweaving of tasks and requests in the edge computing network.

[0004] By analyzing and summarizing the existing edge service status optimization methods, traditional methods have obvious deficiencies in the following aspects: 1) They fail to consider the three complete service status lifecycle stages of update transmission, user access, and user requests simultaneously, resulting in the inability to comprehensively optimize system performance; 2) Traditional optimization methods have insufficient adaptability to network dynamic characteristics and user random behaviors, making it difficult to maintain stable optimization effects in a dynamic edge environment; 3) They ignore the resource efficiency problem in the state update process and fail to effectively balance the relationship between information freshness and network transmission overhead, causing unnecessary resource waste. These deficiencies seriously restrict the practical application effect of the edge service status management system. Summary of the Invention

[0005] To solve the above-mentioned problems, the present invention provides an edge network service status optimization method and system based on reinforcement learning. It can effectively solve the above problems, achieve more effective state updates, and improve the timeliness and accuracy of service status information.

[0006] In the first aspect, an edge network service status optimization method based on reinforcement learning provided by the present invention adopts the following technical solutions: An edge network service status optimization method based on reinforcement learning includes: Obtain the operation data of the edge computing network; Construct an edge computing network optimization model, including Markov decision process (MDP) modeling for a dynamic environment and model training and optimization based on duel deep Q-network (D3QN); Use the trained edge computing network optimization model to optimize data; Output the optimized data.

[0007] Further, the obtaining of the operation data of the edge computing network includes collecting key operation data generated by edge servers, access points, and user terminals during the service process, including status update transmission time, age of information (AoI) at the access point, user access interval, request transmission time, and the age of information corresponding to the request; time stamps are embedded in the status update and request messages to record the sending and receiving times.

[0008] Further, the Markov decision process (MDP) modeling for a dynamic environment includes defining the age of information metric TPAoI, comprehensively considering the three stages of the service status lifecycle to ensure that the state information is comprehensively and accurately evaluated: namely, the state update stage: the edge server sends service status information to the network access point (AP); the user access stage: the user randomly accesses the AP to obtain the status information; the user request stage: the user sends a computing request to the remote edge server based on the obtained status information. The mathematical expression of TPAoI is: , Among them, is the time when the edge server sends the th service status update, is the time when the kth status update arrives at the AP, is the time when the user accesses the AP for the ith time to obtain status information, The time when the user request arrives at the remote edge server.

[0009] Furthermore, the Markov decision process MDP modeling for the dynamic environment includes states , actions and a reward function defined based on the age of information metric TPAoI, and introduces transmission costs to balance information freshness and update costs, expressed as: , Among them, represents the negative reward generated by sending the status update, represents the dimension of, represents the ith component of, represents whether it arrives at the AP, when arrives at the AP, the value of is the TPAoI.

[0010] Furthermore, the Markov decision process MDP modeling for the dynamic environment also includes the injection of prior knowledge based on the service status life cycle, and divides the service status life cycle into three stages: update transmission, user access, and user request for feature segmentation processing to explicitly guide the network to better focus on key state combinations, specifically: Combined features of the update and user access stages: , Features of the user access waiting stage: , Combined features of the user request stage: , Feature fusion constitutes the final network input: , Among them, represents the activation function, represents the concatenation operation, represents the matrix product, and W and b represent trainable parameters.

[0011] Furthermore, the Markov decision process (MDP) modeling for a dynamic environment further includes optimizing for the user access intervals following a long-tail distribution. Specifically, the training data is segmented according to the user access intervals, and models are independently trained for short-interval and long-interval user data respectively; logarithmic transformation is performed on the long-tail distributed data to reduce the data scale difference; and probability filtering is applied to the low-value experiences generated by long intervals to reduce the interference of invalid training samples, thereby enhancing the generalization ability and robustness of the model to the user access behaviors following a long-tail distribution.

[0012] Furthermore, the model training and optimization based on the duel deep Q-network (D3QN) includes training the Markov decision process (MDP) using a deep reinforcement learning algorithm based on the duel network and the double deep Q-network. Among them, the duel network architecture represents the state value function and the advantage function separately to improve the decision-making ability and generalization performance of the network: , , where MLP represents the multi-layer perceptron, represents the trainable parameters, the state value function and the advantage function are respectively modeled through the shared network to improve the generalization ability and learning efficiency.

[0013] Furthermore, the training of the Markov decision process (MDP) using the deep reinforcement learning algorithm based on the duel network and the double deep Q-network includes using an experience replay buffer and a soft update mechanism to improve the training stability and convergence of the model. Among them, the experience replay buffer stores the experience tuples and randomly samples mini-batch data for parameter update during training; and the loss function for network training is defined as the squared expectation of the TD error , and the parameter update uses stochastic gradient descent optimization, which is expressed as: , , , , where represents the expectation, represents the gradient operation, represents the parameters of the neural network, is the discount factor, is the soft update coefficient, and are two neural networks with the same structure but different parameters 。

[0014] In a second aspect, an edge network service state optimization system based on reinforcement learning includes: A data acquisition module configured to acquire edge computing network operation data; A model construction module configured to construct an edge computing network optimization model, including Markov decision process (MDP) modeling for a dynamic environment and model training and optimization based on the dueling deep Q-network (D3QN); An optimization module configured to perform data optimization using the trained edge computing network optimization model and output optimized data.

[0015] In a third aspect, the present invention provides a computer-readable storage medium storing multiple instructions adapted to be loaded and executed by a processor of a terminal device for the method for optimizing the edge network service state based on reinforcement learning.

[0016] In a fourth aspect, the present invention provides a terminal device including a processor and a computer-readable storage medium, the processor being used to implement each instruction; the computer-readable storage medium being used to store multiple instructions adapted to be loaded and executed by the processor for the method.

[0017] In summary, the present invention has the following beneficial technical effects: Aiming at the problems of untimely update of service state information, waste of transmission resources, and decline in user experience in the edge computing network, the present invention proposes an optimization method based on the three-stage age of information (TPAoI) and deep reinforcement learning. This method divides the edge service state update process into three stages: state transmission, user access, and user request, constructs a TPAoI metric to comprehensively measure information freshness, and constructs a Markov decision process model based on this. By introducing a prior knowledge injection mechanism, the state life cycle is segmented and modeled, and a special optimization strategy is designed in combination with the long-tail distribution characteristics of user access behavior, enabling the model to more effectively capture dynamic features. Finally, a stable policy training is achieved using the D3QN model to optimize the service state update decision and improve the overall system performance.

[0018] The present invention has the following technical effects: First, the present invention proposes a new three-stage age of information (TPAoI) metric, which comprehensively considers the entire process of status information in the update, user access, and user request stages, and more accurately reflects the real-time nature of service status information; Second, by constructing a Markov decision process (MDP) model, the dynamics of the edge network environment and the random access behavior of users are effectively characterized, realizing real-time optimization of service status update decisions; Third, a deep reinforcement learning (D3QN) method combining prior knowledge injection and long-tail distribution optimization is proposed, significantly improving the adaptability and generalization performance of the model to complex dynamic environments and unbalanced data distributions; Finally, through the actual deployment and dynamic adjustment mechanism of the model, the long-term optimization performance is effectively maintained, the network resource overhead is reduced, and the overall stability and user experience of the system are improved.

[0019] The present invention has the following advantages: 1) Compared with traditional age of information (AoI) and query age of information (QAoI) methods, the TPAoI metric proposed by the present invention more completely characterizes the life cycle of status information, improving the accuracy of optimization decisions; 2) Compared with traditional optimization methods (heuristics, queuing models, mathematical programming), the present invention uses the D3QN model, which has higher generalization ability, a more stable training mechanism, and better real-time decision-making performance; 3) The proposed prior knowledge injection and long-tail distribution optimization strategy enable the model to more effectively capture the dynamic characteristics of the edge computing network, significantly improving the robustness and generalization ability in actual complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is the overall schematic diagram of the edge computing network service status update optimization in Embodiment 1 of the present invention; Figure 2 is the schematic diagram of the composition and life cycle of the three-stage age of information TPAoI in Embodiment 1 of the present invention; Figure 3 is the schematic diagram of the Markov decision process MDP state modeling in Embodiment 1 of the present invention; Figure 4 is the schematic diagram of the deep reinforcement learning network structure based on D3QN in Embodiment 1 of the present invention; Figure 5 is the detailed flowchart of the D3QN training process in Embodiment 1 of the present invention; Figure 6 is the overall flowchart of the service status update optimization method in Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The present invention will be further described in detail below with reference to the accompanying drawings.

[0022] Embodiment 1 Reference Figure 1 , a method for optimizing the edge network service state based on reinforcement learning in this embodiment includes: Obtain the operation data of the edge computing network; Construct an edge computing network optimization model, which includes Markov decision process (MDP) modeling for dynamic environments and model training and optimization based on the dueling deep Q-network (D3QN); Use the trained edge computing network optimization model to optimize the data.

[0023] Specifically: Step 1: Obtain the operation data of the edge computing network, including collecting key operation data generated by edge servers, access points, and user terminals during the service process, specifically including the following five categories: 1) State update transmission time: Record the transmission delay of the state update sent by the edge server on the link, which is used to model the dynamic process in the state update phase; 2) Age of information (AoI) at the access point: Represents the freshness of the state after it arrives at the access point and is used to evaluate the effectiveness of the state; 3) User access interval: Record the time interval between two user accesses to the access point, which is used to model the randomness of user access; 4) Request transmission time: Represents the transmission delay of the user request from the access point to the edge server; 5) Age of information corresponding to the request: Record the "age" of the state information carried by the request when it arrives at the edge server, which is used to calculate the TPAoI metric.

[0024] The above data is collected through Telemetry, Piggybacking, and the active reporting mechanism of the access point. Timestamps are embedded in the state update and request messages to record the sending and receiving times, and the access point and the edge server can parse and calculate relevant metrics. At the same time, the system adopts a unified time synchronization mechanism to ensure the consistency and accuracy of time data, and to accurately model the state update, access behavior, and request delay. In addition, when the user accesses but does not send a request, the access point will also actively notify the edge server to reflect the change in the access state.

[0025] Step 2: Design the three-phase age of information (TPAoI) metric based on the obtained data, For the convenience of modeling and subsequent policy optimization, the present invention assumes that time is divided into continuous and equal-length time slots. Each state update, user access, and user request behavior occur at the boundaries of time slots. On this basis, the present invention defines a novel age-of-information metric TPAoI, which comprehensively considers the three stages of the service state life cycle to ensure that the state information is comprehensively and accurately evaluated: 1) State update stage: The edge server sends service state information to the network access point (AP); 2) User access stage: The user randomly accesses the AP to obtain state information; 3) User request stage: The user sends a computing request to the remote edge server based on the obtained state information. The mathematical expression of TPAoI is as follows: , where is the time when the edge server sends the th service state update, is the time when the kth state update arrives at the AP, is the time when the user accesses the AP for the ith time to obtain state information, the time when the user request arrives at the remote edge server. The goal of the present invention is to minimize the long-term expected TPAoI, that is, to minimize : , where lim represents the limit operation, sup represents the supremum, E represents the expectation, represents for each i of

[0026] Step 3, Markov decision process (MDP) modeling for a dynamic environment, The Markov decision process (MDP) is a mathematical framework that describes the evolution of decision-making behavior in an uncertain dynamic environment and is widely used in reinforcement learning and intelligent control. Its core idea is that the system executes an action in a certain state, transfers to a new state with a certain probability, and obtains a reward. The MDP is composed of a five-tuple <S, A, P, R, γ>, where S is a finite state set, A is an action set, P is a state transition probability function, R is a reward function, and γ is a discount factor (taking values from 0 to 1) used to balance immediate and future rewards. Through MDP modeling, the system can continuously learn the optimal policy in the interaction to achieve intelligent decision-making and system optimization. The present invention makes a specific definition of MDP modeling in combination with the actual characteristics of the edge computing system: The state space (S) is represented by multiple sub-states : (1) represents the transmission time of state updates, which is an N-dimensional vector indicating that N state updates can be performed simultaneously. The timing starts from when the server sends the state update until the AP receives the update, and is reset to 0 when the update is received: , wherein, represents the transmitted time of the i-th status update being transmitted, represents the action of the current time slot, represents the newly assigned status number, represents whether the edge service sends an update.

[0027] (2) represents the age of information of the AP. If no new status update is received, it increases by 1 in each time slot and is reset to the transmission delay after receiving a new update: .

[0028] (3) represents the time interval between user accesses to the AP. When not accessed, it increases by 1 in each time slot and is reset to 1 when accessed: .

[0029] (4) represents the transmission delay requested by the user. It is an M-dimensional vector indicating that M requests can be made simultaneously. Timing starts after the user sends a request to the AP and is reset to 0 when the request reaches the edge server: , wherein, represents the transmitted time of the i-th user request being transmitted, represents the number of the currently newly initiated request.

[0030] (5) represents the age of information of the user request. It is an M-dimensional vector indicating that M requests can be made simultaneously. It is set to the current AoI of the AP when the request is sent, increments by 1 in each time slot during transmission, and is reset to 0 when it reaches the server: , wherein, represents the age of information corresponding to the i-th request, represents the request number corresponding to the current access.

[0031] Through the modeling and dynamic update of the above five sub-states, the change characteristics of the system in various aspects such as state transmission, state aging, user access behavior, request transmission process, and request information validity can be comprehensively captured, providing sufficient information support for the reinforcement learning agent to make accurate decisions in a dynamic environment. Action space (A) Decision on whether the server sends a service status update in each time slot . Reward function (R) is designed using TPAoI and introduces transmission cost , balancing information freshness and update cost: , wherein, represents the negative reward generated by sending the status update, represents the i-th component of represents whether it reaches the AP. When reaches the AP, its value is the TPAoI of step 1. Through this step, the Markov decision modeling of the TPAoI metric defined in step 1 is realized, and the complex dynamic environment is effectively captured.

[0032] On this basis, the long-term optimization goal of reinforcement learning is to seek a set of optimal policies such that when starting from the initial state , the expected state value is maximized, that is: , wherein, the state value function is defined as the expected cumulative discounted reward starting from the state under the policy : , the cumulative discounted reward is defined as: , wherein, is the discount factor, which is used to balance the importance of immediate rewards and future rewards, and ensures the convergence of the state value function . Optimizing this goal can not only minimize the long-term expected TPAoI metric defined in step 2, but also comprehensively consider the transmission cost brought by state updates, and achieve an effective trade-off between information freshness and system energy consumption.

[0033] Step 4, injection of prior knowledge, 1) A method for injecting prior knowledge based on the service state life cycle, The present invention proposes a method for injecting prior knowledge, which explicitly divides the service state life cycle into three stages: update transmission, user access, and user request for feature segmentation processing, so as to explicitly guide the network to better focus on key state combinations, specifically as follows: (1) Combined features of the update and user access stages: , (2) Features of the user access waiting stage: , (3) Combined features of the user request stage: , Among them, is a trainable linear transformation matrix, is a bias term, is an activation function (such as ReLU), represents a concatenation operation, represents a matrix product.

[0034] Feature fusion constitutes the final network input, and there are two ways of data fusion: (1) Using linear transformation for feature fusion: , (2) Using the attention mechanism for feature fusion: Based on the complete global state , a global feature vector is generated, and its expression is: , Subsequently, is used as the query vector, is used as the value vector, and the attention weights are calculated through the attention mechanism and weighted aggregation is performed to obtain the final fused state representation : , .

[0035] Although the linear transformation has a relatively fast fusion speed, the feature fusion effect is poor; while the attention mechanism can achieve better feature fusion, but the execution speed is relatively slow. Through the explicit injection of the above prior knowledge, the model can more effectively capture the dynamic features at different stages, which helps to improve the generalization performance and training efficiency of the model.

[0036] 2) Optimization method for the user access interval following a long-tail distribution Although the above prior knowledge injection method has a significant effect when dealing with user periodic access, when the user access interval follows a long-tail distribution, the model will face problems such as highly imbalanced training data and difficulty in effectively modeling some long-interval data, resulting in a decline in model performance. To solve this problem, based on the above optimization method, the present invention proposes the following three optimization strategies and implements them in combination with specific mathematical modeling: (1) Data grouping and independent modeling: According to the user behavior characteristics of different access intervals, the training data is divided into a short-interval group and a long-interval group according to the size of the access interval . Set the segmentation threshold , when , it is divided into the short-interval data set , otherwise it is included in the long-interval data set : , Correspondingly, two sub-models are trained respectively, and the sub-models respectively minimize the mean square error of the expected TD error on their respective data sets: , where is the target Q value, calculated based on the double deep Q network, and the training process refers to step 5.

[0037] (2) Logarithmic transformation to reduce scale differences: To alleviate the problem of too large a scale span of access intervals caused by the long-tailed distribution, the access interval feature is logarithmically transformed to standardize the data range. The transformation function is defined as: , After the transformation, the feature range is more concentrated, improving the training stability and convergence speed of the neural network. The final input feature vector is constructed as follows: .

[0038] (3) Experience filtering mechanism based on access intervals: For the low-value experience data generated under long access intervals, a probability sampling mechanism based on access intervals is adopted to reduce the storage probability of invalid samples. The experience sampling probability function is defined as: , where is the maximum sampling probability, is the rate factor that controls the decrease of the sampling probability with the access interval. The probability that the experience tuple enters the experience replay buffer is: , Through this mechanism, the proportion of high-density short access interval data is increased, the interference of long-tailed low-value data on the training process is reduced, and thus the convergence of the model is accelerated and the generalization performance is improved.

[0039] Step 5, Model training and optimization based on the dueling deep Q network (D3QN), To effectively solve the MDP problem, the present invention designs a deep reinforcement learning algorithm based on the dueling network and the double deep Q-network for optimization.

[0040] 1) Model construction, (1) Dueling network architecture, representing the state value function and the advantage function separately to improve the decision-making ability and generalization performance of the network: , , Among them, MLP represents a multi-layer perceptron, represents trainable parameters, and the state value function and the advantage function are respectively modeled through a shared network to improve the generalization ability and learning efficiency.

[0041] Construct a dueling network architecture: First, construct a shared feature extraction network according to the prior knowledge injection method in Step 4. Its input is state information, and its output is a representation vector of the state. Subsequently, according to the fusion formula of the dueling network architecture, construct a Q-value network. Its input is the representation vector of the state, and its output is the state-action value (Q-value) corresponding to each action.

[0042] (2) To alleviate the problem of overestimation that is prone to occur in the calculation of the target Q-value in the traditional deep Q-network, the present invention introduces a double deep Q-network structure. This structure includes two neural networks: the main network and the target network . The two have the same network structure but independent parameters. Both adopt the aforementioned dueling network architecture and are built according to the construction process of the dueling network.

[0043] 2) Model training, (1) The agent interacts with the environment and selects an action according to the current policy , and obtains a reward after execution and the next state , forming an experience tuple , and storing it in the experience replay buffer (Replay Buffer).

[0044] (2) Batch sampling: Randomly sample a small batch of data ( represents the size of the batch) for training.

[0045] (3) Target Q-value calculation: Input and into the target network to calculate the target Q-value: , Among them, represents the action with the largest Q-value selected on the main network corresponding to , , is the discount factor.

[0046] (4) Construct a loss function: Calculate the loss function in the form of the squared expectation of the TD error: , Among them, the TD error is expressed as , reflecting the deviation between the "current estimate" and the "one-step update target".

[0047] (5) Parameter update: Use stochastic gradient descent or the Adam optimizer to perform backpropagation and update on the main network parameters : .

[0048] (6) Soft update of the target network: Update the target network parameters in a soft update manner where is the soft update coefficient.

[0049] During the actual training process, first, according to the policy execution logic in Step 1, the agent continuously interacts with the environment, continuously collects and accumulates experience samples. When the number of interaction steps reaches the set interval, execute a training loop consisting of Steps 2 to 7 to update the parameters of the main network and optimize the policy. This process is executed cyclically during the system operation to achieve continuous optimization and dynamic adaptation of the policy.

[0050] Step 6, model deployment and application in the edge computing network Based on the optimized model trained in Step 5, the present invention further proposes the following deployment solutions: 1) Deploy the trained model to the edge computing server to achieve real-time dynamic decision-making; 2) Online collect actual operation data and incrementally train the model periodically to maintain the long-term effectiveness and robustness of the policy. Through the deployment and application in Step 6, all the above-mentioned optimized designs can truly play a role in the actual edge computing network environment, ensuring the continuous optimization effect of the TPAoI index defined in Step 1 in real applications.

[0051] Among them, in the intelligent manufacturing scenario, the edge network service state optimization method based on reinforcement learning proposed by the present invention is applied to the visual quality inspection task of the factory production line to support multiple device terminals to upload images in real time and obtain defect detection results. Multiple terminal devices (such as industrial cameras, robotic arm control units, etc.) in the intelligent factory communicate with the edge node through the access point to perform image recognition and quality detection tasks. The edge node runs the AI model to detect defects in the uploaded images, judge whether there are problems such as cracks, scratches, and material shortages, and return the results to the terminal for controlling subsequent sorting or alarm processes.

[0052] 1) State update mechanism: Edge service nodes periodically evaluate their own operating status (including computing load, task queuing situation, memory occupancy, etc.), and based on the reinforcement learning strategy, determine whether to send a status update to the access point. The status message is attached with timestamp information for subsequent calculation of the status transmission delay and the age of information metric. 2) Terminal access behavior: Terminal devices on different production lines access the access point at different time intervals according to the task rhythm to obtain the latest service status information, and combined with the current task requirements, determine whether to offload the task to the edge node for processing. Information such as the access interval and status reading time of the terminal is collected and used to optimize the reinforcement learning model. 3) Request upload and result return: After the terminal makes a decision, the image to be analyzed and the diagnostic request are sent to the edge node, which completes the analysis and processing and returns the result to control the production rhythm and pass rate determination. 4) Online optimization mechanism: To improve the adaptability and robustness of the model, the system continuously collects key data such as status update delay, access interval, and request transmission time after deployment, and periodically performs incremental training to update the policy model online, dynamically adapting to changes in terminal behavior and system load, and keeping the policy effective in the long term. 5) Application effect of the optimization strategy: Through the state update optimization method proposed in the present invention, the edge node can effectively control the update frequency while ensuring the freshness of the service status information, reduce unnecessary data transmission and network congestion, enabling the terminal to obtain more accurate status information during access, which helps to improve the timeliness of the overall system processing and the rationality of decision-making.

[0053] Figure 1 is the overall framework diagram of the edge computing network service status update optimization system proposed in the present invention, showing the interaction structure and status update process among the edge server, network access point (AP), and user terminal. Figure 2 is the schematic diagram of the composition and life cycle of the three-stage age of information (TPAoI) in the present invention, explaining the three-stage time composition in the whole process from service status transmission, user access to final request processing. Figure 3 is the state modeling framework diagram of the Markov decision process (MDP) constructed in the present invention, explaining the relationship among the three elements of the state space, action space, and reward function in the system and their logical relationship with the environment interaction. Figure 4 is the deep reinforcement learning network structure diagram based on D3QN in the present invention, showing the overall network structure of state input, feature extraction module with prior knowledge injection, branches of state value function and action advantage function in the dueling structure, and final Q-value output. Figure 5 is the detailed flowchart of the D3QN training process in the present invention, showing the complete training process from experience replay sampling to network parameter update and target network soft update. Figure 6It is the overall flowchart of the service status update optimization method proposed by the present invention, which successively includes core steps such as TPAoI index design, MDP modeling, prior knowledge injection, model training and deployment.

[0054] As a further implementation method, As Figure 1 As shown, the edge computing service status update optimization system proposed by the present invention mainly consists of an edge server, a network access point (AP), a user terminal, and a service status update optimization module. The entire system unfolds around three stages in the service status life cycle and forms a closed-loop optimization structure in combination with an intelligent decision-making mechanism. The system structure can be divided into the following three functional structures: Service status collection and transmission structure (edge server AP): The edge server is responsible for running computing tasks and periodically generating service status information, such as computing load, resource utilization, etc. This status information is sent to the access point (AP) in the form of data packets through the network link. The access point, as an information transfer node at the network edge, is responsible for receiving and caching the status data for user access.

[0055] User access and request interaction structure (AP User edge server): The user terminal accesses the AP irregularly according to its own service needs to obtain service status data, and decides whether to initiate a computing request to the edge server based on this. The AP provides the user with the status information it currently caches. After the user obtains the information, it decides whether to send a request, and this request is returned to the edge server for processing via the network link.

[0056] Service status update optimization and decision-making structure (based on MDP and D3QN): To reasonably control the service status update frequency, the present invention proposes an optimization decision-making mechanism based on the three-stage age of information (TPAoI), and uses reinforcement learning methods for training and deployment.

[0057] As Figure 6 As shown, the service status update optimization method proposed by the present invention aims to achieve efficient update and scheduling of service status information in a dynamic edge computing network, maximize the freshness of service status, and reduce unnecessary update overhead.

[0058] The above planning scheme is only the implementation method in the present invention, but the protection scope of the present invention is not limited thereto. All replacements or transformations that can be understood and conceived by those familiar with the technology within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

[0059] Embodiment 2 This embodiment provides an edge network service status optimization system based on reinforcement learning, including: A data acquisition module, configured to: A computer-readable storage medium storing multiple instructions, which are adapted to be loaded and executed by a processor of a terminal device for the method for optimizing the edge network service status based on reinforcement learning.

[0060] A terminal device, including a processor and a computer-readable storage medium, where the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are adapted to be loaded and executed by the processor for the method for optimizing the edge network service status based on reinforcement learning.

[0061] The above are all preferred embodiments of the present invention. Without limiting the protection scope of the present invention accordingly, therefore: Any equivalent changes made according to the structure, shape, and principle of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A method for optimizing edge network service status based on reinforcement learning, characterized in that: include: Obtain edge computing network operation data; Build an edge computing network optimization model, including Markov decision process (MDP) modeling for dynamic environments and model training and optimization based on dueling deep Q networks (D3QN); Use the trained edge computing network optimization model to optimize data; Output optimization data.

2. According to the edge network service state optimization method based on reinforcement learning in claim 1, it is characterized in that: The obtaining of edge computing network operation data includes collecting key operation data generated by edge servers, access points and user terminals during the service process, including status update transmission time, information age AoI at the access point, user access interval, request transmission time and information age corresponding to the request; embedding timestamps in status update and request messages to record sending and receiving times.

3. The edge network service state optimization method based on reinforcement learning according to claim 2 is characterized in that: The Markov decision process MDP modeling for dynamic environments includes defining an information age indicator TPAoI, comprehensively considering the three stages of the service status life cycle to ensure that the status information is comprehensively and accurately evaluated: namely, the status update stage: the edge server sends the service status information to the network access point AP; User access phase: users randomly access APs to obtain status information; User request phase: users send calculation requests to remote edge servers based on the obtained status information. The mathematical expression of TPAoI is: ， in, The edge server sends The time of the service status update, is the time when the kth status update arrives at the AP, is the time when the user visits the AP for the i-th time to obtain status information, The time it takes for the user request to arrive at the remote edge server.

4. The edge network service state optimization method based on reinforcement learning according to claim 3 is characterized in that: The Markov decision process MDP modeling for dynamic environments includes state ,Behavior And the reward function defined based on the information age indicator TPAoI, and the introduction of transmission cost, balance the information freshness and update cost, expressed as: ， in, represents the negative reward for sending a state update, express The dimension of express The i-th component of express Whether it reaches the AP, when When arriving at the AP, The value is TPAoI.

5. The edge network service state optimization method based on reinforcement learning according to claim 4 is characterized in that: The Markov decision process MDP modeling for dynamic environments also includes prior knowledge injection based on the service state life cycle, dividing the service state life cycle into three stages: update transmission, user access, and user request for feature segmentation processing, so as to explicitly guide the network to better focus on key state combinations, specifically: Update and user access phase combined features: ， User access waiting stage characteristics: ， User request stage combined features: ， Feature fusion constitutes the final network input: ， in, represents the activation function, Represents a splicing operation, represents matrix product, W and b are trainable parameters.

6. The edge network service state optimization method based on reinforcement learning according to claim 5 is characterized in that: The Markov decision process MDP modeling for dynamic environments also includes optimization for user access intervals following long-tail distribution, wherein training data is segmented according to user access intervals, and models are trained independently for short-interval and long-interval user data; logarithmic transformation is performed on long-tail distributed data to reduce data scale differences; and low-value experiences generated by long intervals are probabilistically filtered to reduce interference from invalid training samples, thereby improving the generalization ability and robustness of the model for long-tail distributed user access behaviors.

7. The edge network service state optimization method based on reinforcement learning according to claim 6 is characterized in that: The model training and optimization based on the dueling deep Q network D3QN includes training the Markov decision process MDP using a deep reinforcement learning algorithm based on the dueling network and the dual deep Q network, wherein the dueling network architecture represents the state value function and the advantage function separately to improve the decision-making ability and generalization performance of the network: ， ， Among them, MLP represents multi-layer perceptron, Represents trainable parameters, state value function and advantage function Through shared network Modeling is performed to improve generalization ability and learning efficiency.

8. The edge network service state optimization method based on reinforcement learning according to claim 7 is characterized in that: The Markov decision process MDP is trained using a deep reinforcement learning algorithm based on duel networks and dual deep Q networks. It also includes the introduction of a dual deep Q network structure, which contains two neural networks: a main network and a target network. The two network structures are the same but the parameters are independent. The duel network architecture is adopted and built according to the duel network construction process.

9. The edge network service state optimization method based on reinforcement learning according to claim 8, characterized in that: The method uses a deep reinforcement learning algorithm based on a duel network and a dual deep Q network to train a Markov decision process MDP, and also includes using an experience replay buffer and a soft update mechanism to improve the training stability and convergence of the model, wherein the experience replay buffer stores the experience tuples During training, randomly sample small batches of data to update parameters; and define the loss function of network training as TD error The square expectation of the parameter update is optimized using stochastic gradient descent, expressed as: ， ， ， ， in, Express expectations, represents the gradient operation, represents the parameters of the neural network, is the discount factor, is the soft update coefficient, and They are two neural networks with the same structure but different parameters.

10. An edge network service state optimization system based on reinforcement learning, characterized in that: include: A data acquisition module is configured to acquire edge computing network operation data; A model building module is configured to build an edge computing network optimization model, including Markov decision process MDP modeling for dynamic environments and model training and optimization based on dueling deep Q network D3QN; The optimization module is configured to optimize data using the trained edge computing network optimization model and output optimized data.

Images