Computing network integrated path optimization method based on A* and Q-Learning mixing

Abstract

The invention belongs to the technical field of computing power networks, reinforcement learning and intelligent routing, and discloses an A * and Q-Learning hybrid-based computing network integrated path optimization method, which comprises the steps of constructing a four-dimensional expansion state space, designing a hybrid evaluation function, designing a Sigmoid smooth reward function, generating a network resource reward and computing a resource reward. Based on network resource rewards and computing resource rewards, dynamic weight self-adaptive adjustment is achieved, the distribution proportion of network weights and computing weights is dynamically adjusted, a self-adaptive exploration rate adjustment mechanism is designed, the learning rate is dynamically adjusted according to the number of state access times, an experience playback mechanism is constructed, performance rewards are computed, and the network resource rewards and the computing resource rewards are trained in stages. And finally outputting an optimized routing path. According to the method, by fusing priori knowledge of heuristic search and adaptive ability of reinforcement learning, the convergence speed is increased by 3-5 times while the path quality is ensured, the flow completion rate reaches 100%, and the intelligent routing optimization problem is effectively solved.

Description

Technical Field

[0001] This application belongs to the fields of computing power networks, reinforcement learning, and intelligent routing technology. Specifically, it is a computing network integrated path optimization method based on a hybrid A* and Q-Learning approach. Background Technology

[0002] With the rapid development of computing networks, computing resources are widely deployed in wide area networks, forming a heterogeneous and multi-dimensional network-wide computing resource structure. The integration of computing and networks requires considering both network transmission performance and computing resource status during routing decisions to achieve coordinated optimization of computing power and network resources. However, existing routing optimization methods still have significant limitations in computing-network convergence scenarios.

[0003] While pure Q-learning methods can perceive network resources by expanding the state space, they face slow convergence in large-scale network environments. The expanded state space can reach tens of thousands of nodes, requiring extensive random exploration to discover effective paths, resulting in poor early routing quality and low real-time decision-making efficiency. They also suffer from sparse reward problems, where large rewards are only obtained upon reaching the target node, while reward signals for intermediate steps are weak and unhelpful in guiding the learning process. Furthermore, pure Q-learning methods are prone to getting trapped in local optima in complex mesh topologies; when the network state changes dynamically, the learned Q-values ​​may become outdated, and the algorithm lacks a rapid adaptation mechanism.

[0004] The A* heuristic search algorithm can quickly find paths using prior knowledge, but its heuristic function is usually based on a single distance metric, which cannot accurately reflect the complex resource status of a computing-network integrated environment and lacks adaptability in dynamic network environments. Traditional A* algorithms are mainly applied to path planning in static environments and are insufficiently adaptable to computing-network environments where resource status changes in real time. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a hybrid computation-network path optimization method based on a combination of A* and Q-Learning. This application constructs a hybrid evaluation function, implements dynamic weight adaptation and progressive pre-training strategies, and combines the advantages of heuristic search and reinforcement learning to achieve adaptive optimization while ensuring fast convergence, thus solving the problems of slow convergence and easy getting trapped in local optima in pure Q-Learning.

[0006] To achieve the above objectives, this application employs the following technical solution:

[0007] This application presents a computational network-integrated path optimization method based on a hybrid A* and Q-Learning approach, comprising the following steps:

[0008] Step 1: Obtain the initial data of the computing network environment. The initial data includes the network topology, node computing resource status, link bandwidth resources, and traffic information to be transmitted. Based on the initial data, construct a four-dimensional extended state space that includes node location, network load level, computing load level, and traffic type.

[0009] Step 2: Based on the four-dimensional extended state space constructed in Step 1, design a hybrid evaluation function combining A* heuristic search and Q-Learning:

[0010] f(s,a)=g(s+α h ·h(s,a)+α q ·Q exp (s,a)

[0011] Where g(s) is the actual cumulative cost from the starting point to the current node, h(s,a) is the heuristic estimation function from the current node to the target, and Q... exp (s,a) represents the exploratory Q value, α h and α q These are dynamic weighting coefficients;

[0012] Step 3: Design a Sigmoid smoothing reward function to perform a nonlinear mapping on network resources and computing resources, generating network resource rewards and computing resource rewards respectively, which are used for reward calculation of the hybrid evaluation function in Step 2;

[0013] Step 4: Based on the network resource reward and computing resource reward calculated in Step 3, implement dynamic weight adaptive adjustment, according to the real-time monitored average link utilization of the entire network. Average node computational load across the entire network Dynamically adjust network weights ω net Calculate the weight ω comp and distance weight ω dist The allocation ratio;

[0014] Step 5: Use the network weights ω determined in Step 4 net Calculate the weight ω comp and distance weight ω dist Configure and design an adaptive exploration rate adjustment mechanism based on the convergence degree of Q-value, dynamically adjust the learning rate of Q-value updates according to the number of state accesses, and achieve stable updates of the Q-table;

[0015] Step 6: Based on the Q-table updated in Step 5, construct an experience replay mechanism. After the stream transmission is completed, calculate the performance reward based on the ratio of the actual transmission time to the expected transmission time, and backpropagate to update the Q-value of all state-action pairs on the path.

[0016] Step 7: Repeat steps 1 to 6 to implement the progressive pre-training strategy. The progressive strategy includes the first stage of guided pre-training (n1 rounds), the second stage of intelligent pre-training (first sub-stage n2 rounds), and the third stage of intelligent pre-training (second sub-stage n3 rounds). By training the algorithm of this application in stages, it gradually adapts to the complex network environment and finally outputs the optimized routing path.

[0017] A further improvement in this application is that, in step 2, the heuristic estimation function h(s,a) employs a combination of Manhattan distance and resource penalty:

[0018] h(s,a)=d man (s′,goal)+p res (s′)

[0019] d man (s′,goal)=|x s′ -x goal |+|y s′ -y goal |

[0020] p res (s′)=ω net ·p net (u)+ω comp ·p comp (l)

[0021] Where, d man (s′,goal) represents the Manhattan distance, p res (s′) represents the resource penalty, s′ represents the state of the next node reached after the action is executed, goal is the target node, and x s′ y s′ Let x be the coordinates of the next node. goal y goal p represents the coordinates of the target node. res (s′) represents the resource penalty term, p net (u) and p comp (l) represents the network resource penalty function and computational resource penalty function based on Sigmoid.

[0022]

[0023] Where, k net The steepness coefficient of the network penalty function, k comp To calculate the steepness coefficient of the penalty function, θ net The threshold parameter (inflection point position) for network penalty, θ comp The threshold parameter (inflection point position) for calculating the penalty is u, where u is the link utilization and l is the node computational load rate.

[0024] A further improvement of this application is that, in step 2, the exploratory Q value Q exp The calculation method for (s,a) is as follows:

[0025] Step 2.1: Calculate the Q-normalized value of the current state-action pair:

[0026]

[0027] Among them, Q max The maximum Q value in the current Q table;

[0028] Step 2.2: Convert the Q-value into a negative cost term:

[0029] Q exp (s,a)=-Q norm (s,a).

[0030] A further improvement of this application is that, in step 2, α h and α q Adjustments will be made based on the learning progress:

[0031] α h =α h,init ·(1-β)+α h,final ·β

[0032] α q =α q,init ·β

[0033] Where β represents the learning progress, with a value range of [0,1], and α... h,init α h,final and α q,init These are the preset weight parameters.

[0034] A further improvement in this application is that step 4, dynamic weight adaptive adjustment, specifically includes the following steps:

[0035] Step 4.1: Calculate the average link utilization rate of the entire network. Average node computational load across the entire network

[0036]

[0037] Where E is the set of links, V is the set of nodes, and u e It is the utilization rate of the link, l v This refers to the computational load rate of the nodes; Step 4.2: Dynamically adjust the network weights ω net And calculate the weight ω comp The allocation ratio and dynamic weight adjustment rules are as follows:

[0038] when When δ is the load difference threshold parameter:

[0039] like Increase network weights ω net =min(ω) net +Δw,ω max ), and accordingly calculate the weight ω comp =1.0-ω net ;

[0040] like Increase the calculation weight ω comp =min(ω) comp +Δw,ω max ), and correspondingly network weights ω net =1.0-ω comp ;

[0041] when Time: ω net =ω net,base ω comp =ω comp,base ;

[0042] Where Δw is the weight adjustment step size, ω max ω is the upper limit of the weight. net,base and ω comp =ω comp,base These are the baseline weight values ​​under equilibrium conditions;

[0043] Step 4.3: Make secondary adjustments based on traffic type to ensure the reward in Step 3 accurately reflects the current network status: DATA_MINING: ω comp =ω comp,dm ω net =ω net,dm ω dist =ω dist,dm ;

[0044] WEB_SEARCH:ω comp =ω comp,ws ω net =ω net,ws ω dist =ω dist,ws ;

[0045] CACHE_FOLLOWER:ω comp =ω comp,cf ,ω net =ω net,cf ω dist =ω dist,cf ;

[0046] WEB_SERVER:ω dist =ω dist,WS ,

[0047] Where, ω comp,dm For network weights of traffic type DATA_MINING, ω net,dm The weight for calculating the traffic type DATA_MINING, ω dist,dm Distance weight for traffic type DATA_MINING, ω comp,ws For network weights of traffic type WEB_SEARCH, ω net,ws The weight ω is used to calculate the traffic type WEB_SEARCH. dist,ws For the distance weight of the traffic type WEB_SEARCH, ω comp,cf For network weights of traffic type CACHE_FOLLOWER, ω net,cf The weight ω is used to calculate the traffic type CACHE_FOLLOWER. dist,cf Distance weight for traffic type CACHE_FOLLOWER; ω dist,WS Distance weight for traffic type WEB_SERVER.

[0048] A further improvement in this application is that, in step 5, the adaptive exploration rate adjustment mechanism based on the Q-value convergence degree is specifically as follows:

[0049] Step 5.1: Calculate the variance of the Q-value variation:

[0050]

[0051] Where, ΔQ i Let μ be the change in Q value during the i-th update, N be the sampling window size, and μ be the value of Q. ΔQ The mean of the change in Q value; step

[0052] Step 5.2, dynamically adjust the exploration rate for Q-value updates based on the number of state visits:

[0053]

[0054] Where, ∈ min For the minimum exploration rate, ∈ max For the maximum exploration rate, σ th The variance threshold is the variance σ that changes with the Q value. Q Less than the threshold σ th When the exploration rate decreases, the variance σ of the Q value changes. Q Greater than the threshold σ th At that time, increase the exploration rate.

[0055] A further improvement in this application is that: in step 7, during the first stage of guided pre-training n1 rounds, the network state is ideal and the link bandwidth is B. w1 ,min to B w1,maxThe initial load of the node is L. 1,min % to L 1,max With the number of flows set to F1 and the arrival rate λ = λ1, the optimization objective is to establish a basic location-action mapping and learn the basic structure of the network topology.

[0056] In the first sub-stage n2 rounds of the second phase of intelligent pre-training, the network state is characterized by gradually increasing load, and the link bandwidth is B. w2 ,min to B w2,max The initial load of the node is L. 2,min % to L 2,max The number of streams is set to F2, the arrival rate is λ = λ2, and the threshold of the sigmoid smoothing reward function is θ. net From θ net,1 Reduced to θ net,2 θ comp From θ comp,1 Reduced to θ comp,2 The load difference threshold δ is reduced from δ1 to δ2; the optimization goal is to learn a resource-aware routing strategy to adapt to a medium load environment.

[0057] In the second sub-stage n3 rounds of the third-stage intelligent pre-training: the network state is a real load environment, and the link bandwidth is B. w3 ,min to B w3,max The initial load of the node is L. 3,min % to L 3,max The number of streams is set to F3, and the arrival rate λ = λ3; Optimization objective: to verify the effect of experience playback and optimize the quality of long-term decision-making.

[0058] The total number of training rounds n = n1 + n2 + n3, and the final output is the routing decision after the complete optimization process.

[0059] The beneficial effects of this application are:

[0060] (1) The hybrid evaluation function in this application accelerates convergence: by combining A* heuristic information with the Q-Learning value function, the directional guidance provided by the heuristic information is used to accelerate path search, while the experience of Q-value learning is used to optimize decision quality. Experiments show that the hybrid algorithm can achieve 80% performance after 5000 training rounds, while pure Q-Learning requires 15000 rounds, improving the convergence speed by more than 3 times.

[0061] (2) The application’s dynamic weight adaptive improvement of resource balance: The weight configuration is dynamically adjusted according to the real-time monitoring of the network load status, which can effectively cope with the dynamic changes in the network status, avoid the overuse of a certain type of resource, and improve the utilization rate of various computing network resources by 20%-30%, thus achieving a more balanced resource utilization.

[0062] (3) The progressive pre-training of this application enhances robustness. Through three stages of progressive training from simple to complex, this application can gradually adapt to complex network environments, avoiding the learning difficulties caused by direct training in complex environments. It achieves 100% flow completion rate in both normal and congested environments, demonstrating excellent robustness.

[0063] (4) The overall performance of this application is significantly improved. Compared with traditional methods, this application achieves significant improvements in multiple dimensions such as flow completion rate, resource utilization efficiency, and convergence speed. The flow completion rate is improved by 49.2% and 63.6% in normal and congested environments, respectively, which fully verifies the effectiveness of the hybrid strategy of A* heuristic and Q-Learning in the computing-network integrated environment.

[0064] (5) This application has strong differentiated service capabilities. Through dynamic weight adaptation and traffic type identification, it can provide customized routing strategies for different services. Web Server traffic prioritizes low-latency paths, while Data Mining traffic tends to core server nodes with strong computing power, achieving true collaborative optimization of computing network resources. Attached Figure Description

[0065] Figure 1 This is a flowchart of this application.

[0066] Figure 2 This is a structural diagram of the hybrid evaluation function in this application.

[0067] Figure 3 This is a dynamic balance curve of the A* heuristic weight and Q-value weight as the learning progress in this application.

[0068] Figure 4 This is a comparison chart of the network resource reward function and the computational resource reward function based on Sigmoid in this application.

[0069] Figure 5 This is a flowchart of the dynamic weight adaptive adjustment mechanism in this application.

[0070] Figure 6 This is a schematic diagram of the three-stage progressive pre-training process in this application.

[0071] Figure 7 This is the overall architecture diagram of this application. Detailed Implementation

[0072] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0073] like Figure 1-6 As shown, this application presents a path optimization method based on a hybrid A* and Q-Learning approach, which specifically includes the following steps:

[0074] Step 1: Obtain initial data of the computing network environment. This initial data includes network topology, node computing resource status, link bandwidth resources, and traffic information to be transmitted. Based on this initial data, construct a four-dimensional extended state space containing node location, network load level, computing load level, and traffic type. The constructed four-dimensional extended state space is as follows:

[0075] s=(v,λ n ,λ c ,τ)∈S

[0076] Where τ∈T, T={WebServer,CacheFpllower,WebSearch,DataMining}, v represents the current node, v∈V, λ n Indicates the network load level, λ n ∈{0,1,2},λ c Indicates the calculation load level, λ c ∈{0,1,2}, where τ represents the flow type.

[0077] The mapping function between network load level and compute load level is:

[0078]

[0079] Where x is the load factor, θ low For low-load level threshold parameters, θ high This is a threshold parameter for high load levels. It is typically set to a value of θ. low =0.3, θ high =0.7.

[0080] Step 2: Based on the four-dimensional extended state space constructed in Step 1, design a hybrid evaluation function combining A* heuristic search and Q-Learning:

[0081] f(s,a)=g(s+α h ·h(s,a)+α q ·Q exp (s,a)

[0082] Where g(s) is the actual cumulative cost from the starting point to the current node, h(s,a) is the heuristic estimation function from the current node to the target, and Q... exp (s,a) represents the exploratory Q value, α h and α q These are dynamic weighting coefficients.

[0083] In this step, the heuristic estimation function h(s,a) uses a combination of Manhattan distance and resource penalty:

[0084] h(s,a)=d man (s′,goal)+p res (s′)

[0085] d man (s′,goal)=|x s′ -x goal |+|y s′ -y goal |

[0086] p res (s′)=ω net ·p net (u)+ω comp ·p comp (l)

[0087] Where, d man (s′,goal) represents the Manhattan distance, p res (s′) represents the resource penalty, s′ represents the state of the next node reached after the action is executed, goal is the target node, and x s′ y s′ Let x be the coordinates of the next node. goal y goal p represents the coordinates of the target node. res (s′) represents the resource penalty term, p net (u) and p comp (l) represents the network resource penalty function and computational resource penalty function based on Sigmoid:

[0088]

[0089] Where, k net The steepness coefficient of the network penalty function, k comp To calculate the steepness coefficient of the penalty function, θ net The threshold parameter (inflection point position) for network penalty, θ comp The threshold parameter (inflection point position) for calculating the penalty is u, where u is the link utilization and l is the node computational load rate.

[0090] α h and α q Adjustments will be made based on the learning progress:

[0091] α h =α h,init ·(1-β)+α h,final ·β

[0092] α q =α q,init ·β

[0093] Where β represents the learning progress, with a value range of [0,1], and α... h,init α h,final and α q,init These are the preset weight parameters.

[0094] Exploratory Q-value Q exp The calculation method for (s,a) is as follows:

[0095] Step 2.1: Calculate the Q-normalized value of the current state-action pair:

[0096]

[0097] Among them, Q max The maximum Q value in the current Q table;

[0098] Step 2.2: Convert the Q-value into a negative cost term:

[0099] Q exp (s,a)=-Q norm (s,a).

[0100] Step 3: Design a Sigmoid smoothing reward function to perform a nonlinear mapping on network resources and computing resources, generating network resource rewards and computing resource rewards respectively, which are used for reward calculation of the hybrid evaluation function in Step 2;

[0101] Network resource rewards:

[0102]

[0103] Computing resource rewards:

[0104]

[0105] Where u is the link utilization rate, l is the node computing resource load rate, and A n A c k is the reward magnitude parameter. n k c Here, ρ0 and L0 are the curve steepness parameters, and B is the inflection point parameter. n B c This is the offset parameter.

[0106] Step 4: Based on the network resource reward and computing resource reward calculated in Step 3, implement dynamic weight adaptive adjustment, according to the real-time monitored average link utilization of the entire network. Average node computing load across the entire network Dynamically adjust network weights ωnet Calculate the weight ω comp and distance weight ω dist The allocation ratio. In this step, the dynamic weight adaptive adjustment specifically includes the following steps:

[0107] Step 4.1: Calculate the average link utilization rate of the entire network. Average node computational load across the entire network

[0108]

[0109] Where E is the set of links, V is the set of nodes, and u e It is the utilization rate of the link, l v It is the computational load rate of the node;

[0110] Step 4.2: Dynamically adjust network weights ω net And calculate the weight ω comp The allocation ratio and dynamic weight adjustment rules are as follows:

[0111] when When δ is the load difference threshold parameter:

[0112] like Increase network weights ω net =min(ω) net +Δw,ω max ), and accordingly calculate the weight ω comp =1.0-ω net ;

[0113] like Increase the calculation weight ω comp =min(ω) comp +Δw,ω max ), and correspondingly network weights ω net =1.0-ω comp ;

[0114] when Time: ω net =ω net,base ω comp =ω comp,base ;

[0115] Where Δw is the weight adjustment step size, ω max ω is the upper limit of the weight. net,base and ω comp =ω comp,base These are the baseline weight values ​​under equilibrium conditions;

[0116] Step 4.3: Make secondary adjustments based on traffic type to ensure the reward in Step 3 accurately reflects the current network status: DATA_MINING: ω comp =ω comp,dm ω net =ω net,dm ω dist =ω dist,dm ;

[0117] WEB_SEARCH:ω comp =ω comp,ws ω net =ω net,ws ω dist =ω dist,ws ;

[0118] CACHE_FOLLOWER:ω comp =ω comp,cf ,ω net =ω net,cf ω dist =ω dist,cf ;

[0119] WEB_SERVER:ω dist =ω dist,WS ,

[0120] Where, ω comp,dm For network weights of traffic type DATA_MINING, ω net,dm The weight for calculating the traffic type DATA_MINING, ω dist,dm Distance weight for traffic type DATA_MINING, ω comp,ws For network weights of traffic type WEB_SEARCH, ω net,ws The weight ω is used to calculate the traffic type WEB_SEARCH. dist,ws For the distance weight of the traffic type WEB_SEARCH, ω comp,cf For network weights of traffic type CACHE_FOLLOWER, ω net,cf The weight ω is used to calculate the traffic type CACHE_FOLLOWER. dist,cf Distance weight for traffic type CACHE_FOLLOWER; ω dist,WS Distance weight for traffic type WEB_SERVER.

[0121] Step 5: Use the network weights ω determined in Step 4 net Calculate the weight ω comp and distance weight ω distConfigure and design an adaptive exploration rate adjustment mechanism based on Q-value convergence. This mechanism dynamically adjusts the learning rate for Q-value updates based on the number of state accesses, achieving stable updates to the Q-table. Specifically, the adaptive exploration rate adjustment mechanism based on Q-value convergence is as follows:

[0122] Step 5.1: Calculate the variance of the Q-value variation:

[0123]

[0124] Where, ΔQ i Let μ be the change in Q value during the i-th update, N be the sampling window size, and μ be the value of Q. ΔQ This represents the mean of the changes in the Q value;

[0125] Step 5.2: Dynamically adjust the exploration rate for Q-value updates based on the number of state visits:

[0126]

[0127] Where, ∈ min For the minimum exploration rate, ∈ max For the maximum exploration rate, σ th The variance threshold is the variance σ that changes with the Q value. Q Less than the threshold σ th When the exploration rate decreases, the variance σ of the Q value changes. Q Greater than the threshold σ th At that time, increase the exploration rate.

[0128] Step 6: Based on the Q-table updated in Step 5, construct an experience replay mechanism. After the stream transmission is completed, calculate the performance reward based on the ratio of the actual transmission time to the expected transmission time, and backpropagate to update the Q-value of all state-action pairs on the path.

[0129] In this step, the experience replay mechanism includes:

[0130] Performance ratio calculation:

[0131]

[0132] Where expectedTime is the expected transmission time and actualTime is the actual transmission time;

[0133] Performance bonus settings:

[0134] If the performance ratio η>η high : bonus = b high

[0135] If η low ≤Performance Ratio≤η high : bonus = bmid

[0136] If performance ratio < η low : bonus = b low

[0137] Where η low For low performance threshold, η high For high performance threshold, b high For higher reward values, b mid For the medium reward value, b low For low reward values, the Q-values ​​of each state-action pair on the path are updated backward using the adaptive learning rate based on access frequency from step 5.

[0138] Backpropagation update: Traverse backwards from the end of the path, adjust the reward value according to the position decay factor and update the Q table:

[0139] bonus adjusted (i) = bonus·(1-i / path) length )

[0140] Where i is the index position of the node in the path, and path length This represents the total path length.

[0141] Step 7: Repeat steps 1 to 6 to implement the progressive pre-training strategy. The progressive strategy includes the first stage of guided pre-training (n1 rounds), the second stage of intelligent pre-training (first sub-stage n2 rounds), and the third stage of intelligent pre-training (second sub-stage n3 rounds). By training the algorithm of this application in stages, it gradually adapts to the complex network environment and finally outputs the optimized routing path.

[0142] In the first phase of guided pre-training (n1 rounds), the network is in an ideal state with a link bandwidth of B. w1 ,min to B w1,max The initial load of the node is L. 1,min % to L 1,max With the number of flows set to F1 and the arrival rate λ = λ1, the optimization objective is to establish a basic location-action mapping and learn the basic structure of the network topology.

[0143] In the first sub-stage n2 rounds of the second phase of intelligent pre-training, the network state is characterized by gradually increasing load, and the link bandwidth is B. w2 ,min to B w2,max The initial load of the node is L. 2,min % to L 2,max The number of streams is set to F2, the arrival rate is λ = λ2, and the threshold of the sigmoid smoothing reward function is θ. net From θ net,1 Reduced to θ net,2 θcomp From θ comp,1 Reduced to θ comp,2 The load difference threshold δ is reduced from δ1 to δ2; the optimization goal is to learn a resource-aware routing strategy to adapt to a medium load environment.

[0144] In the second sub-stage n3 rounds of the third-stage intelligent pre-training: the network state is a real load environment, and the link bandwidth is B. w3 ,min to B w3,max The initial load of the node is L. 3,min % to L 3,max The number of streams is set to F3, and the arrival rate λ = λ3; Optimization objective: to verify the effect of experience playback and optimize the quality of long-term decision-making.

[0145] The total number of training rounds n = n1 + n2 + n3, and the final output is the routing decision after the complete optimization process.

[0146] To verify this application, a 20×20 mesh topology was constructed, containing 400 nodes, and 40 heterogeneous server nodes were deployed. The path optimization method is as follows:

[0147] Step 1: Initialize the extended state space of the computing network convergence. Each state contains four dimensions: node ID (400 possible values), network load level (3 levels), computing load level (3 levels), and traffic type (4 types). The total state space size is 14400.

[0148] Step 2: Initialize the hybrid optimization algorithm parameters: Set the initial learning rate α = 0.7, discount factor γ = 0.95, and initial exploration rate ∈ = 0.25. Design a hybrid evaluation function with an initial Q-value weight of 0.6, which increases linearly to 0.8 as the learning progresses.

[0149] Step 3: Implement the Sigmoid smooth reward mechanism. The network resource reward sets an inflection point at 70% link utilization, and the computing resource reward begins to decay rapidly at 60% load, ensuring a safety margin for resource utilization.

[0150] Step 4: Deploy a dynamic weight adaptive mechanism. After every 100 routing decisions, reassess the overall network load and adjust weights based on load differences. Pre-configure differentiated weights for the four traffic types.

[0151] Step 5: Implement adaptive learning rate adjustment, maintaining a high learning rate (0.63) for the first 50,000 updates, gradually decreasing it in the middle 150,000 updates, and stabilizing at 0.35 in the later stages. Simultaneously maintain the state access counter and Q-value variance record.

[0152] Step 6: Deploy an experience replay mechanism to calculate the actual transmission time for each completed stream, compare it with the expected time based on the bottleneck bandwidth, generate performance rewards, and update the Q-values ​​of all state-action pairs on the path in reverse.

[0153] Step 7: Perform 20,000 rounds of pre-training: train for the first 6,000 rounds with a 5% load, gradually increase the load to 45% for the middle 8,000 rounds, and simulate the real network environment for the last 6,000 rounds.

[0154] Step 8: Generate 250 test streams using the Poisson arrival process (λ = 0.5), with the stream size randomly generated within the range of 10KB-5MB based on the type.

[0155] Step 9: Perform route optimization: Use a hybrid algorithm to find the optimal path for each flow. A* provides initial guidance, Q-Learning provides empirical optimization, and the network status and Q table are updated in real time.

[0156] Step 10, Performance Evaluation and Statistics: Record the completion time, transmission path, throughput and other indicators for each stream, and calculate the overall success rate, average transmission time, resource utilization and other performance indicators.

[0157] Experimental results show that the method of this invention achieves a 100% flow completion rate under normal conditions, with an average transmission time of 0.093 seconds and an average throughput of 102.75 Mbps. Maintaining a 100% completion rate even under heavy congestion conditions fully verifies the effectiveness and robustness of the algorithm.

[0158] In summary, this application innovatively integrates the prior knowledge of A* heuristic search with the adaptive learning capability of Q-Learning, combined with a sigmoid smoothing reward and dynamic weight adjustment mechanism, to achieve efficient intelligent routing in a computing-network integrated environment. This method significantly improves convergence speed and routing quality while maintaining the algorithm's adaptive capability, providing effective technical support for the intelligent orchestration of future computing networks.

[0159] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A computational-network integrated path optimization method based on a hybrid A* and Q-Learning approach, characterized in that: The path optimization method specifically includes the following steps: Step 1: Obtain the initial data of the computing network environment. The initial data includes the network topology, node computing resource status, link bandwidth resources, and traffic information to be transmitted. Based on the initial data, construct a four-dimensional extended state space that includes node location, network load level, computing load level, and traffic type. Step 2: Based on the four-dimensional extended state space constructed in Step 1, design a hybrid evaluation function combining A* heuristic search and Q-Learning: f(s,a)=g(s)+α h ·h(s,a)+α q ·Q exp (s,a) Where g(s) is the actual cumulative cost from the starting point to the current node, h(s,a) is the heuristic estimation function from the current node to the target, and Q... exp (s,a) represents the exploratory Q value, α h and α q These are dynamic weighting coefficients; Step 3: Design a sigmoid smoothing reward function to perform a nonlinear mapping on network resources and computing resources, and generate network resource rewards and computing resource rewards respectively. Step 4: Based on the network resource reward and computing resource reward calculated in Step 3, implement dynamic weight adaptive adjustment, according to the real-time monitored average link utilization of the entire network. Average node computational load across the entire network Dynamically adjust network weights ω net Calculate the weight ω comp and distance weight ω dist The allocation ratio; Step 5: Use the network weights ω determined in Step 4 net Calculate the weight ω comp and distance weight ω dist Configure and design an adaptive exploration rate adjustment mechanism based on the convergence degree of Q-value, dynamically adjust the learning rate of Q-value updates according to the number of state accesses, and achieve stable updates of the Q-table; Step 6: Based on the Q-table updated in Step 5, construct an experience replay mechanism. After the stream transmission is completed, calculate the performance reward based on the ratio of the actual transmission time to the expected transmission time, and backpropagate to update the Q-value of all state-action pairs on the path. Step 7: Repeat steps 1 to 6 to implement the progressive pre-training strategy. The progressive strategy includes the first stage of guided pre-training (n1 rounds), the second stage of intelligent pre-training (first sub-stage n2 rounds), and the third stage of intelligent pre-training (second sub-stage n3 rounds). Through phased training, the optimized routing path is finally output.

2. The integrated computation-network path optimization method based on a hybrid A* and Q-Learning method according to claim 1, characterized in that: In step 2, the heuristic estimation function h(s,a) uses a combination of Manhattan distance and resource penalty: h(s,a)=d man (s′,goal)+p res (s′) d man (s′,goal)=|x s′ -x goal |+|y s′ -y goal | p res (s′)=ω net ·p net (u)+ω comp ·p comp (l) Where, d man (s′,goal) represents the Manhattan distance, p res (s′) represents the resource penalty, s′ represents the state of the next node reached after the action is executed, goal is the target node, and x s′ y s′ Let x be the coordinates of the next node. goal y goal p represents the coordinates of the target node. res (s′) represents the resource penalty term, p net (u) and p comp (l) represents the network resource penalty function and computational resource penalty function based on the Sigmoid smooth reward function: Where, k net The steepness coefficient of the network penalty function, k comp To calculate the steepness coefficient of the penalty function, θ net The threshold parameter for network penalty, θ comp The threshold parameter for calculating the penalty is u, where u is the link utilization and l is the node computational load rate.

3. The integrated computation-network path optimization method based on a hybrid A* and Q-Learning method according to claim 2, characterized in that: In step 2, the exploratory Q value Q exp The calculation method for (s,a) is as follows: Step 2.1: Calculate the Q-normalized value of the current state-action pair: Among them, Q max The maximum Q value in the current Q table; Step 2.2: Convert the Q-value into a negative cost term: Q exp (s,a)=-Q norm (s,a)。 4. The integrated computation-network path optimization method based on a hybrid A* and Q-Learning method according to claim 3, characterized in that: In step 2, α h and α q Adjustments will be made based on the learning progress: α h ＝α h,init ·(1-β)+α h,final ·β α q ＝α q,init ·β Where β represents the learning progress, with a value range of [0,1], and α... h,init α h,final and α q,init These are the preset weight parameters.

5. The integrated computation-network path optimization method based on a hybrid A* and Q-Learning method according to claim 1, characterized in that: Step 4, dynamic weight adaptive adjustment, specifically includes the following steps: Step 4.1: Calculate the average link utilization rate of the entire network. Average node computational load across the entire network Where E is the set of links, V is the set of nodes, and u e It is the utilization rate of the link, l v It is the computational load rate of the node; Step 4.2: Dynamically adjust network weights ω net And calculate the weight ω comp The allocation ratio and dynamic weight adjustment rules are as follows: when When δ is the load difference threshold parameter: like Increase network weights ω net =min(ω) net +Δw,ω max ), and accordingly calculate the weight ω comp =1.0-ω net ; like Increase the calculation weight ω comp =min(ω) comp +Δw,ω max ), and correspondingly network weights ω net =1.0-ω comp ; When : ω net = ω net,base , ω comp = ω comp,base ; Where Δw is the weight adjustment step size, ω max ω is the upper limit of the weight. net,base and ω comp =ω comp,base These are the baseline weight values ​​under equilibrium conditions; Step 4.3: Make secondary adjustments based on traffic type: DATA_MINING: oh comp =ω comp,dm Oh, oh net =ω net,dm Oh, oh dist =ω dist,dm ; WEB_SEARCH: oh comp =ω comp,ws Oh, oh net =ω net,ws Oh, oh dist =ω dist,ws ; CACHE_FOLLOWER:oh comp =ω comp,cf ,oh net =ω net,cf Oh, oh dist =ω dist,cf ; WEB_SERVER: oh dist =ω dist,WS , Where, ω comp,dm For network weights of traffic type DATA_MINING, ω net,dm The weight for calculating the traffic type DATA_MINING, ω dist,dm Distance weight for traffic type DATA_MINING, ω comp,ws For network weights of traffic type WEB_SEARCH, ω net,ws The weight ω is used to calculate the weight of the traffic type WEB_SEARCH. dist,ws For the distance weight of the traffic type WEB_SEARCH, ω comp,cf For network weights of traffic type CACHE_FOLLOWER, ω net,cf The weight ω is used to calculate the traffic type CACHE_FOLLOWER. dist,cf Distance weight for traffic type CACHE_FOLLOWER; ω dist,WS Distance weight for traffic type WEB_SERVER.

6. The integrated computation-network path optimization method based on a hybrid A* and Q-Learning method according to claim 1, characterized in that: In step 5, the adaptive exploration rate adjustment mechanism based on the Q-value convergence is specifically as follows: Step 5.1: Calculate the variance of the Q-value variation: Where, ΔQ i Let μ be the change in Q value during the i-th update, N be the sampling window size, and μ be the value of Q. ΔQ This represents the mean of the changes in the Q value; Step 5.2: Dynamically adjust the exploration rate for Q-value updates based on the number of state visits: Where, ∈ min For the minimum exploration rate, ∈ max For the maximum exploration rate, σ th The variance threshold is the variance σ that changes with the Q value. Q Less than the threshold σ th When the exploration rate is reduced, the variance h of the Q-value changes. Q Greater than the threshold σ th At that time, increase the exploration rate.

7. The integrated computation-network path optimization method based on a hybrid A* and Q-Learning method according to claim 1, characterized in that: In step 7, during the first phase of guided pre-training (n1 rounds), the network state is ideal, and the link bandwidth is B. w1 ,min to B w1,max The initial load of the node is L. 1,min % to L 1,max The number of streams is set to F1, and the arrival rate λ = λ1; In the first sub-stage n2 rounds of the second phase of intelligent pre-training, The network is experiencing a gradual increase in load, with a link bandwidth of B. w2 ,min to B w2,max The initial load of the node is L. 2,min % to L 2,max The number of streams is set to F2, the arrival rate is λ = λ2, and the threshold of the sigmoid smoothing reward function is θ. net From θ net,1 Reduced to θ net,2 θ comp From θ comp,1 Reduced to θ comp,2 The load difference threshold δ decreases from δ1 to δ2; In the second sub-stage n3 rounds of the third-stage intelligent pre-training: the network state is a real load environment, and the link bandwidth is B. w3 ,min to B w3,max The initial load of the node is L. 3,min % to L 3,max The number of streams is set to F3, and the arrival rate λ = λ3; The total number of training rounds n = n1 + n2 + n3, and the final output is the routing decision after the complete optimization process.

Images