A proximal strategy optimization method and system for heterogeneous network computing power control

Abstract

A proximal strategy optimization method and system for heterogeneous network computing power control, comprising: setting initial parameters of a behavior network and an evaluation network based on a heterogeneous network computing power control scenario, constructing an experience buffer for storing training data, loading trajectory data of heterogeneous network terminals, setting computing parameters for each heterogeneous network terminal, and adjusting clipping values ​​using a linear attenuation rule; generating a strategy based on input current state data by the behavior network, outputting an action with the highest probability as a computing power allocation plan, and constructing an objective function of the behavior network with the goal of maximizing cumulative rewards; utilizing an evaluation network to evaluate the value of actions executed in the allocation plan, and constructing an objective function of the evaluation network with the goal of minimizing the prediction error of the state value function; executing the plan on the heterogeneous network terminal according to the computing power allocation plan, calculating rewards based on energy consumption, execution time, and plan quality after the plan is completed, and storing the training data in the experience buffer.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning technology, and more specifically, relates to a proximal strategy optimization method and system for heterogeneous network computing power control. Background Art

[0002] Managing and allocating computing resources in heterogeneous network environments is a complex technical challenge. Because different computing nodes vary significantly in performance, energy consumption, and solution processing capabilities, and different solutions have varying computing resource requirements, existing resource allocation methods typically employ static rules or heuristic algorithms. However, these methods lack dynamic adaptability and struggle to cope with real-time changes in solution requirements and device status in heterogeneous networks. This results in low resource utilization and an inability to efficiently process diverse solutions in complex scenarios.

[0003] In recent years, reinforcement learning has been increasingly applied to the scheduling and allocation of computing resources, but its application in heterogeneous networks still faces numerous bottlenecks. For one thing, the policies generated by reinforcement learning can fluctuate, leading to unstable computing allocation schemes. Furthermore, due to the complexity of the state and action spaces in heterogeneous networks, existing reinforcement learning models lack accuracy in assessing the value of allocation schemes, making effective optimization difficult. Furthermore, existing methods lack constraints during policy updates, which can cause policies to deviate significantly from the optimal solution, reducing the reliability and efficiency of resource allocation.

[0004] Furthermore, the complexity of computing power control is further exacerbated by the need to balance real-time performance with energy efficiency. Solutions often have strict latency requirements, but energy conservation also requires limiting excessive use of computing resources. Existing technologies struggle to balance these two objectives. Over-focusing on real-time performance can lead to redundant resource allocation, while over-emphasizing energy efficiency can delay solution completion. Consequently, the lack of a unified optimization framework that effectively addresses this conflict between real-time performance and energy efficiency has become a pressing issue for current technologies. Summary of the Invention

[0005] In order to address the deficiencies in the prior art, the present invention provides a proximal strategy optimization method for heterogeneous network computing power control.

[0006] The present invention adopts the following technical solutions.

[0007] A first aspect of the present invention provides a proximal strategy optimization method for heterogeneous network computing power control, wherein the heterogeneous network includes a behavior network, an evaluation network, an experience buffer, and a terminal; the behavior network is used to generate an action probability distribution and output a computing power allocation plan; the evaluation network is used to evaluate the computing power allocation plan based on state data corresponding to actions in the computing power allocation plan; the experience buffer is used to store state, action, and reward data; the heterogeneous network terminal is used to execute the computing power allocation plan and record the operating status after the execution of the computing power allocation plan; the operating status includes resource usage, energy consumption, and completion status information during the execution of the computing power allocation plan; and the method includes the following steps:

[0008] The behavioral network inputs the trajectory data of heterogeneous network terminals, including the CPU frequency range, energy consumption curve change data and the maximum training time step;

[0009] According to the maximum training time step and the current training time step, the linear attenuation method is used to determine the cropping range;

[0010] The objective function of the behavior network is to maximize the cumulative reward, and the clipping range is used as the constraint condition of the objective function of the behavior network. The behavior network outputs the action probability distribution under the new strategy based on the input state data;

[0011] The action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan. The evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function to evaluate the value of the completion state corresponding to the computing power allocation plan.

[0012] During the execution of the computing power allocation plan, the heterogeneous network terminals obtain resource usage, energy consumption, and completion status information. The behavioral network determines the reward for the computing power allocation plan based on resource usage, energy consumption, completion status information, and the value evaluation results of the completion status.

[0013] The computing power allocation plan and its rewards are stored in the experience buffer, which is used to update the policy parameters of the behavior network and the state value function parameters of the evaluation network to generate a new policy and evaluate the state value of the computing power allocation plan.

[0014] Preferably, the behavior network is initialized using a multi-layer perceptron architecture, with the current state data as input; the evaluation network is initialized using a convolutional neural network architecture, and evaluates the computing power allocation plan based on the state data corresponding to the actions in the computing power allocation plan;

[0015] Trajectory data also includes historical CPU loads of heterogeneous network terminals, completion records of computing power allocation plans, energy consumption data, and device status information. Trajectory data is stored in the experience buffer in the form of time series;

[0016] The experience buffer adopts a fixed-capacity circular storage strategy.

[0017] Preferably, the clipping range is determined using a linear attenuation method based on the maximum training time step and the current training time step, including:

[0018] Set the maximum training time step T and the current training time step t to dynamically generate the cropping range as follows:

[0019]

[0020] Where T represents the maximum training time step, t represents the current training time step; ∈0 represents the initial clipping value, Indicates the clipping range.

[0021] Preferably, the objective function of the behavior network is calculated as follows:

[0022]

[0023] Where, L(s,a,θ old ,θ) is the objective function of the behavior network, a represents action, s represents state, θ represents the new strategy parameter, θ old Represents the old policy parameters; represents the old policy θ old The improvement of action a in state s compared with the average action; clip(·,·,·) is the clipping function; π θ (a|s) represents the probability of executing action a in state s under the new strategy θ; represents the old policy θ old The probability of executing action a in state s; is a hyperparameter, which indicates the maximum deviation allowed between the new strategy and the old strategy.

[0024] Preferably, the clipping function satisfies the following relationship:

[0025]

[0026] The clipping range represented by the clipping function is used as the constraint condition of the objective function of the behavior network.

[0027] Preferably, the action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan; the evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function to evaluate the value of the completion state corresponding to the computing power allocation plan; including:

[0028] Extract the state data of the current time step from the experience buffer. The state data includes the terminal CPU load, solution execution records, and resource usage. The extracted data is normalized and then input into the behavior network.

[0029] The behavior network generates an action probability distribution. Each action corresponds to a computing power allocation scheme. The action with the highest probability is selected as the final computing power allocation scheme based on the probability distribution.

[0030] The evaluation network takes the current state data and the actions generated by the behavior network as input and calculates the state value function. The evaluation network parameters are optimized by comparing the predicted state value of the evaluation network with the accumulated reward after the plan is actually completed and minimizing the error between the two.

[0031] The behavior network and the evaluation network collaborate to optimize. The evaluation network rewards the state data corresponding to the evaluation action according to the final computing power allocation plan, which is used to update the strategy generation of the behavior network.

[0032] Preferably, the objective function of the evaluation network is calculated as follows:

[0033] L(v)=(r k +γV v (s k+1 )-V v (s k )) 2

[0034]

[0035] Where L(v) represents the objective function of the evaluation network; r k represents the immediate reward at time step k; γ represents the discount factor, which is used to balance the weights of current rewards and future rewards; s k represents the state of time step k; V v (s k ) represents the state value function of the current time step k predicted by the evaluation network; V v (s k+1 ) represents the state value function of the time step k+1 predicted by the evaluation network; L(θ) is the objective function of the behavior strategy network; π θ (a k |s k ,h k-1 ) is the new strategy θ, given state s k and historical information k-1 When you select action a k probability; Under the old policy θ, given state s k and historical information k-1 When you select action a k The probability of a k is the action of time step k, s k is the state of time step k, h k-1 For historical information; is the advantage function, used to measure the k Select action a k the expected advantage over the new strategy; represents the old policy θ old The state s at the next time step k k -Action a k The value function of is the expected cumulative reward obtained after performing the action.

[0036] Preferably, the updated behavior network generation strategy is calculated as follows:

[0037]

[0038] Where θ new Represents the new strategy parameters after optimization; represents the strategy to maximize the objective function; Represents the old strategy θ old The expected value of the generated state s and action a; θ represents the new policy parameters; θ old represents the old policy parameters; a represents the action; s represents the state.

[0039] Preferably, the rewards for determining the computing power distribution plan include:

[0040] Based on the computing power allocation plan generated by the behavioral network, the plan is executed on the heterogeneous network terminals, and the heterogeneous network terminals dynamically adjust the CPU frequency and memory allocation according to the computing power allocation plan;

[0041] During the execution of the plan, resource usage, energy consumption, and completion status information are recorded. Resource usage includes CPU usage, memory usage, and network bandwidth. Energy consumption data is collected in real time through the terminal's built-in energy consumption monitoring module. Completion status information includes whether the plan was successfully completed, whether the execution time exceeded the predetermined threshold, and whether the calculation results met the accuracy requirements.

[0042] The reward value is calculated based on the solution execution time, energy consumption and completion status. The reward function is as follows:

[0043]

[0044] Where D is the execution time of the solution, E is the energy consumption of the solution, Q is the quality score of the solution completion status, and w1, w2, and w3 are weight factors;

[0045] The state, action, and reward data after task execution are stored in the experience buffer. The data is stored in the key-value pair format of [state, action, reward] and sorted by timestamp. The experience buffer adopts a fixed-capacity circular storage mechanism. When the storage capacity reaches the upper limit, the earliest data is overwritten.

[0046] A second aspect of the present invention provides a proximal policy optimization system for heterogeneous network computing power control, comprising: an initialization module, a clipping range construction module, a policy generation and optimization module, and a scheme execution and reward update module;

[0047] The initialization module is used to set the initial parameters of the behavior network and evaluation network according to the heterogeneous network computing power control scenario, build an experience buffer for storing states, actions, and rewards, load the trajectory data of the heterogeneous network terminals, and set the computing parameters of each heterogeneous network terminal;

[0048] The clipping range construction module is used to calculate the clipping range based on the set maximum training time step and the current training time step, and adjust the clipping value through the linear decay rule to constrain the deviation between the new policy generated by the behavior network and the old policy;

[0049] The strategy generation and optimization module is used to extract the current state data from the experience buffer and input it into the behavior network. The behavior network generates a strategy based on the input current state, outputs the action probability distribution under the current state, selects the action with the highest probability as the computing power allocation plan, and constructs the objective function of the behavior network based on the clipping range. At the same time, the evaluation network is used to evaluate the value of the plan allocation plan, and the objective function of the evaluation network is constructed by minimizing the state value prediction error.

[0050] The plan execution and reward update module is used to execute the plan on the heterogeneous network terminal according to the computing power allocation plan generated by the behavior network, calculate the reward based on the energy consumption, execution time and plan quality after the plan is completed, and store the current state, selected action, calculated reward value and next state data after the plan is executed in the experience buffer to optimize the behavior network and evaluation network.

[0051] The beneficial effects of the present invention are as follows: computing power allocation strategies are dynamically generated through behavioral networks, and the stability of strategy updates is guaranteed by combining clipping range constraints. At the same time, the value of allocation schemes is accurately evaluated using evaluation networks, and the strategy generation rules of behavioral networks are optimized, thereby achieving efficient computing power resource allocation in heterogeneous network environments. In addition, the reward function comprehensively considers the execution time, resource consumption, and completion quality of the scheme, thereby resolving the contradiction between real-time performance and energy consumption optimization, and further improving the utilization efficiency of computing power resources and the reliability of scheme completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 1 is a graph showing comparative experimental results of five algorithms provided in accordance with an embodiment of the present invention;

[0053] Figure 2 It is a schematic diagram of a heterogeneous network computing power optimization architecture provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. The embodiments described in this application are only part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without making creative efforts are all within the scope of protection of the present invention.

[0055] Example 1 of the present invention provides a proximal strategy optimization method for heterogeneous network computing power control, wherein the heterogeneous network includes a behavior network, an evaluation network, an experience buffer, and a terminal; the behavior network is used to generate an action probability distribution and output a computing power allocation plan; the evaluation network is used to evaluate the computing power allocation plan based on the state data corresponding to the actions in the computing power allocation plan; the experience buffer is used to store state, action, and reward data; the heterogeneous network terminal is used to execute the computing power allocation plan and record the operating status after the execution of the computing power allocation plan; the operating status includes resource usage, energy consumption, and completion status information during the execution of the computing power allocation plan; and the method includes the following steps:

[0056] The behavioral network inputs the trajectory data of heterogeneous network terminals, including the CPU frequency range, energy consumption curve change data and the maximum training time step;

[0057] According to the maximum training time step and the current training time step, the linear attenuation method is used to determine the cropping range;

[0058] Preferably, the clipping range is determined using a linear attenuation method based on the maximum training time step and the current training time step, including:

[0059] Set the maximum training time step T and the current training time step t to dynamically generate the cropping range as follows:

[0060]

[0061] Where T represents the maximum training time step, t represents the current training time step; ∈0 represents the initial clipping value, Indicates the clipping range.

[0062] The objective function of the behavior network is to maximize the cumulative reward, and the clipping range is used as the constraint condition of the objective function of the behavior network. The behavior network outputs the action probability distribution under the new strategy based on the input state data;

[0063] Preferably, the objective function of the behavior network is calculated as follows:

[0064]

[0065] Where, L(s,a,θold ,θ) is the objective function of the behavior network, a represents action, s represents state, θ represents the new strategy parameter, θ old Represents the old policy parameters; represents the old policy θ old The improvement of action a in state s compared with the average action; clip(·,·,·) is the clipping function; π θ (a|s) represents the probability of executing action a in state s under the new strategy θ; represents the old policy θ old The probability of executing action a in state s; is a hyperparameter, which indicates the maximum deviation allowed between the new strategy and the old strategy.

[0066] Preferably, the clipping function satisfies the following relationship:

[0067]

[0068] The clipping range represented by the clipping function is used as the constraint condition of the objective function of the behavior network.

[0069] The action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan. The evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function to evaluate the value of the completion state corresponding to the computing power allocation plan.

[0070] Preferably, the action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan; the evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function to evaluate the value of the completion state corresponding to the computing power allocation plan; including:

[0071] Extract the state data of the current time step from the experience buffer. The state data includes the terminal CPU load, solution execution records, and resource usage. The extracted data is normalized and then input into the behavior network.

[0072] The behavior network generates an action probability distribution. Each action corresponds to a computing power allocation scheme. The action with the highest probability is selected as the final computing power allocation scheme based on the probability distribution.

[0073] The evaluation network takes the current state data and the actions generated by the behavior network as input and calculates the state value function. The evaluation network parameters are optimized by comparing the predicted state value of the evaluation network with the accumulated reward after the plan is actually completed and minimizing the error between the two.

[0074] The behavior network and the evaluation network collaborate to optimize. The evaluation network rewards the state data corresponding to the evaluation action according to the final computing power allocation plan, which is used to update the strategy generation of the behavior network.

[0075] Preferably, the objective function of the evaluation network is calculated as follows:

[0076] L(v)=(r k +γV v (s k+1 )-V v (s k )) 2

[0077]

[0078] Where L(v) represents the objective function of the evaluation network; r k represents the immediate reward at time step k; γ represents the discount factor, which is used to balance the weights of current rewards and future rewards; s k represents the state of time step k; V v (s k ) represents the state value function of the current time step k predicted by the evaluation network; V v (s k+1 ) represents the state value function of the time step k+1 predicted by the evaluation network; L(θ) is the objective function of the behavior strategy network; π θ (a k |s k ,h k-1 ) is the new strategy θ, given state s k and historical information k-1 When you select action a k probability; Under the old policy θ, given state s k and historical information k-1 When you select action a k The probability of a k is the action of time step k, s k is the state of time step k, h k-1 For historical information; is the advantage function, used to measure the k Select action a k the expected advantage over the new strategy; represents the old policy θ old The state s at the next time step k k -Action a k The value function of is the expected cumulative reward obtained after performing the action.

[0079] Preferably, the updated behavior network generation strategy is calculated as follows:

[0080]

[0081] Where θnew Represents the new strategy parameters after optimization; represents the strategy to maximize the objective function; Represents the old strategy θ old The expected value of the generated state s and action a; θ represents the new policy parameters; θ old represents the old policy parameters; a represents the action; s represents the state.

[0082] During the execution of the computing power allocation plan, the heterogeneous network terminals obtain resource usage, energy consumption, and completion status information. The behavioral network determines the reward for the computing power allocation plan based on resource usage, energy consumption, completion status information, and the value evaluation results of the completion status.

[0083] Preferably, the rewards for determining the computing power distribution plan include:

[0084] Based on the computing power allocation plan generated by the behavioral network, the plan is executed on the heterogeneous network terminals, and the heterogeneous network terminals dynamically adjust the CPU frequency and memory allocation according to the computing power allocation plan;

[0085] During the execution of the plan, resource usage, energy consumption, and completion status information are recorded. Resource usage includes CPU usage, memory usage, and network bandwidth. Energy consumption data is collected in real time through the terminal's built-in energy consumption monitoring module. Completion status information includes whether the plan was successfully completed, whether the execution time exceeded the predetermined threshold, and whether the calculation results met the accuracy requirements.

[0086] The reward value is calculated based on the solution execution time, energy consumption and completion status. The reward function is as follows:

[0087]

[0088] Where D is the execution time of the solution, E is the energy consumption of the solution, Q is the quality score of the solution completion status, and w1, w2, and w3 are weight factors;

[0089] The state, action, and reward data after task execution are stored in the experience buffer. The data is stored in the key-value pair format of [state, action, reward] and sorted by timestamp. The experience buffer adopts a fixed-capacity circular storage mechanism. When the storage capacity reaches the upper limit, the earliest data is overwritten.

[0090] The computing power allocation plan and its rewards are stored in the experience buffer, which is used to update the policy parameters of the behavior network and the state value function parameters of the evaluation network to generate a new policy and evaluate the state value of the computing power allocation plan.

[0091] Preferably, the behavior network is initialized using a multi-layer perceptron architecture, with the current state data as input; the evaluation network is initialized using a convolutional neural network architecture, and evaluates the computing power allocation plan based on the state data corresponding to the actions in the computing power allocation plan;

[0092] Trajectory data also includes historical CPU loads of heterogeneous network terminals, completion records of computing power allocation plans, energy consumption data, and device status information. Trajectory data is stored in the experience buffer in the form of time series;

[0093] The experience buffer adopts a fixed-capacity circular storage strategy.

[0094] Example 2 of the present invention provides a proximal strategy optimization system for heterogeneous network computing power control, including: an initialization module, a clipping range construction module, a strategy generation and optimization module, and a solution execution and reward update module;

[0095] The initialization module is used to set the initial parameters of the behavior network and the evaluation network according to the heterogeneous network computing power control scenario, build an experience buffer for storing state, action, reward, and next state data, load the trajectory data of the heterogeneous network terminals, and set the computing parameters of each heterogeneous network terminal;

[0096] The clipping range construction module is used to calculate the clipping range based on the set maximum training time step and the current training time step, and adjust the clipping value through the linear decay rule to constrain the deviation between the new policy generated by the behavior network and the old policy;

[0097] The strategy generation and optimization module is used to extract the current state data from the experience buffer and input it into the behavior network. The behavior network generates a strategy based on the input current state, outputs the action probability distribution under the current state, selects the action with the highest probability as the computing power allocation plan, and constructs the objective function of the behavior network based on the clipping range. At the same time, the evaluation network is used to evaluate the value of the plan allocation plan, and the objective function of the evaluation network is constructed by minimizing the state value prediction error.

[0098] The plan execution and reward update module is used to execute the plan on the heterogeneous network terminal according to the computing power allocation plan generated by the behavior network, calculate the reward based on the energy consumption, execution time and plan quality after the plan is completed, and store the current state, selected action, calculated reward value and next state data after the plan is executed in the experience buffer to optimize the behavior network and evaluation network.

[0099] Based on the proposed improved algorithm, this paper applies it to the DRL agent training process to evaluate its performance. Used to optimize behavioral networks.

[0100] In order to utilize The algorithm performs DRL model training. The present invention stores all samples within a time segment (including all K samples in FL) in the experience buffer D.

[0101] The DRL agent maintains an experience buffer D and a policy π θ (a k |s k ,h k-1 )(behavioral network) and a value function to estimate V v (s k )(evaluation network), where V v (s k ) and v are the parameters of the behavior network and the evaluation network respectively. The update of the evaluation network, v, is completed by minimizing the mean square error between the evaluation network output value function and the target value function. The loss function of the evaluation network update is:

[0102] L(v)=(r k +γV v (s k+1 )-V v (s k )) 2

[0103] According to formula (3.19), the loss function of the policy network in the DRL agent proposed in this chapter is:

[0104]

[0105] Advantage function satisfy:

[0106]

[0107] The detailed offline training process of the DRL agent is shown in Table 1.

[0108]

[0109] Now the present invention proposes The performance of the algorithm was simulated and evaluated. This paper used image classification as a federated learning solution and the standard MNIST dataset. A neural network with 21,840 trainable parameters was used. Each training sample was set to a 28×28 floating-point matrix, and the local machine learning model on each terminal was trained using mini-batch stochastic gradient descent with a batch size of d=20. The federated learning algorithm used was based on FedAvg, and the local epoch was set to 1. For the terminal's computational model, the number of CPU cycles required to execute a sample, c, was calculated. i Set to 20 turns / bit, maximum CPU cycle frequency Set to 3GHz, minimum CPU cycle frequency Set to 0.5GHz, the effective capacitance is 10 -28 , unit energy consumption is model upload p i is 0.5J / s.

[0110] The proposed algorithm is compared with three state-of-the-art methods and two baseline methods:

[0111] (i) DRL-based methods

[0112] (ii) Heuristic methods

[0113] (iii) Average method

[0114] (iv) Greedy Algorithm

[0115] (v) Randomized algorithms

[0116] The experimental results after 20 rounds of FL training are as follows Figure 1 As shown. Note that the improved proximal strategy optimization algorithm proposed in this invention It has the best performance in terms of average loss, average time consumption and average energy consumption.

[0117] like Figure 1 (a) shows the average loss per training round of the federated learning system under different algorithms. The L-PPO algorithm performs best in terms of average loss and has the lowest loss per round, outperforming PPO, heuristic, average, greedy, and random algorithms, demonstrating its significant advantage in optimizing system performance.

[0118] like Figure 1 (b) shows a comparison of the average training time per round of the federated learning system using different algorithms. The results show that the L-PPO algorithm significantly shortens training time, achieving the shortest training time per round, outperforming other methods. This is particularly evident when compared to heuristic and greedy algorithms, demonstrating its superiority in improving training efficiency.

[0119] like Figure 1 (c) shows the average energy consumption per training round of the federated learning system under different algorithms. The experimental results show that the L-PPO algorithm consumes the least energy, significantly outperforming the other five algorithms, demonstrating its significant performance advantage in energy optimization.

[0120] like Figure 2As shown in Figure 1, the modular design of the heterogeneous network computing power control system includes four core modules: the initialization module, the clipping range construction module, the strategy generation and optimization module, and the plan execution and reward update module. The initialization module is responsible for setting the parameters of the behavior network and the evaluation network and loading terminal trajectory data. The clipping range construction module generates a strategy for the behavior network by dynamically calculating the clipping range constraints. The strategy generation and optimization module generates a computing power allocation strategy based on the current state and optimizes network parameters. The plan execution and reward update module completes the execution of the computing power allocation plan, records energy consumption and completion status, and updates the reward value to improve system performance.

[0121] The beneficial effects of the present invention are as follows: computing power allocation strategies are dynamically generated through behavioral networks, and the stability of strategy updates is guaranteed by combining clipping range constraints. At the same time, the value of allocation schemes is accurately evaluated using evaluation networks, and the strategy generation rules of behavioral networks are optimized, thereby achieving efficient computing power resource allocation in heterogeneous network environments. In addition, the reward function comprehensively considers the execution time, resource consumption, and completion quality of the scheme, thereby resolving the contradiction between real-time performance and energy consumption optimization, and further improving the utilization efficiency of computing power resources and the reliability of scheme completion.

[0122] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A proximal strategy optimization method for heterogeneous network computing power control, wherein the heterogeneous network includes a behavior network, an evaluation network, an experience buffer, and a terminal; the behavior network is used to generate an action probability distribution and output a computing power allocation plan; the evaluation network is used to evaluate the computing power allocation plan based on the state data corresponding to the actions in the computing power allocation plan; the experience buffer is used to store state, action, and reward data; the heterogeneous network terminal is used to execute the computing power allocation plan and record the operating status after the execution of the computing power allocation plan; the operating status includes resource usage, energy consumption, and completion status information during the execution of the computing power allocation plan; and the characteristics are: The following steps are involved: The behavioral network inputs the trajectory data of heterogeneous network terminals, including the CPU frequency range, energy consumption curve change data and the maximum training time step; According to the maximum training time step and the current training time step, the linear attenuation method is used to determine the cropping range; The objective function of the behavior network is to maximize the cumulative reward, and the clipping range is used as the constraint condition of the objective function of the behavior network. The behavior network outputs the action probability distribution under the new strategy based on the input state data; The action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan. The evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function to evaluate the value of the completion state corresponding to the computing power allocation plan. Obtain resource usage, energy consumption, and completion status information during the execution of computing power allocation plans by heterogeneous network terminals; The behavior network determines the rewards for the computing power allocation plan based on resource usage, energy consumption, completion status information, and the value assessment results of the completion status; The computing power allocation plan and its rewards are stored in the experience buffer, which is used to update the policy parameters of the behavior network and the state value function parameters of the evaluation network to generate a new policy and evaluate the state value of the computing power allocation plan.

2. The method for optimizing the proximal strategy for heterogeneous network computing power control according to claim 1, characterized in that: The behavior network is initialized using a multi-layer perceptron architecture, with the current state data as input. The evaluation network is initialized using a convolutional neural network architecture, and evaluates the computing power allocation plan based on the state data corresponding to the actions in the computing power allocation plan. Trajectory data also includes historical CPU loads of heterogeneous network terminals, completion records of computing power allocation plans, energy consumption data, and device status information. Trajectory data is stored in the experience buffer in the form of time series; The experience buffer adopts a fixed-capacity circular storage strategy.

3. The method for optimizing the proximal strategy for heterogeneous network computing power control according to claim 1, characterized in that: Based on the maximum training time step and the current training time step, a linear attenuation method is used to determine the cropping range, including: Set the maximum training time step T and the current training time step t to dynamically generate the cropping range as follows: Where T represents the maximum training time step, and t represents the current training time step; represents the initial clipping value, Indicates the clipping range.

4. The method for optimizing the proximal strategy for heterogeneous network computing power control according to claim 1, characterized in that: The objective function of the behavior network is calculated as follows: Where, L(s,a,θ old ,θ) is the objective function of the behavior network, a represents action, s represents state, θ represents the new strategy parameter, θ old Represents the old policy parameters; represents the old policy θ old The improvement of action a in state s compared with the average action; clip(·,·,·) is the clipping function; π θ (a|s) represents the probability of executing action a in state s under the new strategy θ; represents the old policy θ old The probability of executing action a in state s; is a hyperparameter, which indicates the maximum deviation allowed between the new strategy and the old strategy.

5. The method for optimizing the proximal strategy for heterogeneous network computing power control according to claim 4, characterized in that: The clipping function satisfies the following relationship: The clipping range represented by the clipping function is used as the constraint condition of the objective function of the behavior network.

6. The method for optimizing the proximal strategy for heterogeneous network computing power control according to claim 1, characterized in that: Select the action corresponding to the maximum probability under the new strategy as the computing power allocation plan; The evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function, and evaluates the value of the completion state corresponding to the computing power allocation plan; including: Extracting state data for the current time step from the experience buffer, including terminal CPU load, solution execution records, and resource usage; normalizing the extracted data before inputting it into the behavior network; The behavior network generates an action probability distribution. Each action corresponds to a computing power allocation plan. The action with the highest probability is selected as the final computing power allocation plan based on the probability distribution. The evaluation network takes the current state data and the actions generated by the behavior network as input and calculates the state value function. The evaluation network parameters are optimized by comparing the predicted state value of the evaluation network with the accumulated reward after the plan is actually completed and minimizing the error between the two. The behavior network and the evaluation network collaborate to optimize. The evaluation network rewards the state data corresponding to the evaluation action according to the final computing power allocation plan, which is used to update the strategy generation of the behavior network.

7. The method for optimizing proximal strategy for heterogeneous network computing power control according to claim 6, characterized in that: The objective function of the evaluation network is calculated as follows: L(v)=(r k +γV v (s k+1 )-V v (s k )) 2 Where L(v) represents the objective function of the evaluation network; r k represents the immediate reward at time step k; γ represents the discount factor, which is used to balance the weights of current rewards and future rewards; s k represents the state of time step k; V v (s k ) represents the state value function of the current time step k predicted by the evaluation network; V v (s k+1 ) represents the state value function of the time step k+1 predicted by the evaluation network; L(θ) is the objective function of the behavior strategy network; π θ (a k |s k ,h k-1 ) is the new strategy θ, given state s k and historical information k-1 When you select action a k probability; Under the old policy θ, given state s k and historical information k-1 When you select action a k The probability of a k is the action of time step k, s k is the state of time step k, h k-1 For historical information; is the advantage function, used to measure the k Select action a k the expected advantage over the new strategy; represents the old policy θ old The state s at the next time step k k -Action a k The value function of is the expected cumulative reward obtained after performing the action.

8. The method for optimizing the proximal strategy for heterogeneous network computing power control according to claim 6, characterized in that: The updated behavior network generation strategy is calculated as follows: Where θ new Represents the new strategy parameters after optimization; represents the strategy to maximize the objective function; Represents the old strategy θ old The expected value of the generated state s and action a; θ represents the new policy parameters; θ old represents the old policy parameters; a represents the action; s represents the state.

9. The method for optimizing the proximal strategy for heterogeneous network computing power control according to claim 1, characterized in that: Determine the rewards for the computing power distribution plan, including: Based on the computing power allocation plan generated by the behavioral network, the plan is executed on the heterogeneous network terminals, and the heterogeneous network terminals dynamically adjust the CPU frequency and memory allocation according to the computing power allocation plan; During the execution of the plan, resource usage, energy consumption, and completion status information are recorded. Resource usage includes CPU usage, memory usage, and network bandwidth. Energy consumption data is collected in real time through the terminal's built-in energy consumption monitoring module. Completion status information includes whether the plan was successfully completed, whether the execution time exceeded the predetermined threshold, and whether the calculation results met the accuracy requirements. The reward value is calculated based on the solution execution time, energy consumption and completion status. The reward function is as follows: Where D is the execution time of the solution, E is the energy consumption of the solution, Q is the quality score of the solution completion status, and w1, w2, and w3 are weight factors; The state, action, and reward data after task execution are stored in the experience buffer. The data is stored in the key-value pair format of [state, action, reward] and sorted by timestamp. The experience buffer adopts a fixed-capacity circular storage mechanism. When the storage capacity reaches the upper limit, the earliest data is overwritten.

10. A proximal strategy optimization system for heterogeneous network computing power control, comprising: Initialization module, clipping range construction module, strategy generation and optimization module, and plan execution and reward update module; its characteristics are: The initialization module is used to set the initial parameters of the behavior network and evaluation network according to the heterogeneous network computing power control scenario, build an experience buffer for storing states, actions, and rewards, load the trajectory data of the heterogeneous network terminals, and set the computing parameters of each heterogeneous network terminal; The clipping range construction module is used to calculate the clipping range based on the set maximum training time step and the current training time step, and adjust the clipping value through the linear decay rule to constrain the deviation between the new policy generated by the behavior network and the old policy; The strategy generation and optimization module is used to extract the current state data from the experience buffer and input it into the behavior network. The behavior network generates a strategy based on the input current state, outputs the action probability distribution under the current state, selects the action with the highest probability as the computing power allocation plan, and constructs the objective function of the behavior network based on the clipping range. At the same time, the evaluation network is used to evaluate the value of the plan allocation plan, and the objective function of the evaluation network is constructed by minimizing the state value prediction error. The plan execution and reward update module is used to execute the plan on the heterogeneous network terminal according to the computing power allocation plan generated by the behavior network, calculate the reward based on the energy consumption, execution time and plan quality after the plan is completed, and store the current state, selected action, calculated reward value and next state data after the plan is executed in the experience buffer to optimize the behavior network and evaluation network.

Images