Heterogeneous network computing power control near-end strategy optimization method and system

Abstract

The invention discloses a heterogeneous network computing power control near-end strategy optimization method and system, and the method comprises the steps: based on a heterogeneous network computing power control scene, setting initial parameters of a behavior network and an evaluation network, constructing an experience buffer region for storing training data, loading the trajectory data of heterogeneous network terminals, and setting the computing parameters of each heterogeneous network terminal, adjusting the cutting value through a linear attenuation rule; generating a strategy by the behavior network according to the input current state data, outputting and selecting an action with the maximum probability as a computing power distribution scheme, and constructing a target function of the behavior network by taking the maximum accumulated reward as a target; carrying out value evaluation on actions executed in the distribution scheme by utilizing the evaluation network, and constructing a target function of the evaluation network by taking a prediction error of a minimum state value function as a target; and executing the scheme at the heterogeneous network terminal according to the computing power distribution scheme, calculating rewards in combination with energy consumption, execution time and scheme quality after the scheme is completed, and storing training data in an experience buffer area.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep reinforcement learning, and more specifically, relates to a proximal strategy optimization method and system for heterogeneous network computing power control. Background Art

[0002] In a heterogeneous network environment, the management and allocation of computing resources is a complex technical challenge. Since different computing nodes have significant differences in performance, energy consumption, and solution processing capabilities, and different solutions have different requirements for computing resources, existing resource allocation methods usually use static rules or heuristic algorithms. However, these methods lack dynamic adaptability and have difficulty coping with real-time changes in solution requirements and device status in heterogeneous networks, resulting in low resource utilization and failure to meet the efficient processing requirements of diversified solutions in complex scenarios.

[0003] In recent years, reinforcement learning has gradually been applied to the scheduling and allocation of computing resources, but its application in heterogeneous networks still faces many bottlenecks. On the one hand, the strategies generated by reinforcement learning may fluctuate, resulting in unstable computing power allocation schemes; on the other hand, due to the complexity of the state space and action space in heterogeneous networks, the existing reinforcement learning models are not accurate enough in evaluating the value of scheme allocation, making it difficult to achieve effective optimization. In addition, the existing methods lack a constraint mechanism in the policy update process, which may cause the strategy to deviate significantly from the optimal solution range, thereby reducing the reliability and efficiency of resource allocation.

[0004] In addition, the allocation of solutions in heterogeneous networks needs to achieve a balance between real-time and energy optimization, which further exacerbates the complexity of computing power control. Solutions usually have strict latency requirements, but in order to save energy, it is necessary to limit the excessive use of computing resources. Existing technologies are difficult to take both goals into account at the same time. Excessive pursuit of real-time may lead to redundant resource allocation, while over-emphasizing energy optimization may delay the completion time of the solution. Therefore, the lack of a unified optimization framework that can effectively solve the contradiction between real-time and energy consumption has become a problem that needs to be solved urgently in current technology. Summary of the invention

[0005] In order to solve the deficiencies in the prior art, the present invention provides a proximal strategy optimization method for heterogeneous network computing power control.

[0006] The present invention adopts the following technical solution.

[0007] A first aspect of the present invention provides a proximal strategy optimization method for heterogeneous network computing power control, wherein the heterogeneous network includes a behavior network, an evaluation network, an experience buffer, and a terminal; the behavior network is used to generate an action probability distribution and output a computing power allocation plan; the evaluation network is used to evaluate the computing power allocation plan according to the state data corresponding to the action in the computing power allocation plan; the experience buffer is used to store state, action and reward data; the heterogeneous network terminal is used to execute the computing power allocation plan and record the running state after the computing power allocation plan is executed; the running state includes resource usage, energy consumption and completion state information during the execution of the computing power allocation plan; and the following steps are included:

[0008] The behavior network inputs the trajectory data of the heterogeneous network terminals, including the CPU frequency range, the change data of the energy consumption curve, and the maximum training time step;

[0009] According to the maximum training time step and the current training time step, the linear attenuation method is used to determine the cropping range;

[0010] The objective function of the behavior network is to maximize the cumulative reward, and the clipping range is used as the constraint condition of the objective function of the behavior network. The behavior network outputs the action probability distribution under the new strategy according to the input state data.

[0011] The action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan; the evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function, and evaluates the value of the completion state corresponding to the computing power allocation plan;

[0012] The heterogeneous network terminals obtain resource usage, energy consumption and completion status information during the execution of the computing power allocation plan; the behavioral network determines the reward of the computing power allocation plan based on the resource usage, energy consumption, completion status information and the value evaluation results of the completion status;

[0013] The computing power allocation plan and the rewards of the computing power allocation plan are stored in the experience buffer, which is used to update the policy parameters of the behavior network and the state value function parameters of the evaluation network to generate a new policy and evaluate the state value of the computing power allocation plan.

[0014] Preferably, the behavior network is initialized using a multi-layer perceptron architecture, and the input is the current state data; the evaluation network is initialized using a convolutional neural network architecture, and the computing power allocation plan is evaluated according to the state data corresponding to the actions in the computing power allocation plan;

[0015] The trajectory data also includes the historical CPU load of heterogeneous network terminals, the completion record of computing power allocation scheme, energy consumption data and device status information. The trajectory data is stored in the experience buffer in the form of time series;

[0016] The experience buffer adopts a fixed-capacity circular storage strategy.

[0017] Preferably, according to the maximum training time step and the current training time step, a linear attenuation method is used to determine the cropping range, including:

[0018] Set the maximum training time step T and the current training time step t to dynamically generate the cropping range as follows:

[0019]

[0020] Where T represents the maximum training time step, t represents the current training time step; ∈0 represents the initial clipping value, Indicates the clipping range.

[0021] Preferably, the objective function of the behavior network is calculated as follows:

[0022]

[0023] In the formula, L(s,a,θ old ,θ) is the objective function of the behavior network, a represents action, s represents state, θ represents the new strategy parameter, θ old Represents the old policy parameters; represents the old policy θ old The improvement of action a in state s compared with the average action; clip(·,·,·) is the clipping function; π θ (a|s) represents the probability of executing action a in state s under the new strategy θ; represents the old policy θ old The probability of executing action a in state s; is a hyperparameter, which indicates the maximum deviation allowed between the new strategy and the old strategy.

[0024] Preferably, the clipping function satisfies the following relationship:

[0025]

[0026] The trimming range represented by the trimming function is used as the constraint condition of the objective function of the behavior network.

[0027] Preferably, the action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan; the evaluation network takes the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function, and performs value evaluation on the completion state corresponding to the computing power allocation plan; including:

[0028] Extract the state data of the current time step from the experience buffer, which includes the terminal CPU load, solution execution record and resource usage; the extracted data is normalized and then input into the behavior network;

[0029] The behavior network generates an action probability distribution. Each action corresponds to a computing power allocation plan. The action with the highest probability is selected as the final computing power allocation plan based on the probability distribution.

[0030] The evaluation network inputs the current state data and the actions generated by the behavior network to calculate the state value function. The evaluation network parameters are optimized by comparing the predicted state value of the evaluation network with the accumulated reward after the plan is actually completed and minimizing the error between the two.

[0031] The behavior network and the evaluation network collaborate to optimize. The evaluation network rewards the state data corresponding to the evaluation action according to the final computing power allocation plan, which is used to update the strategy generation of the behavior network.

[0032] Preferably, the objective function of the evaluation network is calculated as follows:

[0033] L(v)=(r k +γV v (s k+1 )-V v (s k )) 2

[0034]

[0035] Where L(v) represents the objective function of the evaluation network; r k represents the immediate reward at time step k; γ represents the discount factor, which is used to balance the weight of current rewards and future rewards; s k represents the state at time step k; V v (s k ) represents the state value function of the current time step k predicted by the evaluation network; V v (s k+1 ) represents the state value function of the time step k+1 predicted by the evaluation network; L(θ) is the objective function of the behavior strategy network; π θ (a k |s k ,h k-1 ) is the new strategy θ, given the state s k and historical information k-1 When k probability; Given the state s under the old policy θ k and historical information k-1 When k The probability of k is the action of time step k, s k is the state of time step k, h k-1 For historical information; is the advantage function, which is used to measure k Select action a k the expected advantage over the new strategy; represents the old policy θ old The state s at the next time step k k -Action a k The value function of is the expected cumulative reward after performing the action.

[0036] Preferably, the update behavior network generation strategy is calculated as follows:

[0037]

[0038] In the formula, θ new Represents the new strategy parameters after optimization; represents the strategy to maximize the objective function; Represents the old strategy θ old The expected value of the generated state s and action a; θ represents the new policy parameter; θ old represents the old policy parameters; a represents the action; s represents the state.

[0039] Preferably, the rewards for determining the computing power allocation plan include:

[0040] According to the computing power allocation plan generated by the behavior network, the plan is executed on the heterogeneous network terminal, and the heterogeneous network terminal dynamically adjusts the CPU frequency and memory allocation according to the computing power allocation plan;

[0041] During the execution of the plan, resource usage, energy consumption and completion status information are recorded. Resource usage includes CPU usage, memory usage and network bandwidth; energy consumption data is collected in real time through the terminal's built-in energy consumption monitoring module; completion status information includes whether the plan is successfully completed, whether the execution time exceeds the predetermined threshold, and whether the calculation result meets the accuracy requirements;

[0042] The reward value is calculated based on the solution execution time, energy consumption and completion status. The reward function is as follows:

[0043]

[0044] In the formula, D is the execution time of the solution, E is the energy consumption of the solution, Q is the quality score of the solution completion status, and w1, w2, and w3 are weight factors;

[0045] The state, action and reward data after task execution are stored in the experience buffer. The data is stored in the key-value pair format of [state, action, reward] and sorted by timestamp. The experience buffer adopts a fixed-capacity circular storage mechanism. When the storage capacity reaches the upper limit, the earliest data is overwritten.

[0046] A second aspect of the present invention provides a proximal strategy optimization system for heterogeneous network computing power control, including: an initialization module, a clipping range construction module, a strategy generation and optimization module, and a scheme execution and reward update module;

[0047] The initialization module is used to set the initial parameters of the behavior network and the evaluation network according to the heterogeneous network computing power control scenario, build an experience buffer for storing states, actions, and rewards, load the trajectory data of the heterogeneous network terminals, and set the computing parameters of each heterogeneous network terminal;

[0048] The clipping range construction module is used to calculate the clipping range according to the set maximum training time step and the current training time step, and adjust the clipping value through the linear decay rule to constrain the deviation between the new strategy generated by the behavior network and the old strategy;

[0049] The strategy generation and optimization module is used to extract the current state data from the experience buffer and input it into the behavior network. The behavior network generates a strategy based on the input current state, outputs the action probability distribution under the current state, selects the action with the highest probability as the computing power allocation plan, and constructs the objective function of the behavior network based on the clipping range. At the same time, the evaluation network is used to evaluate the value of the plan allocation plan, and the objective function of the evaluation network is constructed by minimizing the state value prediction error;

[0050] The scheme execution and reward update module is used to execute the scheme in the heterogeneous network terminal according to the computing power allocation scheme generated by the behavior network, calculate the reward based on the energy consumption, execution time and scheme quality after the scheme is completed, and store the current state, selected action, calculated reward value and next state data after the scheme is executed in the experience buffer to optimize the behavior network and evaluation network.

[0051] The beneficial effects of the present invention are as follows: the computing power allocation strategy is dynamically generated through the behavior network, and the stability of the strategy update is guaranteed by combining the clipping range constraint. At the same time, the value of the allocation plan is accurately evaluated by using the evaluation network, and the strategy generation rules of the behavior network are optimized, thereby realizing efficient computing power resource allocation in a heterogeneous network environment. In addition, the reward function comprehensively considers the execution time, resource consumption and completion quality of the plan, thereby resolving the contradiction between real-time and energy consumption optimization, and further improving the utilization efficiency of computing power resources and the reliability of plan completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a graph showing comparative experimental results of five algorithms provided according to an embodiment of the present invention;

[0053] Figure 2 It is a schematic diagram of a heterogeneous network computing power optimization architecture provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0055] Example 1 of the present invention provides a proximal strategy optimization method for heterogeneous network computing power control, wherein the heterogeneous network includes a behavior network, an evaluation network, an experience buffer and a terminal; the behavior network is used to generate an action probability distribution and output a computing power allocation plan; the evaluation network is used to evaluate the computing power allocation plan according to the state data corresponding to the action in the computing power allocation plan; the experience buffer is used to store state, action and reward data; the heterogeneous network terminal is used to execute the computing power allocation plan and record the running state after the computing power allocation plan is executed; the running state includes resource usage, energy consumption and completion state information during the execution of the computing power allocation plan; and the following steps are included:

[0056] The behavior network inputs the trajectory data of the heterogeneous network terminals, including the CPU frequency range, the change data of the energy consumption curve, and the maximum training time step;

[0057] According to the maximum training time step and the current training time step, the linear attenuation method is used to determine the cropping range;

[0058] Preferably, according to the maximum training time step and the current training time step, a linear attenuation method is used to determine the cropping range, including:

[0059] Set the maximum training time step T and the current training time step t to dynamically generate the cropping range as follows:

[0060]

[0061] Where T represents the maximum training time step, t represents the current training time step; ∈0 represents the initial clipping value, Indicates the clipping range.

[0062] The objective function of the behavior network is to maximize the cumulative reward, and the clipping range is used as the constraint condition of the objective function of the behavior network. The behavior network outputs the action probability distribution under the new strategy according to the input state data.

[0063] Preferably, the objective function of the behavior network is calculated as follows:

[0064]

[0065] In the formula, L(s,a,θold ,θ) is the objective function of the behavior network, a represents action, s represents state, θ represents the new strategy parameter, θ old Represents the old policy parameters; represents the old policy θ old The improvement of action a in state s compared with the average action; clip(·,·,·) is the clipping function; π θ (a|s) represents the probability of executing action a in state s under the new strategy θ; represents the old policy θ old The probability of executing action a in state s; is a hyperparameter, which indicates the maximum deviation allowed between the new strategy and the old strategy.

[0066] Preferably, the clipping function satisfies the following relationship:

[0067]

[0068] The trimming range represented by the trimming function is used as the constraint condition of the objective function of the behavior network.

[0069] The action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan; the evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function, and evaluates the value of the completion state corresponding to the computing power allocation plan;

[0070] Preferably, the action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan; the evaluation network takes the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function, and performs value evaluation on the completion state corresponding to the computing power allocation plan; including:

[0071] Extract the state data of the current time step from the experience buffer, which includes the terminal CPU load, solution execution record and resource usage; the extracted data is normalized and then input into the behavior network;

[0072] The behavior network generates an action probability distribution. Each action corresponds to a computing power allocation plan. The action with the highest probability is selected as the final computing power allocation plan based on the probability distribution.

[0073] The evaluation network inputs the current state data and the actions generated by the behavior network to calculate the state value function. The evaluation network parameters are optimized by comparing the predicted state value of the evaluation network with the accumulated reward after the plan is actually completed and minimizing the error between the two.

[0074] The behavior network and the evaluation network collaborate to optimize. The evaluation network rewards the state data corresponding to the evaluation action according to the final computing power allocation plan, which is used to update the strategy generation of the behavior network.

[0075] Preferably, the objective function of the evaluation network is calculated as follows:

[0076] L(v)=(r k +γV v (s k+1 )-V v (s k )) 2

[0077]

[0078] Where L(v) represents the objective function of the evaluation network; r k represents the immediate reward at time step k; γ represents the discount factor, which is used to balance the weight of current rewards and future rewards; s k represents the state at time step k; V v (s k ) represents the state value function of the current time step k predicted by the evaluation network; V v (s k+1 ) represents the state value function of the time step k+1 predicted by the evaluation network; L(θ) is the objective function of the behavior strategy network; π θ (a k |s k ,h k-1 ) is the new strategy θ, given the state s k and historical information k-1 When k The probability of Given the state s under the old policy θ k and historical information k-1 When k The probability of k is the action of time step k, s k is the state of time step k, h k-1 For historical information; is the advantage function, which is used to measure k Select action a k the expected advantage over the new strategy; represents the old policy θ old The state s at the next time step k k -Action a k The value function of is the expected cumulative reward after performing the action.

[0079] Preferably, the update behavior network generation strategy is calculated as follows:

[0080]

[0081] In the formula, θnew Represents the new strategy parameters after optimization; represents the strategy to maximize the objective function; Represents the old strategy θ old The expected value of the generated state s and action a; θ represents the new policy parameter; θ old represents the old policy parameters; a represents the action; s represents the state.

[0082] The heterogeneous network terminals obtain resource usage, energy consumption and completion status information during the execution of the computing power allocation plan; the behavioral network determines the reward of the computing power allocation plan based on the resource usage, energy consumption, completion status information and the value evaluation results of the completion status;

[0083] Preferably, the rewards for determining the computing power allocation plan include:

[0084] According to the computing power allocation plan generated by the behavior network, the plan is executed on the heterogeneous network terminal, and the heterogeneous network terminal dynamically adjusts the CPU frequency and memory allocation according to the computing power allocation plan;

[0085] During the execution of the plan, resource usage, energy consumption and completion status information are recorded. Resource usage includes CPU usage, memory usage and network bandwidth; energy consumption data is collected in real time through the terminal's built-in energy consumption monitoring module; completion status information includes whether the plan is successfully completed, whether the execution time exceeds the predetermined threshold, and whether the calculation result meets the accuracy requirements;

[0086] The reward value is calculated based on the solution execution time, energy consumption and completion status. The reward function is as follows:

[0087]

[0088] In the formula, D is the execution time of the solution, E is the energy consumption of the solution, Q is the quality score of the solution completion status, and w1, w2, and w3 are weight factors;

[0089] The state, action and reward data after task execution are stored in the experience buffer. The data is stored in the key-value pair format of [state, action, reward] and sorted by timestamp. The experience buffer adopts a fixed-capacity circular storage mechanism. When the storage capacity reaches the upper limit, the earliest data is overwritten.

[0090] The computing power allocation plan and the rewards of the computing power allocation plan are stored in the experience buffer, which is used to update the policy parameters of the behavior network and the state value function parameters of the evaluation network to generate a new policy and evaluate the state value of the computing power allocation plan.

[0091] Preferably, the behavior network is initialized using a multi-layer perceptron architecture, and the input is the current state data; the evaluation network is initialized using a convolutional neural network architecture, and the computing power allocation plan is evaluated according to the state data corresponding to the actions in the computing power allocation plan;

[0092] The trajectory data also includes the historical CPU load of heterogeneous network terminals, the completion record of computing power allocation scheme, energy consumption data and device status information. The trajectory data is stored in the experience buffer in the form of time series;

[0093] The experience buffer adopts a fixed-capacity circular storage strategy.

[0094] Example 2 of the present invention provides a proximal strategy optimization system for heterogeneous network computing power control, including: an initialization module, a clipping range construction module, a strategy generation and optimization module, and a scheme execution and reward update module;

[0095] The initialization module is used to set the initial parameters of the behavior network and the evaluation network according to the heterogeneous network computing power control scenario, build an experience buffer for storing state, action, reward and next state data, load the trajectory data of the heterogeneous network terminal, and set the computing parameters of each heterogeneous network terminal;

[0096] The clipping range construction module is used to calculate the clipping range according to the set maximum training time step and the current training time step, and adjust the clipping value through the linear decay rule to constrain the deviation between the new strategy generated by the behavior network and the old strategy;

[0097] The strategy generation and optimization module is used to extract the current state data from the experience buffer and input it into the behavior network. The behavior network generates a strategy based on the input current state, outputs the action probability distribution under the current state, selects the action with the highest probability as the computing power allocation plan, and constructs the objective function of the behavior network based on the clipping range. At the same time, the evaluation network is used to evaluate the value of the plan allocation plan, and the objective function of the evaluation network is constructed by minimizing the state value prediction error;

[0098] The scheme execution and reward update module is used to execute the scheme in the heterogeneous network terminal according to the computing power allocation scheme generated by the behavior network, calculate the reward based on the energy consumption, execution time and scheme quality after the scheme is completed, and store the current state, selected action, calculated reward value and next state data after the scheme is executed in the experience buffer to optimize the behavior network and evaluation network.

[0099] Based on the proposed improved algorithm, the present invention applies it to the DRL agent training process to evaluate its performance. Used to optimize behavioral networks.

[0100] To utilize The algorithm performs DRL model training. The present invention stores all samples within a time segment (including all K samples in FL) in the experience buffer D.

[0101] The DRL agent maintains an experience buffer D and a policy π θ (a k |s k ,h k-1 )(behavior network) and a value function to estimate V v (s k )(evaluation network), where V v (s k ) and v are the parameters of the behavior network and the evaluation network respectively. The update of the evaluation network, v, is completed by minimizing the mean square error between the output value function of the evaluation network and the target value function. The loss function of the evaluation network update is:

[0102] L(v)=(r k +γV v (s k+1 )-V v (s k )) 2

[0103] According to formula (3.19), the loss function of the policy network in the DRL agent proposed in this chapter is:

[0104]

[0105] Advantage function satisfy:

[0106]

[0107] The detailed offline training process of the DRL agent is shown in Table 1.

[0108]

[0109] Now the present invention proposes The performance of the algorithm is simulated and evaluated. The present invention uses image classification as a federated learning scheme and uses the standard dataset MNIST. A neural network with 21840 trainable parameters is used. Each training sample is set to a 28×28 floating-point matrix, and the local machine learning model on each terminal is trained using a small batch stochastic gradient descent method with a batch size of d=20. The federated learning algorithm used is based on FedAvg, and the local epoch is set to 1. For the computing model of the terminal, the number of CPU cycles required to execute a sample c is set to i Set to 20 turns / bit, maximum CPU cycle frequency Set to 3GHz, minimum CPU cycle frequency Set to 0.5GHz, the effective capacitance is 10 -28 , unit energy consumption is model upload p i It is 0.5J / s.

[0110] The proposed algorithm is compared with three state-of-the-art methods and two baseline methods:

[0111] (i) DRL-based methods

[0112] (ii) Heuristic Methods

[0113] (iii) Average value method

[0114] (iv) Greedy Algorithm

[0115] (v) Randomized algorithms

[0116] The experimental results after 20 rounds of FL training are as follows Figure 1 As shown. Note that the improved proximal strategy optimization algorithm proposed in this invention It has the best performance in terms of average loss, average time consumption and average energy consumption.

[0117] like Figure 1 (a) shows the average loss of each round of training in the federated learning system under different algorithms. The L-PPO algorithm performs best in terms of average loss and has the lowest loss per round, outperforming PPO, heuristic, average, greedy, and random algorithms, indicating its significant advantage in optimizing system performance.

[0118] like Figure 1 (b) shows a comparison of the average time spent on each round of training for the federated learning system under different algorithms. The results show that the L-PPO algorithm significantly shortens the training time, with the shortest training time per round, outperforming other methods, especially in comparison with the heuristic and greedy algorithms, which demonstrates its superiority in improving training efficiency.

[0119] like Figure 1 (c) shows the average energy consumption of each round of training of the federated learning system under different algorithms. The experimental results show that the L-PPO algorithm consumes the least energy and is significantly better than the other five algorithms, proving that it has obvious performance advantages in energy consumption optimization.

[0120] like Figure 2As shown in the figure, the modular design of the heterogeneous network computing power control system includes four core modules: initialization module, cutting range construction module, strategy generation and optimization module, and scheme execution and reward update module. The initialization module is responsible for setting the parameters of the behavior network and the evaluation network and loading the terminal trajectory data; the cutting range construction module generates the strategy of the behavior network by dynamically calculating the cutting range constraint; the strategy generation and optimization module generates the computing power allocation strategy according to the current state and optimizes the network parameters; the scheme execution and reward update module completes the execution of the computing power allocation plan, records the energy consumption and completion status, and updates the reward value to improve the system performance.

[0121] The beneficial effects of the present invention are as follows: the computing power allocation strategy is dynamically generated through the behavior network, and the stability of the strategy update is guaranteed by combining the clipping range constraint. At the same time, the value of the allocation plan is accurately evaluated by using the evaluation network, and the strategy generation rules of the behavior network are optimized, thereby realizing efficient computing power resource allocation in a heterogeneous network environment. In addition, the reward function comprehensively considers the execution time, resource consumption and completion quality of the plan, thereby resolving the contradiction between real-time and energy consumption optimization, and further improving the utilization efficiency of computing power resources and the reliability of plan completion.

[0122] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A proximal strategy optimization method for heterogeneous network computing power control, wherein the heterogeneous network includes a behavior network, an evaluation network, an experience buffer, and a terminal; the behavior network is used to generate an action probability distribution and output a computing power allocation plan; the evaluation network is used to evaluate the computing power allocation plan according to the state data corresponding to the action in the computing power allocation plan; the experience buffer is used to store state, action and reward data; the heterogeneous network terminal is used to execute the computing power allocation plan and record the operating state after the computing power allocation plan is executed; the operating state includes resource usage, energy consumption and completion state information during the execution of the computing power allocation plan; it is characterized in that The following steps are involved: The behavior network inputs the trajectory data of the heterogeneous network terminals, including the CPU frequency range, the change data of the energy consumption curve, and the maximum training time step; According to the maximum training time step and the current training time step, the linear attenuation method is used to determine the cropping range; The objective function of the behavior network is to maximize the cumulative reward, and the clipping range is used as the constraint condition of the objective function of the behavior network. The behavior network outputs the action probability distribution under the new strategy according to the input state data. The action corresponding to the maximum probability under the new strategy is selected as the computing power allocation plan; the evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function, and evaluates the value of the completion state corresponding to the computing power allocation plan; Obtain resource usage, energy consumption, and completion status information during the execution of computing power allocation plans by heterogeneous network terminals; The behavior network determines the rewards for the computing power allocation plan based on resource usage, energy consumption, completion status information, and the value assessment results of the completion status; The computing power allocation plan and the rewards of the computing power allocation plan are stored in the experience buffer, which is used to update the policy parameters of the behavior network and the state value function parameters of the evaluation network to generate a new policy and evaluate the state value of the computing power allocation plan.

2. According to claim 1, a proximal strategy optimization method for heterogeneous network computing power control is characterized by: The behavior network is initialized with a multi-layer perceptron architecture, and the input is the current state data; the evaluation network is initialized with a convolutional neural network architecture, and the computing power allocation plan is evaluated based on the state data corresponding to the actions in the computing power allocation plan; The trajectory data also includes the historical CPU load of heterogeneous network terminals, the completion record of computing power allocation scheme, energy consumption data and device status information. The trajectory data is stored in the experience buffer in the form of time series; The experience buffer adopts a fixed-capacity circular storage strategy.

3. The method for optimizing the proximal strategy of heterogeneous network computing power control according to claim 1, characterized in that: According to the maximum training time step and the current training time step, a linear attenuation method is used to determine the cropping range, including: Set the maximum training time step T and the current training time step t to dynamically generate the cropping range as follows: Where T represents the maximum training time step, t represents the current training time step; ∈0 represents the initial clipping value, Indicates the clipping range.

4. The method for optimizing the proximal strategy of heterogeneous network computing power control according to claim 1, characterized in that: The objective function of the behavior network is calculated as follows: In the formula, L(s,a,θ old ,θ) is the objective function of the behavior network, a represents action, s represents state, θ represents the new strategy parameter, θ old Represents the old policy parameters; represents the old policy θ old The improvement of action a in state s compared with the average action; clip(·,·,·) is the clipping function; π θ (a|s) represents the probability of executing action a in state s under the new strategy θ; represents the old policy θ old The probability of executing action a in state s; is a hyperparameter, which indicates the maximum deviation allowed between the new strategy and the old strategy.

5. The method for optimizing the proximal strategy of heterogeneous network computing power control according to claim 4 is characterized in that: The clipping function satisfies the following relationship: The trimming range represented by the trimming function is used as the constraint condition of the objective function of the behavior network.

6. The method for optimizing the proximal strategy of heterogeneous network computing power control according to claim 1, characterized in that: Select the action corresponding to the maximum probability under the new strategy as the computing power allocation plan; The evaluation network uses the minimum prediction error of the state corresponding to the action in the computing power allocation plan as the objective function, and evaluates the value of the completion state corresponding to the computing power allocation plan; including: Extract the state data of the current time step from the experience buffer, which includes the terminal CPU load, solution execution record and resource usage; the extracted data is normalized and then input into the behavior network; The behavior network generates an action probability distribution. Each action corresponds to a computing power allocation plan. The action with the highest probability is selected as the final computing power allocation plan based on the probability distribution. The evaluation network inputs the current state data and the actions generated by the behavior network to calculate the state value function. The evaluation network parameters are optimized by comparing the predicted state value of the evaluation network with the accumulated reward after the plan is actually completed and minimizing the error between the two. The behavior network and the evaluation network collaborate to optimize. The evaluation network rewards the state data corresponding to the evaluation action according to the final computing power allocation plan, which is used to update the strategy generation of the behavior network.

7. The method for optimizing the proximal strategy of heterogeneous network computing power control according to claim 6, characterized in that: The objective function of the evaluation network is calculated as follows: L(v)=(r k +γV v (s k+1 )-V v (s k )) 2 Where L(v) represents the objective function of the evaluation network; r k represents the immediate reward at time step k; γ represents the discount factor, which is used to balance the weights of current rewards and future rewards; s k represents the state at time step k; V v (s k ) represents the state value function of the current time step k predicted by the evaluation network; V v (s k+1 ) represents the state value function of the time step k+1 predicted by the evaluation network; L(θ) is the objective function of the behavior strategy network; π θ (a k |s k ,h k-1 ) is the new strategy θ, given the state s k and historical information k-1 When , select action a k probability; Given the state s under the old policy θ k and historical information k-1 When , select action a k The probability of k is the action of time step k, s k is the state of time step k, h k-1 For historical information; is the advantage function, which is used to measure k Choose the expected advantage of action k over the new policy; represents the old policy θ old The state s at the next time step k k -Action a k The value function of is the expected cumulative reward after performing the action.

8. The method for optimizing the proximal strategy of heterogeneous network computing power control according to claim 6, characterized in that: The updated behavior network generation strategy is calculated as follows: In the formula, θ new Represents the new strategy parameters after optimization; represents the strategy to maximize the objective function; Represents the old strategy θ old The expected value of the generated state s and action a; θ represents the new policy parameter; θ old represents the old policy parameters; a represents the action; s represents the state.

9. The method for optimizing the proximal strategy of heterogeneous network computing power control according to claim 1, characterized in that: Determine the rewards for the hashrate distribution plan, including: According to the computing power allocation plan generated by the behavior network, the plan is executed on the heterogeneous network terminal, and the heterogeneous network terminal dynamically adjusts the CPU frequency and memory allocation according to the computing power allocation plan; During the execution of the plan, resource usage, energy consumption and completion status information are recorded. Resource usage includes CPU usage, memory usage and network bandwidth; energy consumption data is collected in real time through the terminal's built-in energy consumption monitoring module; completion status information includes whether the plan is successfully completed, whether the execution time exceeds the predetermined threshold, and whether the calculation result meets the accuracy requirements; The reward value is calculated based on the solution execution time, energy consumption and completion status. The reward function is as follows: In the formula, D is the execution time of the solution, E is the energy consumption of the solution, Q is the quality score of the solution completion status, and w1, w2, and w3 are weight factors; The state, action and reward data after task execution are stored in the experience buffer. The data is stored in the key-value pair format of [state, action, reward] and sorted by timestamp. The experience buffer adopts a fixed-capacity circular storage mechanism. When the storage capacity reaches the upper limit, the earliest data is overwritten.

10. A proximal strategy optimization system for heterogeneous network computing power control, comprising: Initialization module, clipping range construction module, strategy generation and optimization module, and program execution and reward update module; characterized by: The initialization module is used to set the initial parameters of the behavior network and the evaluation network according to the heterogeneous network computing power control scenario, build an experience buffer for storing states, actions, and rewards, load the trajectory data of the heterogeneous network terminals, and set the computing parameters of each heterogeneous network terminal; The clipping range construction module is used to calculate the clipping range according to the set maximum training time step and the current training time step, and adjust the clipping value through the linear decay rule to constrain the deviation between the new strategy generated by the behavior network and the old strategy; The strategy generation and optimization module is used to extract the current state data from the experience buffer and input it into the behavior network. The behavior network generates a strategy based on the input current state, outputs the action probability distribution under the current state, selects the action with the highest probability as the computing power allocation plan, and constructs the objective function of the behavior network based on the clipping range. At the same time, the evaluation network is used to evaluate the value of the plan allocation plan, and the objective function of the evaluation network is constructed by minimizing the state value prediction error; The scheme execution and reward update module is used to execute the scheme in the heterogeneous network terminal according to the computing power allocation scheme generated by the behavior network, calculate the reward based on the energy consumption, execution time and scheme quality after the scheme is completed, and store the current state, selected action, calculated reward value and next state data after the scheme is executed in the experience buffer to optimize the behavior network and evaluation network.

Images