Energy-saving parameter configuration method and device, network equipment, medium and program product

Abstract

The invention provides an energy-saving parameter configuration method and device, network equipment, a medium and a program product, and relates to the technical field of computers. The cell energy-saving parameter configuration method comprises the following steps: inputting state information of a first state generated by a cell based on an energy-saving strategy before optimization into a strategy network, and outputting an optimized energy-saving parameter threshold value by the strategy network; on the basis of a reward and punishment function, an instant reward transferred to a second state after the optimized energy-saving strategy is executed in the first state is calculated; enabling the evaluation network to construct a first state value function corresponding to the first state based on forward propagation; constructing a first loss function of the evaluation network and a second loss function of the strategy network based on the first state value function; and based on the first loss function and the second loss function, jointly iteratively training the strategy network and the evaluation network, and determining the trained strategy network as a strategy model. According to the technical scheme of the invention, the method can achieve the automatic adjustment and optimization of the energy-saving parameter threshold value of a large-scale base station cell.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for configuring cell energy-saving parameters, a device for configuring cell energy-saving parameters, a network device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of mobile Internet and the widespread popularity of various smart terminals, users' demand for data traffic has exploded. Although dense base station deployment can effectively improve the network's carrying capacity, it also raises the issue of base station energy consumption optimization. Energy consumption optimization of base stations can be achieved by configuring energy-saving strategies for base stations. However, the energy efficiency of energy-saving strategies is highly dependent on the energy-saving parameter thresholds of the strategies. Therefore, how to optimize the energy-saving parameter thresholds of energy-saving strategies has become a technical problem that needs to be solved urgently.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a cell energy-saving parameter configuration method, configuration device, network equipment, storage medium and computer program product, which at least to some extent overcome the problem of poor configuration of energy-saving parameter thresholds of energy-saving strategies in related technologies.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a method for configuring cell energy-saving parameters is provided, comprising: inputting state information of a first state generated by a pre-optimization energy-saving strategy of the cell into a policy network, the pre-optimization energy-saving strategy being configured based on a pre-optimization energy-saving parameter threshold, so that the policy network outputs the optimized energy-saving parameter threshold, and the optimized energy-saving parameter threshold is used to configure the optimized energy-saving strategy; calculating an immediate reward for transferring to a second state after executing the optimized energy-saving strategy in the first state based on a reward and punishment function; inputting the state information of the first state, the immediate reward, and the state information of the second state into an evaluation network, so that the evaluation network constructs a first state value function corresponding to the first state based on forward propagation; constructing a first loss function of the evaluation network and a second loss function of the policy network based on the first state value function; jointly iteratively training the policy network and the evaluation network based on the first loss function and the second loss function, and determining the trained policy network as a policy model, so that the target threshold of the cell energy-saving parameter is configured by the policy model.

[0007] In an embodiment of the present disclosure, the first state value function comprises an evaluation state value function of the first state and a first target state value function, the first loss function of the evaluation network and the second loss function of the policy network are constructed based on the first state value function, comprising: constructing the first loss function based on the evaluation state value function and the first target state value function, and evaluating the advantage function of the optimized energy-saving parameter threshold; constructing the second loss function based on the advantage function.

[0008] In an embodiment of the present disclosure, further comprising: inputting the state information of the second state into the evaluation network for recursive calculation, outputting the second state value function corresponding to the second state, and determining the second state value function as the second target state value function corresponding to the optimized energy-saving strategy; determining the first target state value function based on the immediate reward and the second target state value function.

[0009] In an embodiment of the present disclosure, constructing the first loss function based on the evaluation state value function and the first target state value function, and evaluating the advantage function of the optimized energy-saving parameter threshold, comprises: bringing the evaluation state value function and the first target state value function into a mean square error function to obtain the first loss function; and constructing the advantage function based on the difference between the first target state value function and the evaluation state value function.

[0010] In an embodiment of the present disclosure, constructing the second loss function based on the advantage function, comprising: constructing an unconstrained update strength based on the advantage difference between the optimized energy-saving strategy and the energy-saving strategy before optimization and the advantage function; constructing a restriction term that limits the advantage difference within a specified range based on a clipping function, to construct a constrained update strength based on the restriction term and the advantage function; constructing the second loss function based on the operation of taking the negative expectation of the smaller value between the unconstrained update strength and the constrained update strength, wherein the unconstrained update strength is p1=exp(Y new -Y old )A(s t ,a t ), the constrained update strength is p2=Clamp(exp(Y new -Y old ),1-μ,1+μ)A(s t ,a t ), Y new is the optimized energy-saving strategy, Y old is the energy-saving strategy before optimization, s t is the state information of the first state, and at wherein, μ is a range parameter of the specified range, and the energy saving parameter threshold is an optimized energy saving parameter threshold.

[0011] In an embodiment of the present disclosure, the immediate reward of a state corresponding to state information of a second state after the optimized energy saving strategy is executed in a state corresponding to state information of a first state is calculated based on a reward and punishment function, including: determining an energy saving duration score corresponding to the optimized energy saving parameter threshold and a corresponding network quality and user perception comprehensive score based on state information of the second state and state information of the first state; inputting the energy saving duration score and the comprehensive score into the reward and punishment function to obtain the immediate reward, wherein the reward and punishment function is r(s,a) = E(mH t ,nG t |s t =s,a t =a;Y), s represents a state, a represents an executed action, s t is state information of the first state, a t is the optimized energy saving parameter threshold, H t represents the energy saving duration score, G t represents a comprehensive score of network quality and user perception corresponding to a t , m is a first weight, n is a second weight, E represents an expectation function, and Y represents an energy saving strategy.

[0012] In an embodiment of the present disclosure, the policy network and the evaluation network are jointly iteratively trained based on the first loss function and the second loss function, and the trained policy network is determined as a policy model to configure a target threshold of the cell energy saving parameter by the policy model, including: in one round of the iterative training, calculating a mean square error between an evaluation state value and a first target state value of the first state based on the first loss function; updating parameters of the evaluation network based on gradient descent minimization of the mean square error, and obtaining an updated first target state value and the evaluation state value; calculating an advantage value based on the updated first target state value and the evaluation state value; inputting the advantage value into the second loss function to obtain a policy gradient; updating parameters of the policy network based on descent maximization of the policy gradient to complete one round of iterative training; and continuing to perform iterative training until it is detected that the second state generated corresponding to the optimized energy saving parameter threshold output by the policy network reaches a target state to determine that the training is completed, so as to obtain the policy model.

[0013] In an embodiment of the present disclosure, the state information of the first state, the instant reward and the state information of the second state are input into the evaluation network to enable the evaluation network to construct the first state value function corresponding to the first state based on forward propagation, including: inputting the state information of the first state, the instant reward and the state information of the second state into an encoding layer for encoding processing and then splicing as an input vector of the evaluation network; inputting the input vector into the evaluation network, transmitting to a hidden layer through an input layer of the evaluation network, performing non-linear transformation on input features through an activation function of the hidden layer to extract key patterns related to state transition and rewards, and then transmitting the output of the hidden layer to an output layer for linear transformation, to construct the evaluation state value function of the first state based on the encoding layer, the input layer, the hidden layer and the output layer.

[0014] In an embodiment of the present disclosure, the state information includes an energy saving effect item, a network quality item and a user perception item of the energy saving strategy.

[0015] In an embodiment of the present disclosure, the energy saving strategy includes a deep sleep strategy, and corresponding parameter thresholds include a load threshold and / or a duration threshold; the energy saving strategy includes a channel shutdown strategy for redundant carriers of a multi-carrier aggregation cell, and corresponding parameter thresholds include a carrier utilization rate threshold and / or a service type threshold; the energy saving strategy includes a shutdown strategy for idle OFDM symbols, and corresponding parameter thresholds include a symbol idle rate threshold.

[0016] According to another aspect of the present disclosure, a cell energy saving parameter configuration device is provided, including: a first output module configured to input state information of a first state of the cell generated based on an energy saving strategy before optimization into a strategy network, the energy saving strategy before optimization being configured based on energy saving parameter thresholds before optimization, to output the energy saving parameter thresholds after optimization from the strategy network, the energy saving parameter thresholds after optimization being used to configure the energy saving strategy after optimization; a calculation module configured to calculate an instant reward of a second state transferred from the first state after executing the energy saving strategy after optimization based on a reward and punishment function; a second output module configured to input the state information of the first state, the instant reward and the state information of the second state into an evaluation network to enable the evaluation network to construct a first state value function corresponding to the first state based on forward propagation; a construction module configured to construct a first loss function of the evaluation network and a second loss function of the strategy network based on the first state value function; and a training module configured to jointly and iteratively train the strategy network and the evaluation network based on the first loss function and the second loss function, and determine the trained strategy network as a strategy model to configure target thresholds of the cell energy saving parameters by the strategy model.

[0017] According to still another aspect of the present disclosure, a network device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to perform the cell energy saving parameter configuration method of the first aspect above via executing the executable instructions.

[0018] According to yet another aspect of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, the computer program being executed by a processor to implement the cell energy saving parameter configuration method above.

[0019] According to yet another aspect of the present disclosure, a computer program product is provided, having stored thereon a computer program, the computer program being executed by a processor to implement the cell energy saving parameter configuration method above.

[0020] The cell energy saving parameter configuration scheme provided by the embodiments of the present disclosure, through the cooperative training mechanism of constructing a policy network and an evaluation network based on a PPO algorithm, configures a new policy based on the optimized energy saving parameter threshold output by the policy network, calculates an immediate reward for quantifying the short-term effect of policy adjustment from a first state to a second state through a reward and punishment function after executing the new policy, constructs a value function corresponding to the first state based on the first state information, the immediate reward and the second state information, respectively constructs a first loss function of the evaluation network and a second loss function of the policy network based on the value function, and iteratively trains the two networks through the two loss functions, so that the policy network continuously learns a more optimal parameter threshold configuration until it converges to a policy model that can be directly used for cell energy saving parameter optimization, which is conducive to achieving the balance optimization between energy saving effect and network quality, and further realizing the automatic optimization of large-scale base station cell energy saving parameter thresholds.

[0021] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] The drawings herein are incorporated into the specification and form a part of the specification, show embodiments consistent with the present disclosure, and together with the specification serve to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 A flow chart of a cell energy saving parameter configuration method in an embodiment of the present disclosure is shown;

[0024] Figure 2 A flow chart of another cell energy saving parameter configuration method in an embodiment of the present disclosure is shown;

[0025] Figure 3Fig. 3 shows a flow chart of another method for configuring cell energy saving parameters in an embodiment of the present disclosure;

[0026] Figure 4 Fig. 4 shows a flow chart of another method for configuring cell energy saving parameters in an embodiment of the present disclosure;

[0027] Figure 5 Fig. 5 shows a schematic diagram of a device for configuring cell energy saving parameters in an embodiment of the present disclosure;

[0028] Figure 6 Fig. 6 shows a structural block diagram of a computer device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth in this disclosure. Rather, these implementations are provided as example (s) so that this disclosure will be thorough and complete, and will fully convey the scope of the example implementations to those skilled in the art. Features described in the description, examples, or claims that are unified under a single header may be implemented in any suitable manner.

[0030] In addition, the accompanying drawings are only schematic and are non-limiting detailed descriptions of the disclosure. Identical reference signs in the drawings indicate identical or similar parts, and thus repeated descriptions of them will be omitted. Some of the block diagrams shown in the drawings are functional entities that do not necessarily have to correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] With the commercialization of the application scale of 5G technology, 5G base stations are intensively deployed, and the application of AI technology opens up a new path for intelligent, safe and efficient base station energy saving. Currently, 5G base stations mainly rely on energy saving strategies such as deep sleep, channel shutdown, and symbol shutdown to optimize energy consumption. The effectiveness of these strategies highly depends on the accurate configuration of energy saving parameter thresholds. Given the large scale of base station cell deployment and the significant differences in application scenarios and user groups, manually configuring threshold parameters for each base station cell would face huge labor cost pressure. If fixed energy saving parameters are used uniformly, it will be difficult to fully realize the energy saving potential. Therefore, it is necessary to introduce intelligent technologies such as reinforcement learning to build an automatic optimization mechanism for base station cell energy saving parameters, in order to break through the effectiveness bottleneck of traditional configuration mode.

[0032] In the following, each step of the cell energy saving parameter configuration method in the example implementation will be described in more detail in conjunction with the accompanying drawings and examples.

[0033] Figure 1 A flow chart of a cell energy saving parameter configuration method in an embodiment of the present disclosure is shown.

[0034] As shown, the cell energy saving parameter configuration method according to one embodiment of the present disclosure comprises: Figure 1

[0035] Step S102, input the state information of the first state of the cell generated based on the pre-optimization energy saving strategy into the strategy network, the pre-optimization energy saving strategy is configured based on the pre-optimization energy saving parameter threshold, and output the post-optimization energy saving parameter threshold from the strategy network, the post-optimization energy saving parameter threshold is used to configure the post-optimization energy saving strategy.

[0036] In some embodiments, the energy saving strategy refers to a specific operation scheme such as deep sleep, channel shutdown, etc. for achieving the cell energy saving target, and its execution needs to rely on the preset parameter boundary, i.e. the energy saving parameter threshold such as the load threshold, the duration threshold, etc.

[0037] In some embodiments, the energy saving parameter threshold can be the critical condition parameter for triggering the strategy to take effect when the 5G base station executes the energy saving strategies such as deep sleep, channel shutdown, symbol shutdown, etc.

[0038] In some embodiments, the strategy network is a kind of neural network, which is used to output the post-optimization energy saving parameter threshold based on the state information of the current state of the cell, i.e. the state information of the first state such as the energy saving effect, the network quality, etc., i.e. the strategy network reads the state such as the energy saving effect, the network quality and the user perception during the cell energy saving period, and outputs the post-optimization energy saving parameter threshold, and the network adjusts the optimized strategy according to the advantage function to evaluate the pros and cons of the current energy saving parameter threshold.

[0039] In some embodiments, the state information includes the actually executed energy saving parameters, the energy saving duration, the energy saving, the network KPI corresponding to the energy saving period, etc. and the data is automatically collected and reported by the base station network management.

[0040] Step S104, calculate the immediate reward of the second state transferred from the first state after executing the post-optimization energy saving strategy based on the reward and punishment function.

[0041] In some embodiments, the reward and punishment function is a function for quantifying the execution effect of the energy saving strategy, which can be constructed by configuring different reward and punishment weights for the state such as the energy saving effect, the network quality and the user perception, etc.

[0042] In some embodiments, the first state is the cell state after executing the pre-optimization strategy, and the second state is the new state after executing the post-optimization strategy, the difference between the two reflects the influence of the strategy adjustment, and the immediate reward is the short-term benefit calculated by the reward and punishment function after transferring from the first state to the second state, which directly reflects the pros and cons of the post-optimization strategy.

[0043] ​Step S106, inputting the state information of the first state, the immediate reward and the state information of the second state into the evaluation network to make the evaluation network construct the first state value function corresponding to the first state based on the forward propagation.

[0044] In some embodiments, the evaluation network reads the state during the cell energy saving period, calculates the evaluation function of the current state through forward propagation, thereby improving the decision performance of the policy network to solve the optimal energy saving parameters of the cell.

[0045] In some embodiments, the input of the evaluation network can include the state information of the first state, the current state feature, the immediate reward representing the short-term reward and the state information of the second state, i.e. the new state feature after the policy execution, which can be understood as the context for the network learning. Through forward propagation, the output is the estimation of the long-term cumulative reward that can be obtained by executing the optimized policy in the first state, i.e. the first state value function, which can also be understood as the evaluation function of the first state. The first state value function is used to provide a basis for constructing a subsequent loss function.

[0046] Step S108, constructing the first loss function of the evaluation network and the second loss function of the policy network based on the first state value function.

[0047] In some embodiments, the first loss function and the second loss function can be constructed based on the output of the evaluation network.

[0048] In some embodiments, the policy network and the evaluation network in the present disclosure can be modules in PPO (Proximal Policy Optimization), and the second loss function can be configured by a clipping objective function and an advantage function. The clipping objective function is used to limit the policy update amplitude, and the advantage function is used to evaluate the goodness of the parameter threshold.

[0049] Step S110, jointly and iteratively training the policy network and the evaluation network based on the first loss function and the second loss function, and determining the trained policy network as the policy model to configure the target threshold of the cell energy saving parameters by the policy model.

[0050] In some embodiments, the network parameter of the evaluation network is updated by gradient descent based on the first loss function to make the state value function more accurate, and then the state value is calculated based on the updated state value function, and the policy gradient is obtained by substituting the state value into the second loss function, and the policy network parameter is optimized by gradient descent to improve the quality of the optimized parameter threshold output by the policy, and the above process is repeated, and new samples are continuously collected and iterated until the second state corresponding to the parameter threshold output by the policy network reaches the target state, such as the balance and stability of energy saving effect and network quality, at which time the training is completed, and the policy network can be used as a policy model to configure the target threshold of the cell energy saving parameter.

[0051] In this embodiment, the collaborative training mechanism of the policy network and the evaluation network is constructed based on the PPO algorithm, the optimized energy saving parameter threshold output by the policy network is used to configure a new policy, and after the new policy is executed, the immediate reward for quantifying the short-term effect of policy adjustment from the first state to the second state is calculated by the reward and punishment function, the value function corresponding to the first state is constructed based on the first state information, the immediate reward and the second state information, the first loss function of the evaluation network and the second loss function of the policy network are constructed based on the value function, and the double network is trained by the two loss functions to make the policy network learn better parameter threshold configuration until it converges to a policy model that can be directly used for cell energy saving parameter optimization, which is beneficial to achieve the balance optimization between energy saving effect and network quality, and further realize the automatic optimization of large-scale base station cell energy saving parameter threshold.

[0052] In one embodiment of the present disclosure, the first state value function includes an evaluation state value function of the first state and a first target state value function, and the first loss function of the evaluation network and the second loss function of the policy network are constructed based on the first state value function, including:

[0053] The first loss function is constructed based on the evaluation state value function and the first target state value function, and the advantage function of the optimized energy saving parameter threshold is evaluated.

[0054] In some embodiments, the principle of constructing the first loss function is to minimize the prediction error of the evaluation network, and the loss is optimized by gradient descent to make the evaluation network more accurately estimate the state value.

[0055] The second loss function is constructed based on the advantage function.

[0056] In some embodiments, the advantage function is shown in formula (1).

[0057] A(s t ,a t )=-E(X target (s t )-X(s t )) (1)

[0058] wherein X(s t ) is an evaluation state value function, X target (s t ) is a first target state value function.

[0059] The advantage function is used to measure the value gain of the strategy a t corresponding to the optimized energy-saving parameter threshold in state s t compared with the average strategy, if A(s t , a t ) is greater than 0, it indicates that the strategy is above the average level, thereby providing a clear value gradient for strategy optimization.

[0060] In this embodiment, by constructing the loss function of the evaluation state value and the target state value, the cooperative optimization of the evaluation network and the policy network is realized, the evaluation network accurately estimates the state value through the first loss function to provide a reliable benchmark for policy optimization, and the policy network adjusts the energy-saving parameter threshold in the direction of higher value gain based on the second loss function of the advantage function, thereby being conducive to improving the optimization efficiency and stability of the energy-saving strategy.

[0061] In one embodiment of the present disclosure, further comprising: inputting the state information of the second state into the evaluation network for recursive calculation, outputting a second state value function corresponding to the second state, and determining the second state value function as a second target state value function corresponding to the optimized energy-saving strategy; and determining the first target state value function based on the immediate reward and the second target state value function.

[0062] In some embodiments, the first target state value function is as shown in equation (2).

[0063] X target (s t ) = r t + X target (s t+1 ) (2)

[0064] wherein r t is an immediate reward, and X target (s t+1 ) is a second target state value function.

[0065] In some embodiments, the second state is the result of the optimized strategy executed by the first state, and its corresponding second state value function needs to be recursively calculated through the evaluation network. The evaluation network first takes the state information of the second state as input, outputs an estimate of the long-term value of the state, and determines it as the second target state value function. On this basis, the calculation of the first target state value function follows the logic of immediate reward + future value, that is, the target value after the first state executes the strategy not only includes the immediate reward currently obtained, but also needs to be superimposed with the target value of the second state, so that the first target state value function can more comprehensively reflect the long-term effect of the strategy, rather than being limited to short-term benefits, providing a benchmark that is more in line with actual scenarios for the training of the evaluation network and the policy network.

[0066] In one embodiment of the present disclosure, a first loss function and an advantage function of the energy-saving parameter threshold after evaluation optimization are constructed based on the evaluation state value function and the first target state value function, including: substituting the evaluation state value function and the first target state value function into the mean square error function to obtain the first loss function; and constructing an advantage function based on the difference between the first target state value function and the evaluation state value function, as shown in formula (2).

[0067] In some embodiments, the first loss function is as shown in formula (3).

[0068] Loss X (s t )=E(X target (s t )-X(s t )) 2 (3)

[0069] In this embodiment, the evaluation state value function is the evaluation network's current estimate of the first state value, while the first target state value function is recursively calculated based on the immediate reward and subsequent state value, and is a target value that is more in line with the actual expectation of long-term cumulative rewards. Substituting both into the mean square error function can quantify the deviation between the evaluation network's estimate and the target value. By minimizing the loss function, the parameters of the evaluation network can be continuously corrected to improve the accuracy of its estimate of the state value, thereby providing a reliable value benchmark for subsequent strategy optimization.

[0070] like Figure 2 As shown, in one embodiment of the present disclosure, constructing a second loss function based on the advantage function includes:

[0071] Step S202 : constructing an unconstrained update intensity based on the advantage difference and advantage function of the optimized energy-saving strategy relative to the unoptimized energy-saving strategy.

[0072] In some embodiments, the construction principle of the unconstrained update intensity is based on the policy gradient theory, and the core is to measure the improvement degree of the optimized energy-saving policy relative to the energy-saving policy before optimization.

[0073] In some embodiments, the unconstrained update intensity is shown in formula (4).

[0074] p1=exp(Y new -Y old )A(s t ,a t ) (4)

[0075] Step S204, constructing a restriction term that limits the advantage difference within a specified range based on the clipping function, to construct the constrained update intensity based on the restriction term and the advantage function.

[0076] In some embodiments, since the energy-saving parameter threshold does not allow too large changes, too large changes may not only cause the policy network training to diverge, but also may affect the energy-saving effect and network KPI balance of the existing network, so it is necessary to limit the sampling ratio of the new and old policies within a certain range.

[0077] In some embodiments, the introduction of the constrained update intensity is to prevent the training from being unstable due to too large policy update amplitude, and the clipping function limits the probability ratio of the new and old policies within the range of [1-μ, 1+μ], and generates a restriction term, when the probability ratio exceeds the range, the constrained update intensity is truncated. This clipping mechanism not only guarantees the optimization direction, but also prevents performance shocks caused by excessive exploration, and is especially suitable for scenarios such as cell energy saving that require smooth transition.

[0078] In some embodiments, the constrained update intensity is shown in formula (5).

[0079] p2=Clamp(exp(Y new -Y old ),1-μ,1+μ)A(s t ,a t ) (5)

[0080] Wherein, Y new is the optimized energy-saving policy, Y old is the energy-saving policy before optimization, s t is the state information of the first state, a t is the energy-saving parameter threshold after optimization, and μ is the range parameter of the specified range.

[0081] Step S206, constructing a second loss function based on the operation of taking the negative expectation of the smaller value between the unconstrained update intensity and the constrained update intensity.

[0082] In some embodiments, the second loss function is shown in formula (6).

[0083] Loss Y (s t ,a t )=-E(min(p1,p2)) (6)

[0084] In this embodiment, the second loss function takes the smaller value of the unconstrained update strength and the constrained update strength, that is, when the unconstrained update strength is reasonable, the unconstrained update is directly used, and the optimization signal provided by the advantage function is fully utilized, and when the unconstrained update strength is too large, the constrained update is switched to, and the update step is forced to be limited, so as to ensure the smooth iteration of the policy.

[0085] In one embodiment of the present disclosure, the instant reward of the state corresponding to the state information of the second state after the energy-saving policy optimized in the state corresponding to the state information of the first state is executed is calculated based on a reward-punishment function, and the method comprises the following steps:

[0086] Based on the state information of the second state and the state information of the first state, the energy-saving duration score corresponding to the optimized energy-saving parameter threshold value and the corresponding network quality and user perception comprehensive score are determined; and the energy-saving duration score and the comprehensive score are input into the reward-punishment function to obtain the instant reward.

[0087] In some embodiments, the reward-punishment function is as shown in formula (7).

[0088] r(s,a)=E(mH t ,nG t |s t =s,a t =a;Y) (7)

[0089] Wherein, s represents the state, a represents the executed action, s t is the state information of the first state, a t is the optimized energy-saving parameter threshold value, H t represents the energy-saving duration score, G t represents the network quality and user perception comprehensive score corresponding to a t , m is the first weight, n is the second weight, E represents the expectation function, and Y represents the energy-saving policy.

[0090] In some embodiments, based on the energy-saving situation and the cell network KPI situation at the current moment, the reward-punishment function r(s,a) is calculated, which is the instant reward after the a t energy-saving parameter threshold value of the model policy network optimization executed in the cell state s t .

[0091] In this embodiment, the energy saving effect of the optimized strategy is evaluated by comparing the energy consumption data of the first state and the second state, such as device sleep time, power reduction amplitude, etc., as the energy saving time score, combined with network performance indicators such as signal strength, packet loss rate, delay, etc., and user experience indicators such as complaint rate and connection success rate, etc., to measure the impact of the strategy on the quality of service as the comprehensive score of network quality and user perception, and to construct a reward and punishment function in the form of weighted expectation, to adjust the priority of energy saving effect and service quality through parameters m and n, to get the balance value of energy saving benefit and quality loss, so as to adaptively balance energy saving and service quality in different scenarios, especially suitable for small cell environment with dynamic load.

[0092] As shown in Figure 3 In one embodiment of the present disclosure, the strategy network and the evaluation network are jointly iteratively trained based on the first loss function and the second loss function, and the trained strategy network is determined as the strategy model to configure the target threshold of the cell energy saving parameters by the strategy model, including:

[0093] In step S302, in one round of iterative training, the mean square error between the evaluation state value of the first state and the first target state value is calculated based on the first loss function.

[0094] In some embodiments, the evaluation state value of the first state is the output value of the evaluation state value function, and the first target state value is the output value of the first target state value function.

[0095] In step S304, the parameters of the evaluation network are updated based on the gradient descent minimization of the mean square error, and the updated first target state value and evaluation state value are obtained.

[0096] In some embodiments, the first loss function is constructed based on the mean square error between the evaluation state value (the value prediction of the current state by the evaluation network) and the first target state value (the true target value based on the immediate reward and the future state value). In each iteration, the gradient descent method is used to minimize this error, so that the evaluation network learns a more accurate state value judgment, that is, by continuously correcting the deviation between the predicted value and the target value, the evaluation network can more reliably evaluate the cumulative benefits that can be obtained in the future after executing a certain strategy in the current state. The updated evaluation state value and the first target state value will provide more accurate basic data for the calculation of the advantage value.

[0097] In step S306, the advantage value is calculated based on the updated first target state value and the evaluation state value.

[0098] In some embodiments, the advantage value can be determined by the difference between the updated first target state value and the evaluation state value, which identifies the additional benefit of executing the current optimized energy saving strategy compared to the average strategy. For example, if the advantage value of a certain energy saving parameter threshold is positive, it means that the strategy is better than the average performance of the current strategy network and should be prioritized. If it is negative, it needs to be avoided.

[0099] In step S308, the advantage value is brought into the second loss function to obtain a policy gradient.

[0100] In step S310, the parameters of the policy network are updated based on the descent maximization of the policy gradient to complete a round of iterative training.

[0101] In some embodiments, the second loss function is used to guide the parameter update of the policy network using the advantage value. After the advantage value is substituted into the second loss function, the policy gradient obtained represents the parameter adjustment direction that makes the policy network more inclined to select high-advantage actions. By gradient descent, that is, gradient ascent of the advantage maximization, the parameters of the policy network are updated, which can make the energy saving parameter threshold output by the policy network gradually optimized in the direction of high energy saving benefit and low quality loss. For example, if a certain parameter threshold can significantly prolong the energy saving time, that is, high H t , and has little effect on network quality, that is, high G t , the advantage value will be higher, and the policy network will output this parameter with a higher probability.

[0102] In step S312, the iterative training is continued until it is detected that the optimized energy saving parameter threshold output by the policy network corresponds to the generated second state reaching the target state, and the training is completed, to obtain a policy model.

[0103] In some embodiments, the iterative process continues until the optimized parameter threshold output by the policy network can stably transfer the system from the first state to the preset target state, such as a state in which the energy saving efficiency meets the standard and the network quality and user perception meet the minimum requirements. At this time, the policy network has learned the optimal energy saving strategy in this scenario, and the training is terminated. The final policy model can be directly used to configure the target threshold of the cell energy saving parameter.

[0104] In this embodiment, by quantifying the policy benefit through the evaluation value and the advantage value of the evaluation network and the closed-loop processing of the policy network following the advantage optimization action, the generated policy model can achieve the dynamic balance between energy saving and service quality, and is particularly suitable for complex scenarios such as cells that need to consider multiple targets, i.e., energy saving, network stability, and user experience.

[0105] In one embodiment of the present disclosure, the state information of the first state, the immediate reward, and the state information of the second state are input into the evaluation network to enable the evaluation network to construct a first state value function corresponding to the first state based on forward propagation, including:

[0106] The state information of the first state, the immediate reward and the state information of the second state are input into the encoding layer for encoding processing, and then spliced as an input vector of the evaluation network; the input vector is input into the evaluation network, and is transmitted to the hidden layer through the input layer of the evaluation network, the hidden layer performs nonlinear transformation on the input features through the activation function to extract the key pattern of the association between the state transition and the reward, and then the output of the hidden layer is transmitted to the output layer for linear transformation, so as to construct the evaluation state value function of the first state based on the encoding layer, the input layer, the hidden layer and the output layer.

[0107] In some embodiments, the first state value function is specifically the evaluation state value function.

[0108] In some embodiments, the state information of the first state, such as the device running parameters, user load and the like of the current cell, the immediate reward, such as the short-term income obtained after executing the strategy, such as the energy saving effect, the network quality score and the like, and the state information of the second state, that is, the new state features transferred after executing the strategy, are the data describing the cycle of state, action, feedback and new state, but there may be differences in data types or dimensions. Through encoding processing, these information can be converted into a unified vector form, and then integrated into an input vector through splicing operation, so that the network can simultaneously receive and associate multi-dimensional information such as state change and immediate feedback.

[0109] In some embodiments, after the input vector is transmitted to the hidden layer through the input layer of the evaluation network, the hidden layer performs nonlinear transformation on the input features through the activation function to learn complex rules, and through the superposition of multiple hidden layers, the network can gradually abstract higher-level features.

[0110] In some embodiments, after the high-level features output by the hidden layer are transmitted to the output layer, the output layer performs linear transformation to obtain the first state value function which can map the features to a specific numerical value.

[0111] In this embodiment, the encoding layer converts and integrates the state information of the first state, the immediate reward and the state information of the second state to generate an input vector in a unified format, the input layer transmits the vector to the hidden layer, the hidden layer uses the activation function to perform nonlinear transformation on the features to mine the deep association pattern between the state transition and the reward, and the output layer maps these abstract features to specific value numerical values through linear transformation. The entire network collectively constitutes the evaluation state value function to provide a value benchmark for strategy optimization, and the value estimation value output by the function gradually approaches the real long-term cumulative income, thereby guiding the strategy network to generate more optimal energy saving parameter thresholds.

[0112] As Figure 4The energy saving parameters that need to be optimized are shown, and the upper and lower limits of the threshold of the actual load service are defined, and the data required to calculate the energy saving parameter reward and punishment function is determined, such as the actual energy saving parameters of the cell, the energy saving time, the energy saving, the network KPI corresponding to the energy saving time, and other data. During the operation of the cell, the state sequence <s t , t , t , t+1 , t is the current state, a t is the executed energy saving parameter threshold strategy, r t is the reward, s t+1 is the new state after execution, the policy network takes the current cell state s t , including energy saving effect, network quality, user perception, etc. as input, and outputs the optimized energy saving parameter threshold through the neural network architecture.

[0113] The evaluation network also reads the cell state s t , calculates the current state value function X(s t ) through forward propagation, i.e. input layer-hidden layer-output layer feature transformation, combines the target value X target (s t ), and calculates the Loss X (s t ) = E(X target (s t )-X(s t )) 2 , and updates the parameters of the evaluation network in the reverse direction. At the same time, based on the value function, the advantage function A(s t ,a t ) is output, which tells the policy network that the current output parameter threshold strategy is good or bad, and the revenue is high or low, which helps the policy network adjust the optimization direction.

[0114] Further, after executing the policy a t , the cell generates a reward r t , and the policy network outputs a new policy Y new and the old policy Y old , and the advantage function A(s t ,a t ) of the evaluation network is used to calculate the Loss Y (s t ,a t ) = -E(min(p1, p2)), and the parameters of the evaluation network are updated in the reverse direction. In this way, the policy network gradually learns the parameter threshold strategy that can balance energy saving, network quality, user perception and long-term revenue optimization, and finally obtains a stable and high-quality energy saving parameter configuration.

[0115] In this embodiment, by improving the policy network loss function and the neural network architecture of the PPO algorithm, an energy saving parameter optimization reward and punishment function of the base station is constructed, the model parameters are trained and optimized, so that the model has the self-optimization capability of the base station energy saving parameter, can dynamically adjust the base station energy saving parameter threshold under the premise of meeting the network perception index, and maximize the energy saving efficiency.

[0116] In one embodiment of the present disclosure, the state information includes an energy saving effect item, a network quality item and a user perception item of the energy saving strategy.

[0117] In some embodiments, the energy saving effect item includes but is not limited to device sleep duration, power adjustment amplitude, and energy saving amount, etc.

[0118] In some embodiments, the network quality item includes but is not limited to signal strength, packet loss rate, network delay and throughput, etc.

[0119] In some embodiments, the user perception item includes but is not limited to connection success rate, complaint rate and service interruption times, etc.

[0120] In one embodiment of the present disclosure, the energy saving strategy includes a deep sleep strategy, and the corresponding parameter threshold includes a load threshold and / or a duration threshold.

[0121] In this embodiment, the policy network takes the current load of the cell, the historical sleep effect, the network quality and other state information as input, and outputs the optimized load threshold, such as the minimum number of users triggering sleep, and the duration threshold, such as the maximum sleep duration; evaluates the energy saving effect of the network based on the execution of the new strategy, such as the energy consumption reduction ratio, the network quality, such as signal strength fluctuation, and the user perception, such as the change of connection success rate, calculates the immediate reward, constructs the target function through the recursive calculation of the subsequent state value, constructs the loss function combined with the advantage function and the pruning mechanism, and trains the double networks jointly. For example, when the cell load is low and the future traffic is smooth, the algorithm will increase the load threshold and prolong the sleep time, if the network quality does not decrease significantly, a positive reward is given, otherwise the parameters are adjusted and punished, and finally the parameter configuration that can maximize the sleep energy saving time and guarantee the network quality and user experience is converged.

[0122] In one embodiment of the present disclosure, the energy saving strategy includes a channel shutdown strategy for redundant carriers of a cell with multi-carrier aggregation, and the corresponding parameter threshold includes a carrier utilization rate threshold and / or a service type threshold.

[0123] In this embodiment, the policy network outputs the optimal carrier utilization threshold based on the current traffic distribution of each carrier, the proportion of service types such as voice, video, and data service, and other state information, for example, shutting down the carrier when the threshold is lower than the threshold, and the service type threshold, for example, the carrier for priority reservation of high-priority services; the evaluation network calculates the energy saving benefits after the execution of the shutdown strategy, such as the reduction of redundant carrier energy consumption, network quality changes such as the throughput reduction amplitude, and user perception impact such as the high-priority service interruption rate; the recursive state value calculation and advantage function are used to guide the policy update, and the parameter adjustment step is limited by the pruning mechanism to prevent violent fluctuations. For example, when the carrier utilization is lower than the threshold for a long time and most of the low-priority services are carried, the algorithm will shut down the carrier, and if the user complaint rate does not rise, the decision will be strengthened, otherwise the parameters will be recalled; finally, the parameter configuration of maximizing the shutdown of redundant carriers is learned on the premise of guaranteeing the quality of service of various types of services.

[0124] In one embodiment of the present disclosure, the energy saving strategy includes a shutdown strategy for idle OFDM symbols, and the corresponding parameter threshold includes a symbol idle rate threshold.

[0125] In this embodiment, the policy network takes the symbol idle rate of the current time slot, the adjacent time slot load prediction, and the channel quality as input, and outputs the optimal symbol idle rate threshold, for example, shutting down the symbol when the threshold is higher than the threshold; the evaluation network calculates the immediate reward based on the energy saving effect after the shutdown, such as reducing the radio frequency power consumption, the signal quality change such as the bit error rate rising, and the service impact such as the data transmission delay; the two networks are jointly trained to make the policy network continuously adjust the threshold to balance energy saving and communication quality. For example, when it is detected that the symbol idle rate of a time slot is continuously higher than the threshold and does not significantly affect signal demodulation after being shut down, the algorithm will increase the threshold to further save energy; if the bit error rate increases sharply or the service is interrupted, the threshold will be reduced and punished; finally, the parameter configuration of maximizing the shutdown of idle symbols is converged while guaranteeing the reliability of OFDM symbol transmission.

[0126] It should be noted that the above-described figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not for limiting purposes. It is easy to understand that the processes shown in the above-described figures do not indicate or limit the time sequence of the processes. In addition, it is also easy to understand that the processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0127] The cell energy saving parameter configuration apparatus 500 according to the embodiments of the present disclosure will be described below with reference to Figure 5 Figure 5 The cell energy saving parameter configuration apparatus 500 shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0128] ​The cell energy saving parameter configuration apparatus 500 is in the form of a hardware module. The components of the cell energy saving parameter configuration apparatus 500 can include, but are not limited to: a first output module 502 configured to input state information of a first state of a cell generated based on a pre-optimization energy saving strategy into a policy network, the pre-optimization energy saving strategy being based on pre-optimization energy parameter threshold configuration, to output, by the policy network, post-optimization energy parameter threshold, the post-optimization energy parameter threshold being used to configure a post-optimization energy saving strategy; a calculation module 504 configured to calculate, based on a reward and punishment function, an immediate reward of a second state transferred from the first state after execution of the post-optimization energy saving strategy in the first state; a second output module 506 configured to input the state information of the first state, the immediate reward, and the state information of the second state into an evaluation network, to enable the evaluation network to construct a first state value function corresponding to the first state based on forward propagation; a construction module 508 configured to construct a first loss function of the evaluation network and a second loss function of the policy network based on the first state value function; and a training module 510 configured to jointly and iteratively train the policy network and the evaluation network based on the first loss function and the second loss function, and determine the trained policy network as a policy model, to configure, by the policy model, a target threshold of a cell energy saving parameter.

[0129] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method or a program product. Therefore, various aspects of the present disclosure can be embodied as a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0130] The electronic device 600 according to this embodiment of the present disclosure will be described below with reference to Figure 6 It can be a network device or a terminal. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0131] As Figure 6 shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to: the above-mentioned at least one processing unit 610, the above-mentioned at least one storage unit 620, and a bus 630 connecting different system components, including the storage unit 620 and the processing unit 610.

[0132] The storage unit stores program code which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of the present specification. For example, the processing unit 610 can perform the schemes as Figure 1 described.

[0133] Storage 620 can include a readable medium in the form of volatile memory, such as random access memory (RAM) 6201 and / or cache memory 6202, and can further include non-volatile memory, such as read only memory (ROM) 6203.

[0134] Storage 620 can also include a program / utility 6204 having a set of program modules 6205 such as an operating system, one or more application programs, other program modules, and program data, each of which can give the electronic device its functionality, at least in part. Each of the operating system, one or more application programs, other program modules, and program data can include implementations of a web environment, either alone or in some combination.

[0135] Bus 630 can represent one or more of several types of bus structures, including a storage bus or bus controller, peripheral bus, graphics bus, processor or local bus using any of a variety of bus architectures.

[0136] Electronic device 600 can also communicate with one or more external devices 670 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices that enable a user to interact with electronic device 600; and / or any devices (e.g., a router, a modem, a printer, etc.) that enable electronic device 600 to communicate with one or more other computing devices. Such communication can occur via input / output (I / O) interface 650. Still yet, electronic device 600 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, via network adapter 660. As depicted, network adapter 660 can be communicatively coupled to the other components of electronic device 600 via bus 630. It should be appreciated that the many of the components depicted herein can be implemented in software and / or firmware and / or hardware, and that the depicted examples are merely illustrative. Accordingly, results of methods and techniques disclosed herein can be stored in one or more of the components depicted and / or described herein, including, but not limited to, RAM 6201, cache memory 6202, ROM 6203, and / or storage 620.

[0137] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or firmware, and / or by software in combination with necessary hardware. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0138] In exemplary embodiments of the present disclosure, a computer readable storage medium having stored thereon a program product capable of implementing the above-described methods of the present specification is also provided. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product including a program code that, when run on an electronic device, causes the electronic device to perform the steps described in the above "Exemplary Methods" section according to various exemplary embodiments of the present disclosure.

[0139] A program product for implementing the above-described methods according to embodiments of the present disclosure can take the form of a portable compact disc read-only memory (CD-ROM) and include a program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in the present document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0140] The program product can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0141] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the readable program code is embodied. Such propagated data signal can take multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a readable storage medium and that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0142] The program code contained on the readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, and the like, or any suitable combination thereof.

[0143] Program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0144] It should be noted that although several modules or units of the devices for action execution are mentioned in the above detailed description, such division is not mandatory. Indeed, features and functionalities of two or more modules or units described above can be embodied in one module or unit according to embodiments of the present disclosure. Conversely, features and functionalities of one module or unit described above can be further divided into multiple modules or units for embodiment. Moreover, although various steps of the methods in the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in that particular order, or that all of the illustrated steps must be performed to achieve the desired results. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, one step can be split into multiple steps, and the like.

[0145] From the above description of the embodiments, those skilled in the art will readily perceive that the example embodiments described herein can be implemented by software and / or by hardware and / or by a combination of software and hardware. Accordingly, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium, such as a CD-ROM, a USB flash drive, a mobile hard disk, or the like, or on a network, and includes a number of instructions for causing a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure. Those skilled in the art will readily perceive other embodiments of the present disclosure upon consideration of the specification and the accompanying drawings. The present application is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including modifications and equivalents that are obvious to those skilled in the art. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for configuring cell energy-saving parameters, characterized in that: include: inputting state information of a first state of the cell generated based on a pre-optimized energy-saving strategy into a policy network, wherein the pre-optimized energy-saving strategy is configured based on a pre-optimized energy-saving parameter threshold, so that the policy network outputs the optimized energy-saving parameter threshold, and the optimized energy-saving parameter threshold is used to configure the optimized energy-saving strategy; Calculating, based on a reward and penalty function, an immediate reward for executing the optimized energy-saving strategy in the first state and transferring to the second state; Inputting the state information of the first state, the instant reward, and the state information of the second state into an evaluation network, so that the evaluation network constructs a first state value function corresponding to the first state based on forward propagation; Constructing a first loss function of the evaluation network and a second loss function of the policy network based on the first state-value function; The policy network and the evaluation network are jointly and iteratively trained based on the first loss function and the second loss function, and the trained policy network is determined as a policy model, so that the target threshold of the cell energy-saving parameter is configured by the policy model.

2. The method for configuring cell energy-saving parameters according to claim 1, wherein: The first state value function includes an evaluation state value function and a first target state value function of the first state, and constructing a first loss function of the evaluation network and a second loss function of the policy network based on the first state value function includes: Constructing the first loss function and the advantage function of the energy-saving parameter threshold after evaluation optimization based on the evaluation state value function and the first target state value function; The second loss function is constructed based on the advantage function.

3. The method for configuring cell energy-saving parameters according to claim 2, characterized in that: Also includes: Inputting the state information of the second state into the evaluation network for recursive calculation, outputting a second state value function corresponding to the second state, and determining the second state value function as the second target state value function corresponding to the optimized energy-saving strategy; The first target state-value function is determined based on the immediate reward and the second target state-value function.

4. The method for configuring cell energy-saving parameters according to claim 2, wherein: Constructing the first loss function and the advantage function of the energy-saving parameter threshold after evaluation optimization based on the evaluation state value function and the first target state value function includes: Substituting the evaluation state value function and the first target state value function into the mean square error function to obtain the first loss function; and The advantage function is constructed based on the difference between the first target state value function and the evaluation state value function.

5. The method for configuring cell energy-saving parameters according to claim 2, wherein: Constructing the second loss function based on the advantage function includes: constructing an unconstrained update intensity based on the advantage difference between the optimized energy-saving strategy and the energy-saving strategy before optimization and the advantage function; constructing a restriction term for limiting the advantage difference within a specified range based on a clipping function, and constructing a constraint update strength based on the restriction term and the advantage function; The second loss function is constructed based on the operation of taking a negative expectation of the smaller value of the unconstrained update strength and the constrained update strength, Wherein, the unconstrained update strength is p1=exp(Y new -Y old )A(s t ,a t ), the constraint update strength is p2 = Clamp(exp(Y new -Y old ),1-μ,1+μ)A(s t ,a t ), Y new is the optimized energy-saving strategy, Y old is the energy-saving strategy before optimization, s t is the state information of the first state, a t is the optimized energy-saving parameter threshold, and μ is the range parameter of the specified range.

6. The method for configuring cell energy-saving parameters according to claim 1, characterized in that: Calculating, based on a reward and penalty function, an immediate reward for executing the optimized energy-saving strategy in a state corresponding to the state information of the first state and transferring to a state corresponding to the state information of the second state, includes: Determining, based on the state information of the second state and the state information of the first state, an energy-saving duration score corresponding to the optimized energy-saving parameter threshold, and a corresponding comprehensive score of network quality and user perception; Input the energy-saving duration score and the comprehensive score into the reward and punishment function to obtain the instant reward, The reward and punishment function is r(s,a)=E(mH t ,nG t |s t =s,a t =a; Y), s represents the state, a represents the execution action, s t is the state information of the first state, a t is the optimized energy-saving parameter threshold, H t Characterize the energy saving time score, G t Representation and a t The corresponding comprehensive score of network quality and user perception, m is the first weight, n is the second weight, E represents the expectation function, and Y represents the energy-saving strategy.

7. The method for configuring cell energy-saving parameters according to claim 1, characterized in that: Iteratively training the policy network and the evaluation network based on the first loss function and the second loss function, and determining the trained policy network as a policy model, so as to configure a target threshold value of the cell energy-saving parameter by the policy model, including: In one round of the iterative training, calculating a mean square error between an evaluation state value of the first state and a first target state value based on the first loss function; updating the parameters of the evaluation network based on the gradient descent minimization of the mean square error, and obtaining the updated first target state value and the evaluation state value; Calculating an advantage value based on the updated first target state value and the evaluation state value; Substitute the advantage value into the second loss function to obtain a policy gradient; Updating the parameters of the policy network based on the descent maximization of the policy gradient to complete a round of iterative training; The iterative training is continued until it is detected that the second state generated corresponding to the optimized energy-saving parameter threshold output by the policy network reaches the target state, and the training is determined to be completed to obtain the policy model.

8. The method for configuring cell energy-saving parameters according to claim 2, characterized in that: Inputting the state information of the first state, the immediate reward, and the state information of the second state into an evaluation network, so that the evaluation network constructs a first state value function corresponding to the first state based on forward propagation, including: Inputting the state information of the first state, the instant reward, and the state information of the second state into the encoding layer for encoding processing and then splicing them as the input vector of the evaluation network; The input vector is input into the evaluation network and passed to the hidden layer through the input layer of the evaluation network. The hidden layer performs a nonlinear transformation on the input features through an activation function to extract the key pattern of state transition and reward association. The output of the hidden layer is then passed to the output layer for linear change to construct an evaluation state value function of the first state based on the encoding layer, the input layer, the hidden layer and the output layer.

9. The method for configuring cell energy-saving parameters according to claim 1, wherein: The status information includes an energy-saving effect item of the energy-saving strategy, a network quality item, and a user perception item.

10. The method for configuring cell energy-saving parameters according to claim 1, characterized in that: The energy-saving strategy includes a deep sleep strategy, and the corresponding parameter threshold includes a load threshold and / or a duration threshold; The energy-saving strategy includes a channel shut-down strategy for redundant carriers of a cell in which multiple carriers are aggregated, and the corresponding parameter thresholds include a carrier utilization threshold and / or a service type threshold; The energy-saving strategy includes a shutdown strategy for idle OFDM symbols, and the corresponding parameter threshold includes a symbol idle rate threshold.

11. A cell energy-saving parameter configuration device, characterized in that: include: a first output module, configured to input state information of a first state of the cell generated based on a pre-optimization energy-saving policy into a policy network, wherein the pre-optimization energy-saving policy is configured based on a pre-optimization energy-saving parameter threshold, so that the policy network outputs the optimized energy-saving parameter threshold, and the optimized energy-saving parameter threshold is used to configure the optimized energy-saving policy; A calculation module, configured to calculate, based on a reward and penalty function, an immediate reward for executing the optimized energy-saving strategy in the first state and transferring to the second state; a second output module, configured to input the state information of the first state, the instant reward, and the state information of the second state into an evaluation network, so that the evaluation network constructs a first state value function corresponding to the first state based on forward propagation; A construction module, configured to construct a first loss function of the evaluation network and a second loss function of the policy network based on the first state-value function; A training module is used to jointly iteratively train the policy network and the evaluation network based on the first loss function and the second loss function, and determine the trained policy network as a policy model so that the target threshold of the cell energy-saving parameter is configured by the policy model.

12. A network device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the cell energy-saving parameter configuration method according to any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for configuring cell energy-saving parameters according to any one of claims 1 to 10 is implemented.

14. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for configuring cell energy-saving parameters according to any one of claims 1 to 10 is implemented.

Images