Power Internet of Things terminal access parameter adaptive optimization method and system based on hierarchical reinforcement learning

Abstract

The invention discloses a power Internet of Things terminal access parameter adaptive optimization method and system based on hierarchical reinforcement learning, and the method comprises the steps: carrying out the optimization modeling of a terminal access parameter into a hierarchical reinforcement learning decision process, and carrying out the access strategy selection and parameter fine-grained adjustment through a high-layer intelligent agent and a low-layer intelligent agent, and in combination with Monte Carlo sampling, simulation evaluation is carried out on a dynamic network environment, so that online adaptive optimization of access parameters of the terminal equipment is realized. In consideration of diversity and complexity faced by equipment access in an electric power internet of things system, the access parameters of the equipment are optimized in real time by dynamically analyzing the running state and service requirements of the terminal equipment and combining five modules including a sampler, a parameter judgment device, an object model state machine, a benchmark tester and a reward model. By adopting the method provided by the invention, challenges such as terminal large-scale access and heterogeneous network environment dynamic change in the power internet of things can be effectively handled, and the terminal access reliability, the network throughput and the communication efficiency are remarkably improved.

Description

An Adaptive Optimization Method and System for Power Internet of Things Terminal Access Parameters Based on Hierarchical Reinforcement Learning Technical Field

[0001] This invention belongs to the field of power Internet of Things (IoT) terminal equipment access optimization technology, specifically involving a multi-parameter tuning method that combines Monte Carlo algorithm with hierarchical reinforcement learning (HRL) to achieve adaptive optimization configuration of terminal equipment access parameters in complex power IoT environments. Background Technology

[0002] As power systems evolve towards digitalization and ubiquitous IoT, a large number of smart terminal devices are connecting to the power IoT, leading to a dramatic increase in data traffic and connections. Existing communication networks and traditional parameter configuration strategies are no longer sufficient to meet the power IoT's demands for high reliability, low latency, and high bandwidth. Terminal access scenarios involve optimizing the configuration of numerous parameters, such as the terminal's access frequency, transmit power, channel selection, and data reporting interval. The values ​​of these parameters directly impact performance indicators such as communication latency, throughput, reliability, and energy consumption. However, if these key parameters are configured with fixed or default settings, it is often difficult to achieve globally optimal performance in the complex and ever-changing IoT environment. For example, in large-scale messaging systems within an AIoT (Artificial Intelligence of Things) environment, numerous terminals exchange messages at varying frequencies; default parameters cannot support optimal system throughput, necessitating adaptive parameter optimization to improve performance. Similarly, in scenarios involving large-scale access to power IoT terminals, the traditional approach of relying on manual experience or static strategies for parameter tuning has significant shortcomings: First, manual configuration is difficult to adjust in a timely manner according to environmental changes, which may lead to network congestion or resource waste; Second, the parameters have high dimensionality and complex mutual influences, and the "curse of dimensionality" makes exhaustive search impractical, and traditional optimization algorithms (such as grid search, genetic algorithms, etc.) are prone to getting stuck in local optima and have slow convergence in high-dimensional spaces; Third, some existing optimization methods lack robustness to random environmental disturbances and cannot meet the real-time and reliability requirements of terminal access in power IoT.

[0003] In recent years, reinforcement learning, as a data-driven self-learning optimization method, has received widespread attention and has demonstrated advantages in fields such as network access control. Reinforcement learning allows agents to adjust their policies based on reward feedback through repeated interactions with the environment, enabling online learning and adaptation to environmental changes. For example, some studies have proposed using digital twin technology to assist reinforcement learning in optimizing IoT terminal access scheduling to improve terminal collaboration efficiency in smart park energy management. Furthermore, existing methods employ deep reinforcement learning (such as DQN and policy gradient) to optimize access selection and resource allocation in communication networks, achieving some success in improving system throughput and reducing latency. However, ordinary deep reinforcement learning still faces challenges when dealing with high-dimensional continuous parameter optimization: directly making decisions on a large-scale parameter space leads to excessively high dimensionality in the state and action spaces, multiplying the training difficulty and required sample size, making it difficult to converge to a satisfactory policy within a limited time. Simultaneously, single-layer reinforcement learning structures lack hierarchical decomposition capabilities, making it difficult for the policy learning process to efficiently capture the correlation between decisions at different levels (such as policy selection and specific parameter fine-tuning).

[0004] To address the above pain points and limitations of existing technologies, it is necessary to provide a new technical solution that can efficiently search high-dimensional parameter spaces and achieve adaptive optimization of terminal access parameters in order to meet the communication performance requirements of the power Internet of Things in a large-scale dynamic environment. Summary of the Invention

[0005] Purpose of the Invention: The main purpose of this invention is to overcome the shortcomings of existing power Internet of Things (IoT) terminal access parameter tuning methods and to provide an adaptive optimization method and system for terminal access parameters based on Monte Carlo sampling and hierarchical reinforcement learning. This method enables automatic optimization and adjustment of terminal access parameters (such as communication parameters and access control parameters) in the power IoT, improves the efficiency of high-dimensional parameter space search and strategy optimization effect, enhances the system's adaptability to dynamic environments, and supports online deployment and operation in actual networks, thereby significantly improving the overall performance and stability of terminal access.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] An adaptive optimization method for power Internet of Things (IoT) terminal access parameters based on hierarchical reinforcement learning includes the following steps:

[0008] The initial state of the current power Internet of Things environment is collected by a sampler. The initial state includes the operating parameters of each device and environmental variables. The collected initial state is input into the physical model state machine for state representation. The physical model state machine outputs hierarchical state information, where the high-level global state information includes network-level aggregated indicators and the low-level local state information includes device / link local observations and sub-target encoding.

[0009] Based on the current global state information, the high-level agent outputs high-level actions through the high-level policy network. These high-level actions limit the target range that the low-level agent needs to optimize, including the setting of sub-targets or the selection of parameter subspaces.

[0010] The lower-level intelligent agent receives the sub-goals set by the higher-level intelligent agent and the current lower-level local state information provided by the object model state machine. It outputs specific control action commands to the access devices through the lower-level policy network. The control action commands are applied to the power Internet of Things environment through the object model state machine, causing the system state to change.

[0011] The parameter judge monitors and evaluates the changed system state, while the benchmark tester acquires the performance data of the current stage and compares it with the preset benchmark indicators.

[0012] The reward model evaluates and scores the effectiveness of low-level actions based on the evaluation results of the parameter judge and the comparison results of the benchmark tester, and generates corresponding reward values ​​as feedback signals. These reward values ​​reflect the degree of contribution of the current action to the global goal. The low-level agent updates its short-term action policy based on the received immediate reward value. The high-level agent obtains a corresponding comprehensive reward feedback after a specified decision period based on the comprehensive effect of the low-level agent in completing the sub-goals within a certain number of time steps, which is used to update the high-level policy network.

[0013] The high-level intelligent agent updates the meta-policy using comprehensive reward feedback and current global state information, and resets the sub-objectives for the next stage according to the new policy. It then returns to the state sampling step to enter the next decision execution cycle. Through iterative training, the policies of the high-level and low-level intelligent agents tend to converge, thereby achieving adaptive optimization of the access parameters of the power Internet of Things terminal equipment.

[0014] Furthermore, the sampler employs Monte Carlo scenario sampling: at the beginning of each decision cycle, scenario parameters θ are extracted from the historical distribution and perturbation model, and multiple trajectories τ are generated by rolling within a window W to estimate variance and baseline returns. The scenario parameters θ include one or more of arrival intensity, interference spectrum, shadow fading, and fault injection.

[0015] Furthermore, the power Internet of Things environmental information collected by the sampler also includes network device computing power, channel conditions, and terminal service quality requirements. This information, together with the operating parameters of each device and environmental variables, constitutes the initial state and is input into the physical model state machine for state representation.

[0016] Furthermore, both the high-level and low-level intelligent agents adopt the Actor-Critic framework for their policy networks. The Actor-Critic framework includes a policy network and a value network. The policy network is responsible for outputting the corresponding action decisions, while the value network is used to approximate the value of a given state and provide a benchmark signal for the action decisions of the policy network.

[0017] Furthermore, the low-level agent adopts an adaptive time-domain normalized nt-step time-series differential update method to realize short-term action strategy updates. The value of nt is adaptively adjusted according to the fluctuations of the power Internet of Things environment. The environmental fluctuation is estimated by the sampler. The fluctuation specifically includes at least one of the following: channel carrier-to-noise ratio, queue length, and access collision rate. During the update process, the variance of the time-series differential error and the dominance function are normalized. At the same time, the low-level agent modulates the update step size with the service priority weight. The service priority weight is determined by the benchmark tester based on the throughput gain and latency cost.

[0018] Furthermore, when updating the high-level intelligent agent, a consistency regularization of sub-objective switching costs is introduced. This consistency regularization is jointly evaluated and determined by the sampler and the benchmark tester. Specifically, it is achieved by evaluating whether subspace switching leads to a decrease in revenue. At the same time, baseline comparison and steady-state penalty are introduced to favor high-throughput and stable strategies. By suppressing unnecessary sub-objective jitter, the engineering requirements of the power Internet of Things are met.

[0019] Furthermore, when generating reward values, the reward model comprehensively considers the degree of improvement in energy efficiency, the improvement in power supply quality, and the compliance with safety constraints, without the need for manually setting fixed weights. The reward value reflects the contribution of the current action to the overall goal.

[0020] Furthermore, the method also includes a Markov decision process modeling step, in which the power Internet of Things terminal access problem is modeled as a Markov decision process, defining a state space, an action space, and a reward function. The state space includes the terminal operating state, channel carrier-to-noise ratio, and edge computing load. The action space includes a set of adjustable access parameters, including transmit power, bandwidth, and backoff window. The reward function determines the immediate reward based on the terminal access success status, latency, and throughput.

[0021] Furthermore, in the Markov decision process modeling step, a scenario discount correction factor produced by the sampler and parameter judge and a comparison reward based on the benchmark are introduced, and a cumulative discount reward Gt is defined. The cumulative discount reward Gt is the sum of all discount rewards that the agent will obtain in the future from the current moment, which is used to measure the agent's long-term gains. The optimization objective of the agent's policy is to maximize the expected cumulative discount reward at the initial moment.

[0022] Furthermore, the method also includes the execution of a Q-Learning update mechanism, which iteratively estimates the optimal action value based on the Bellman optimal equation. Specifically, it updates the action value function of the state-action pair according to a preset learning rate, wherein the update of the action value function refers to the maximum expected value that can be obtained in the next state. This Q-Learning update mechanism is used to pre-train the policy in a simulation environment or to provide reference value for the decision-making of high-level agents, thereby improving the stability and convergence speed of the algorithm training.

[0023] An adaptive optimization system for terminal access parameters in the power Internet of Things (IoT) based on hierarchical reinforcement learning includes a sampler, an object model state machine, a high-level agent, a low-level agent, a parameter judge, a benchmark tester, and a reward model.

[0024] The output of the sampler is connected to the input of the physical model state machine to collect state information of the power Internet of Things environment and transmit it to the physical model state machine. This state information includes equipment operating parameters, environmental variables, network equipment computing power, channel conditions and terminal service quality requirements.

[0025] The output of the object model state machine is connected to the input of the high-level agent and the input of the low-level agent, respectively. It is used to represent the state information collected by the sampler and provide the represented state information to the high-level agent and the low-level agent, respectively. The object model state machine outputs hierarchical state information, of which the high-level global state information includes network-level aggregation indicators and the low-level local state information includes device / link local observations and sub-target encoding.

[0026] The output of the high-level intelligent agent is connected to the input of the low-level intelligent agent. It is used to formulate sub-goals based on the global state information provided by the object model state machine and transmit them to the low-level intelligent agent. At the same time, the high-level policy is updated based on the comprehensive reward feedback from the reward model.

[0027] The output of the low-level agent is connected to the input of the object model state machine. It is used to output control action commands based on the sub-goals formulated by the high-level agent and the current local state information provided by the object model state machine and transmit them to the object model state machine. At the same time, the low-level policy is updated based on the real-time rewards fed back by the reward model.

[0028] The input of the parameter judge is connected to the output of the physical model state machine. It is used to receive the changed system state output by the physical model state machine and to monitor and evaluate the changes in system state, the satisfaction of constraints, and the completion of sub-goals. The output of the parameter judge is connected to the input of the reward model and is used to transmit the monitoring and evaluation results to the reward model.

[0029] The output of the benchmark is connected to the input of the reward model to obtain the performance data of the current stage and compare it with the preset benchmark index, and transmit the comparison result to the reward model.

[0030] The output of the reward model is connected to the inputs of both the high-level agent and the low-level agent. It generates reward values ​​based on the monitoring and evaluation results of the parameter judge and the comparison results of the benchmark tester, and feeds them back to the high-level agent and the low-level agent, respectively. The reward value received by the low-level agent is the immediate reward value output by the reward model, which is used to update the low-level policy network. The high-level agent obtains a corresponding comprehensive reward feedback after a specified decision period based on the comprehensive effect of the low-level agent in completing the sub-objectives within a certain number of time steps, which is used to update the high-level policy network.

[0031] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the adaptive optimization method for power Internet of Things terminal access parameters based on hierarchical reinforcement learning as described above.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the adaptive optimization method for power Internet of Things terminal access parameters based on hierarchical reinforcement learning as described above.

[0033] Beneficial effects:

[0034] (1) To address the problem that existing fixed or single-layer optimization strategies struggle to achieve global optimum in dynamic high-dimensional scenarios, this invention proposes an adaptive method. Through a high-low layered structure, the high-level agent is responsible for guiding long-term goals and global strategy planning, ensuring the optimization process avoids short-sightedness; the low-level agent focuses on the short-term effects of specific actions, achieving efficient execution of sub-tasks. This combination balances long-term strategy planning with short-term parameter fine-tuning, and allows for adaptive adjustments to different device behavior patterns and environmental changes, improving robustness against dynamic load changes and fault disturbances.

[0035] (2) This invention abandons the fixed weighted index summation mode and instead uses an adaptive strategy optimization based on reinforcement learning. The agent learns autonomously how to balance multiple optimization objectives (such as the balance between energy saving and stability) through interaction with the environment, without the need for manually setting fine weight parameters. It is more flexible in the complex and ever-changing scenarios of the power Internet of Things and can dynamically adjust the strategy according to environmental feedback to meet different objective requirements.

[0036] (3) Faced with the high-dimensional control parameter space in the power Internet of Things (e.g., simultaneously adjusting the operating parameters of numerous devices), traditional single-layer algorithms often struggle to efficiently search for the optimal solution. This invention divides high-dimensional problems into more manageable low-dimensional subproblems through subspace selection and task decomposition. After the high-level agent narrows the problem scope, the low-level agent focuses on in-depth optimization within the subspace, reducing learning difficulty and computational complexity. Simultaneously, the high-level strategy can reuse skills or experiences learned by the low-level agent; when similar subtasks recur, there is no need to learn from scratch, improving sample utilization efficiency.

[0037] (4) Because the high- and low-level policies are co-optimized at different time scales, the method of this invention is more likely to converge to the global optimum than the policy based on a single-step reward. The guidance provided by the high-level policy reduces ineffective exploration and avoids the low-level policy from getting stuck in local optima; while the rapid feedback from the low-level policy promotes timely correction of the direction by the high-level policy. The two complement each other, making the entire reinforcement learning process more stable and efficient, and ultimately achieving better global performance than traditional methods. Attached Figure Description

[0038] Figure 1 is a schematic diagram of the deployment method of the present invention in an Internet of Things environment;

[0039] Figure 2 is a flowchart of the method of the present invention;

[0040] Figure 3 is a framework diagram of the specific implementation process of Actor-Critic. Detailed Implementation

[0041] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0042] Figure 1 shows a schematic diagram of the deployment of the present invention in the field of Power Internet of Things (PIoT): The sampler collects operating parameters and environmental variables from the terminal, gateway, edge node and scheduling system, and after the state machine of the physical model is used for state representation, it is supplied to the high-level / low-level intelligent agents respectively; the parameter judge and the benchmark tester monitor and evaluate the system performance after the action is executed; the reward model feeds back reward signals to the high-level / low-level.

[0043] Figure 2 shows the flowchart of the method of the present invention: it includes S1 initialization and state sampling, S2 high-level sub-goal formulation, S3 low-level strategy execution, S4 parameter judgment and comparison evaluation, S5 reward calculation and feedback, and S6 high-level strategy update and iteration; wherein the benchmark tester is an independent module, and its output and the result of the parameter judge are input into the reward model together.

[0044] Referring to Figures 1 and 2, this invention provides an adaptive optimization method for terminal access parameters in the power Internet of Things (IoT). By constructing a hierarchical reinforcement learning model and a Monte Carlo sampling mechanism, it achieves automatic optimization and adjustment of terminal access parameters (such as communication parameters and access control parameters) in the power IoT. This method is referred to in this paper as the HRL-MC (Hierarchical RL+Monte-Carlo) adaptive method. The method includes the following steps:

[0045] S1. Initialization and State Sampling: The sampler collects the initial state of the current power IoT environment, including the operating parameters of each device and environmental variables, and inputs it into the physical model state machine for state representation. The higher-level intelligent agent obtains global state information as the basis for decision-making. The sampler uses Monte Carlo scenario sampling: at the beginning of each decision cycle, scenario parameters θ (such as arrival intensity, interference spectrum, shadowing fading, and fault injection) are extracted from historical distributions and disturbance models, and multiple trajectories τ are generated within a window W to estimate variance and baseline return. The physical model state machine outputs hierarchical states: the higher-level global state. = [Network load, access congestion, spectrum utilization, edge computing power utilization, QoS gap vector, critical service queue summary...]; Low-level local state = [device] Link quality (C / N, RSRP / RSRQ), queue length, recent collision count, power margin, timing advance, etc.; and embed high-level sub-targets. encoding .

[0046] S2. High-level decision-making to set sub-goals: Based on the current global state, the high-level agent outputs a high-level action through its policy network (such as the Actor part of an Actor-Critic). This high-level action typically manifests as setting sub-goals or selecting parameter subspaces. For example, the high-level agent might decide to prioritize optimizing the power output of a certain type of device, or focus on the voltage stability of a specific region. High-level decision-making is equivalent to decomposing a complex global task into sub-tasks for the current stage and defining the scope of objectives that the lower-level agent should optimize next.

[0047] S3. Low-level policy execution: The low-level agent receives sub-goals set by the higher-level agent and the current state provided by the physical model state machine (which may include local state details affected by the actions of the higher-level agent). Through its policy network, the low-level agent outputs specific control action commands based on the current state and sub-goals (e.g., adjusting the operating parameters of a device, switching the state of a circuit breaker, etc.). This action is applied by the physical model state machine to the simulation or real environment, causing a corresponding change in the state of the power Internet of Things system.

[0048] S4. Parameter Judgment and Environmental Feedback: After the lower-level agent performs an action, the parameter judge monitors and evaluates the new system state. It judges changes in key indicators, whether constraints are met, and the degree of completion of sub-objectives. For example, the parameter judge can check whether the voltage is maintained within a safe range and whether the load balance has improved. At the same time, the benchmark acquires the performance data of the current stage and compares it with preset benchmark indicators or traditional strategy results to evaluate the merits of the current strategy.

[0049] S5. Reward Evaluation and Feedback: The reward model evaluates and scores the effectiveness of low-level actions based on data provided by the parameter judge and benchmark, generating corresponding reward values ​​as feedback signals. These reward values ​​reflect the contribution of the current action to the global objective, such as the degree of improvement in energy efficiency, power quality, and compliance with safety constraints. However, these evaluation processes are implicit to the agent and do not require manually setting fixed weights. The feedback output by the reward model is used to train both high-level and low-level agents: low-level agents receive immediate rewards to update their short-term action policies (e.g., calculating advantage estimates and updating the Actor policy through a Critic network); high-level agents receive a summary reward feedback after a longer decision period, based on the overall effect of the low-level agents completing sub-objectives over several time steps (which can be the sum of discounted rewards accumulated over these steps or sub-task completion metrics), to update the high-level policy network. This design ensures that high-level optimization focuses on long-term results, while low-level optimization emphasizes immediate effects, achieving hierarchical decoupling on a time scale.

[0050] S6. High-Level Policy Update and Cycle: Utilizing the summarized feedback and considering the current global state evolution, the high-level agent updates its meta-policy (e.g., updating the high-level actor's policy parameters through high-level value assessment within the Actor-Critic framework), thereby adjusting towards a better long-term planning direction in the policy space. Subsequently, the high-level agent can reset the sub-objectives for the next stage based on the new policy. The system continues to cycle through steps S1-S5, entering the next decision execution cycle. Through continuous iterative training, the policies of the high-level and low-level agents will converge, collaboratively achieving overall optimized control of the power Internet of Things system.

[0051] According to an embodiment of the present invention, the power Internet of Things (IoT) terminal access problem is modeled as a Markov decision process (MDP). Let the state space be denoted as... This includes environmental information such as network device computing power, channel conditions, and terminal QoS requirements; the action space is... This represents a set of adjustable access parameters (such as bandwidth allocation, access timing, or channel selection). At time... Intelligent agents observe environmental conditions Take action (For example, selecting a specific access configuration for a terminal). After the action is executed, the environment provides an immediate reward. For example, numerical rewards can be given based on metrics such as successful terminal access, latency, and throughput. Then the environment transitions to the next state. The above process satisfies the Markov property, which states that the next state and reward depend only on the current state and action, and are independent of earlier history.

[0052] Specifically, this invention constructs a two-layer MDP, as described below:

[0053] state space High-level status Network-level aggregated metrics (access congestion, spectrum utilization, edge computing power, critical service QoS gap vector, regional load heatmap, etc.); lower-layer status Device / link local observations (C / N, queues, collision rate, power margin, RSSI / RSRQ, timing, etc.) and subtarget coding;

[0054] Action space High-level actions Sub-target / subspace selection; low-level actions Fine-grained access parameters for a single device or a cluster of devices;

[0055] award :

[0056] In the reinforcement learning framework, the goal of an agent is to learn an optimal policy. This maximizes the expected reward accumulated during interaction with the environment. To quantify long-term returns, a cumulative discounted return is introduced. Defined as an intelligent agent from time 1000 to 11000. This is the sum of all future discount rewards. Furthermore, a "scene discount correction factor" is introduced, generated by the sampler and parameter judge. And benchmark-based feedback (Instant slice of MC Monte Carlo sampling benchmark / default strategy rolling return under the same initial state), defined as: ,in is the discount factor used to balance the relative importance of immediate and long-term rewards. k is the time step count index, and is the summation coefficient. The optimization objective of the agent's policy is to maximize the expected cumulative discounted reward at the initial time step, i.e. This achieves the optimal trade-off for long-term performance.

[0057] For evaluation strategy To characterize the advantages and disadvantages of a state, this invention uses a value function. State value function Indicates the agent's state According to the strategy Expected cumulative returns: ,in Indicating in strategy The expected value of randomness. Similarly, the action-value function. Defined as an agent in a state Take action Then, follow the strategy The expected cumulative return from continued action: Based on the above value function, the dominance function is further defined. It is used to measure the quality of a particular action relative to the average level of a state: This means that the reward from an action is relative to the state. How much higher (or lower) is the general level? The introduction of the advantage function can reduce high variance in policy optimization, helping the agent to more stably evaluate the relative merits of each action.

[0058] For parameterized randomization strategies (from parameter vector) (Control), the policy gradient theorem gives the optimization objective. The gradient form is used to guide the direction of policy parameter updates. Basic policy gradient algorithms (such as REINFORCE) utilize Monte Carlo sampling to estimate the policy gradient. (depending on Monte Carlo sampling), the strategy is improved according to the following formula:

[0059]

[0060] in Indicates from state The cumulative reward obtained from the actual sampling, derived from the summation of samples from the MC Monte Carlo replay trajectory, is a direct manifestation of "estimating the policy gradient using Monte Carlo sampling." Because it directly uses... Using it as an evaluation metric can lead to large variance, so a baseline function is usually introduced. To reduce variance, the state-value function is chosen as the baseline. Then, the approximate form of the above gradient can be expressed as:

[0061]

[0062] In practical implementation, this invention employs the Actor-Critic framework to efficiently approximate the aforementioned policy gradient. The Actor-Critic algorithm comprises two main modules: a policy network (actor) and a value network (critic). The policy network is responsible for outputting action decisions, and its parameters are updated based on the policy gradient; the value network approximates the value of a given state, providing a benchmark signal for policy updates. Specifically, the critic can utilize time difference (TD) error. To update the value estimate, thereby approximating the true value. The actor then uses the advantage estimate provided by the critic to adjust the policy parameters, and its update rule can be expressed as:

[0063] in is the learning rate of the policy network. In the hierarchical reinforcement learning architecture, high-level and low-level policies are each equipped with corresponding Actor-Critic modules, which are responsible for optimizing sub-objective policies and specific action policies, respectively. The specific implementation framework of Actor-Critic is shown in Figure 3.

[0064] The hierarchical Actor-Critic update mechanism of this invention is as follows:

[0065] 1) Normalization in the low-level adaptive time domain -step TD”

[0066] Increase the step size of lower-level returns Adaptive to scene fluctuations ( (As a hyperparameter), it adapts to environmental fluctuations, and extends the reward horizon when congestion is high / disturbance is strong:

[0067]

[0068] in The fluctuation within the window estimated by the sampler (channel C / N, i.e., carrier-to-noise ratio, queue length, access collision rate, etc.). Given the maximum step size, a dynamic discount chain is defined accordingly. -step TD error ( (for lower-level TD errors)

[0069]

[0070] in For short-term returns, Estimated for Bootstrap. For low-level action value estimation (Critic), the parameters are: . The meaning is a discount correction related to the state / sub-goal.

[0071] Then perform uncertainty normalization (from the Monte Carlo rolling standard deviation) ):

[0072]

[0073] Low-level update (weight) (The benchmark provides the priority of "throughput gain - latency cost")

[0074]

[0075]

[0076]

[0077] Both TD and Advantage perform variance normalization; lower-level Actors are weighted according to business priority. Modulation update step size (amplified when throughput increases and latency decreases). For low-level value network parameters, These are the parameters for the low-level policy network.

[0078] 2) Consistency regularization between high-level "comparison-steady-state-safety" aggregated returns and objectives

[0079] High-level single-cycle aggregated returns:

[0080]

[0081] in As a baseline return, the return is estimated in the sampler's MC replay by "a return slice obtained by rolling according to the baseline strategy under the same initial state". Additional rewards (or penalties) are used to encourage compliance with specific constraints / goals (such as exceeding energy-saving thresholds, meeting safety constraints, etc.). Deviation is a steady-state deviation penalty term. It can be defined as the L2 norm of key indicators (such as voltage deviation from the target, load imbalance, etc.), with λ≥0 being the penalty coefficient, used for strategies that prefer to tend to the steady state. Penalty for variance in returns, The strategy for suppressing high variance is calculated on the MC Monte Carlo sample, where ω is the weight.

[0082] And use the same dynamic discount chain as the lower layer for T-step correction:

[0083]

[0084] During high-level updates, a consistency regularization rule for "sub-target switching cost" is added (the sampler and benchmarker evaluate whether subspace switching leads to a decrease in benefits):

[0085]

[0086]

[0087] in, . For high-level value network parameters, These are the network parameters for high-level strategies. The consistency regularization strength coefficient (≥0) is used to penalize frequent switching that leads to decreased returns.

[0088] Introducing baseline control With steady-state penalty They prefer "high throughput and stability"; through Suppressing unnecessary sub-target jitter meets the engineering requirements of "fewer switching and stable operation" for the power Internet of Things.

[0089] To complement value function learning, this invention also introduces the classic Q-Learning update mechanism to iteratively estimate the optimal action value. For state-action pairs... Q-Learning updates based on the Bellman optimal equation: in For learning rate, Indicates the next state The maximum expected value that can be obtained. Through the above iterative updates, The estimated value will gradually approach the true optimal action value function. In the method of this invention, the Q-Learning mechanism can be used to pre-train the policy in a simulation environment or to provide reference value for high-level decision-making, thereby improving the stability and convergence speed of algorithm training.

[0090] This invention also provides an adaptive optimization system for power Internet of Things (IoT) terminal access parameters based on hierarchical reinforcement learning, comprising a sampler, an object model state machine, a high-level agent, a low-level agent, a parameter judge, a benchmark tester, and a reward model.

[0091] The output of the sampler is connected to the input of the physical model state machine to collect state information of the power Internet of Things environment and transmit it to the physical model state machine. This state information includes equipment operating parameters, environmental variables, network equipment computing power, channel conditions and terminal service quality requirements.

[0092] The output of the object model state machine is connected to the input of the high-level agent and the input of the low-level agent, respectively. It is used to represent the state information collected by the sampler and provide the represented state information to the high-level agent and the low-level agent, respectively. The object model state machine outputs hierarchical state information, of which the high-level global state information includes network-level aggregation indicators and the low-level local state information includes device / link local observations and sub-target encoding.

[0093] The output of the high-level intelligent agent is connected to the input of the low-level intelligent agent. It is used to formulate sub-goals based on the global state information provided by the object model state machine and transmit them to the low-level intelligent agent. At the same time, the high-level policy is updated based on the comprehensive reward feedback from the reward model.

[0094] The output of the low-level agent is connected to the input of the object model state machine. It is used to output control action commands based on the sub-goals formulated by the high-level agent and the current local state information provided by the object model state machine and transmit them to the object model state machine. At the same time, the low-level policy is updated based on the real-time rewards fed back by the reward model.

[0095] The input of the parameter judge is connected to the output of the physical model state machine. It is used to receive the changed system state output by the physical model state machine and to monitor and evaluate the changes in system state, the satisfaction of constraints, and the completion of sub-goals. The output of the parameter judge is connected to the input of the reward model and is used to transmit the monitoring and evaluation results to the reward model.

[0096] The output of the benchmark is connected to the input of the reward model to obtain the performance data of the current stage and compare it with the preset benchmark index, and transmit the comparison result to the reward model.

[0097] The output of the reward model is connected to the inputs of both the high-level agent and the low-level agent. It generates reward values ​​based on the monitoring and evaluation results of the parameter judge and the comparison results of the benchmark tester, and feeds them back to the high-level agent and the low-level agent, respectively. The reward value received by the low-level agent is the immediate reward value output by the reward model, which is used to update the low-level policy network. The high-level agent obtains a corresponding comprehensive reward feedback after a specified decision period based on the comprehensive effect of the low-level agent in completing the sub-objectives within a certain number of time steps, which is used to update the high-level policy network.

[0098] The present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the adaptive optimization method for power Internet of Things terminal access parameters based on hierarchical reinforcement learning as described above.

[0099] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the adaptive optimization method for power Internet of Things terminal access parameters based on hierarchical reinforcement learning as described above.

[0100] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus (systems), computer devices, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This invention is described with reference to a flowchart of a method according to embodiments of the invention. It should be understood that each step in the flowchart and combinations of steps in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more steps of the flowchart.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more processes of a flowchart.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart.

Claims

1. An adaptive optimization method for power Internet of Things (IoT) terminal access parameters based on hierarchical reinforcement learning, characterized in that, Includes the following steps: The initial state of the current power Internet of Things environment is collected by a sampler. The initial state includes the operating parameters of each device and environmental variables. The collected initial state is input into the physical model state machine for state representation. The physical model state machine outputs hierarchical state information, where the high-level global state information includes network-level aggregated indicators and the low-level local state information includes device / link local observations and sub-target encoding. The high-level agent outputs high-level actions through the high-level policy network based on the current global state information. These high-level actions limit the target range to be optimized by the low-level agent, including the setting of sub-objectives or the selection of parameter subspaces. The low-level agent receives the sub-objectives set by the high-level agent and the current low-level local state information provided by the object model state machine. It then outputs specific control action commands to the access devices through the low-level policy network. These control action commands are applied to the power Internet of Things environment via the object model state machine, causing a change in the system state. The parameter judge monitors and evaluates the changed system state, while the benchmark tester acquires the performance data of the current stage and compares it with preset benchmark indicators. The reward model evaluates and scores the effect of the low-level actions based on the evaluation results of the parameter judge and the comparison results of the benchmark tester, generating a corresponding reward value as a feedback signal. This reward value reflects the contribution of the current action to the global objective. The low-level agent updates its short-term action strategy based on the received immediate reward value. The high-level agent obtains a corresponding comprehensive reward feedback after a specified decision period based on the comprehensive effect of the low-level agent completing the sub-objectives within a certain number of time steps, which is used to update the high-level policy network. The high-level intelligent agent updates the meta-policy using comprehensive reward feedback and current global state information, and resets the sub-objectives for the next stage according to the new policy. It then returns to the state sampling step to enter the next decision execution cycle. Through iterative training, the policies of the high-level and low-level intelligent agents tend to converge, thereby achieving adaptive optimization of the access parameters of the power Internet of Things terminal equipment.

2. The method according to claim 1, characterized in that, The sampler employs Monte Carlo scenario sampling: at the beginning of each decision cycle, scenario parameters θ are extracted from the historical distribution and disturbance model, and multiple trajectories τ are generated within a window W to estimate variance and baseline return. The scenario parameters θ include one or more of arrival strength, interference spectrum, shadowing fading, and fault injection. The power IoT environmental information collected by the sampler also includes network device computing power, channel conditions, and terminal service quality requirements. This information, together with the operating parameters of each device and environmental variables, constitutes the initial state and is input into the physical model state machine for state representation.

3. The method according to claim 1, characterized in that, Both the high-level and low-level intelligent agents adopt the Actor-Critic framework for their policy networks. The Actor-Critic framework includes a policy network and a value network. The policy network is responsible for outputting the corresponding action decisions, while the value network is used to approximate the value of a given state and provide a benchmark signal for the action decisions of the policy network.

4. The method according to claim 3, characterized in that, The low-level agent adopts an adaptive time-domain normalized nt-step time-series differential update method to update short-term action strategies. The value of nt is adaptively adjusted according to the fluctuations of the power Internet of Things environment. The environmental fluctuation is estimated by the sampler. The fluctuation specifically includes at least one of the following: channel carrier-to-noise ratio, queue length, and access collision rate. During the update process, the variance of the time-series differential error and the dominance function are normalized. At the same time, the low-level agent modulates the update step size with the service priority weight. The service priority weight is determined by the benchmark tester based on the throughput gain and latency cost.

5. The method according to claim 3, characterized in that, When updating high-level intelligent agents, a consistency regularization of sub-objective switching costs is introduced. This consistency regularization is jointly evaluated and determined by the sampler and the benchmark tester. Specifically, it is achieved by evaluating whether subspace switching leads to a decrease in revenue. At the same time, baseline comparison and steady-state penalty are introduced to favor high-throughput and stable strategies. By suppressing unnecessary sub-objective jitter, the engineering requirements of the power Internet of Things are met.

6. The method according to claim 1, characterized in that, When generating reward values, the reward model comprehensively considers the degree of improvement in energy efficiency, the improvement in power supply quality, and the compliance with safety constraints. It does not require manual setting of fixed weights, and the reward value reflects the contribution of the current action to the overall goal.

7. The method according to claim 1, characterized in that, It also includes a Markov decision process modeling step, in which the power Internet of Things terminal access problem is modeled as a Markov decision process, defining a state space, an action space and a reward function. The state space includes the terminal operating state, channel carrier-to-noise ratio and edge computing load, the action space includes a set of adjustable access parameters including transmit power, bandwidth and backoff window, and the reward function determines the instantaneous reward based on the terminal access success status, latency and throughput.

8. The method according to claim 7, characterized in that, In the Markov decision process modeling step, a scenario discount correction factor produced by the sampler and parameter judge and a comparison reward based on the benchmark are introduced. The cumulative discount reward Gt is defined as the sum of all discount rewards that the agent will receive in the future from the current moment, which is used to measure the agent's long-term gains. The optimization objective of the agent's policy is to maximize the expected cumulative discount reward at the initial moment.

9. The method according to claim 1, characterized in that, It also includes the execution of a Q-Learning update mechanism, which iteratively estimates the optimal action value based on the Bellman optimal equation. Specifically, it updates the action value function of the state-action pair according to a preset learning rate. The update of the action value function refers to the maximum expected value that can be obtained in the next state. This Q-Learning update mechanism is used to pre-train the policy in a simulation environment or to provide reference value for the decision-making of high-level agents, thereby improving the stability and convergence speed of the algorithm training.

10. An adaptive optimization system for power Internet of Things (IoT) terminal access parameters based on hierarchical reinforcement learning, characterized in that, The system includes a sampler, a physical model state machine, a high-level agent, a low-level agent, a parameter judge, a benchmark tester, and a reward model. The output of the sampler is connected to the input of the physical model state machine to collect state information of the power Internet of Things (IoT) environment and transmit it to the physical model state machine. This state information includes equipment operating parameters, environmental variables, network equipment computing power, channel conditions, and terminal service quality requirements. The output of the physical model state machine is connected to the inputs of both the high-level and low-level agents to represent the state information collected by the sampler and provide the represented state information to the high-level and low-level agents respectively. The physical model state machine outputs hierarchical state information, where the high-level global state information includes network-level aggregated indicators, and the low-level local state information includes device / link local observations and sub-target encoding. The output of the high-level agent is connected to the input of the low-level agent to formulate sub-targets based on the global state information provided by the physical model state machine and transmit them to the low-level agent. At the same time, it updates the high-level strategy based on the comprehensive reward feedback from the reward model. The output of the low-level agent is connected to the input of the object model state machine. It outputs control action commands based on the sub-goals set by the high-level agent and the current local state information provided by the object model state machine, and transmits these commands to the object model state machine. Simultaneously, it updates the low-level strategy based on the immediate rewards fed back from the reward model. The input of the parameter judge is connected to the output of the object model state machine. It receives the changed system state output from the object model state machine and monitors and evaluates system state changes, constraint satisfaction, and sub-goal completion. The output of the parameter judge is connected to the input of the reward model, transmitting the monitoring and evaluation results to the reward model. The output of the benchmark is connected to the input of the reward model. The terminal connection is used to acquire performance data of the current stage and compare it with preset benchmark indicators, and transmit the comparison results to the reward model. The output terminal of the reward model is connected to the input terminals of the high-level agent and the low-level agent respectively, and is used to generate reward values ​​based on the monitoring and evaluation results of the parameter judge and the comparison results of the benchmark tester, and feed them back to the high-level agent and the low-level agent respectively. The reward value received by the low-level agent is the instant reward value output by the reward model, which is used to update the low-level policy network. The high-level agent obtains a corresponding comprehensive reward feedback after a specified decision period based on the comprehensive effect of the low-level agent in completing the sub-objective within a certain number of time steps, which is used to update the high-level policy network.

11. An electronic device, comprising: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the adaptive optimization method for power Internet of Things terminal access parameters based on hierarchical reinforcement learning as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the adaptive optimization method for power Internet of Things terminal access parameters based on hierarchical reinforcement learning as described in any one of claims 1-9.

Images