Object model access parameter dynamic optimization method and system based on reinforcement learning

Abstract

The invention discloses an object model access parameter dynamic optimization method and system based on reinforcement learning. According to the method, a multi-agent reinforcement learning model is constructed, a device-level state and a network-level state of the Internet of Things terminal device are used as environment state input, access parameters of the terminal device are modeled as an action decision of a reinforcement learning decision model, and the reinforcement learning decision model outputs optimized access parameters according to the current environment state. And continuously updating the strategy network through a reward function. And finally, the control platform deploys the trained strategy model, automatically selects the optimal access parameter configuration when the terminal equipment accesses, and adjusts the access strategy in real time according to the state change of the equipment and the network. Compared with an existing static strategy, the method has the advantages that efficient, low-time-delay and low-power-consumption access of the power internet-of-things terminal equipment can be achieved, and the system resource utilization rate and the communication reliability are remarkably improved.

Description

Technical Field

[0001] This invention relates to a reinforcement learning method for optimizing the parameters of object models in multi-agent systems, particularly in the fields of artificial intelligence, especially in the Internet of Things and intelligent control systems. Background Technology

[0002] With the rapid development of new power systems and Power IoT technologies, an increasing number of terminal devices (such as smart meters, substation sensors, and distribution automation terminals) are being deployed in various power scenarios. While these terminals perform metering, monitoring, and control tasks, they also place higher demands on the access communication system, especially in terms of high reliability, low latency, low power consumption, and high resource utilization. Traditional static access mechanisms often cannot adapt to the dynamically changing environmental conditions and heterogeneous characteristics of terminals in the power grid, easily leading to resource waste, communication congestion, or access failures, thereby affecting the overall dispatch efficiency and safe operation of the power grid.

[0003] In power Internet of Things (IoT) applications, different types of terminal devices have varying requirements for computing resources, power consumption limits, communication performance, and service priorities. For example, latency-sensitive terminals (such as protective sensors) need to complete data reporting within a very short time; while energy-sensitive devices (such as distribution controllers) require minimizing power consumption to extend their operational lifespan. Furthermore, in actual deployment, these devices often face issues such as channel conflicts, bandwidth contention, and uneven allocation of scheduling resources, further exacerbating the system's access pressure.

[0004] Most existing access optimization methods are based on heuristic policies or rule settings, making it difficult to achieve general and efficient resource coordination in complex, variable, and large-scale access scenarios. Some studies have attempted to introduce reinforcement learning mechanisms for power dispatching and resource optimization, but they struggle to fully perceive network conditions and make collaborative decisions under single-agent conditions. Furthermore, some current reinforcement learning methods rely on centralized modeling and control, failing to fully consider the resource constraints and distributed autonomy requirements of terminal devices in actual operation.

[0005] Therefore, in order to address the problems of unreasonable access parameter selection, low resource scheduling efficiency, and insufficient policy generalization ability in the current power Internet of Things terminal access, it is urgent to design a dynamic access optimization method and system with adaptive and autonomous collaborative capabilities to achieve efficient, reliable, and low-power terminal access strategies. Summary of the Invention

[0006] Objective: To address the problems mentioned above in the background technology, this invention proposes a method and system for dynamic optimization of object model access parameters based on reinforcement learning, solving the problem that static access strategies in the prior art cannot meet the dynamic access environment requirements of power Internet of Things (IoT) terminal devices. Through the optimization method provided by this invention, low-power, low-latency dynamic access of terminal devices can be efficiently achieved, thereby improving the resource utilization efficiency, device access stability, and overall performance of the power IoT.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] A method for dynamically optimizing access parameters of an object model based on reinforcement learning includes the following steps:

[0009] Based on the business needs, priorities, and resource needs of terminal devices in the power Internet of Things, construct object model intelligent agents, with each object model intelligent agent corresponding to a type of access terminal device.

[0010] Collect device-level and network-level state information from each terminal device, and model this state information as the current environment state S for reinforcement learning. t ; Set the current environmental state S t Input a pre-built reinforcement learning decision model, which outputs access parameter optimization decisions for each terminal device based on a multi-agent reinforcement learning algorithm. Adjust the access parameters of each terminal device according to the access parameter optimization decisions and execute device access operations.

[0011] Calculate the reward value R based on the access execution results of each terminal device. t According to the current environmental state S t Action decision and the reward value R obtained t Update the policy parameters of the reinforcement learning decision model and optimize the access parameter strategy of each terminal device under the principle of maximizing cumulative reward;

[0012] The trained reinforcement learning decision model is integrated into the IoT control platform. When a new terminal device is connected, a corresponding agent instance of the object model is created, parameters are configured in real time, and the access strategy is adjusted in real time according to changes in the terminal device's own state and network state.

[0013] Preferably, the device-level status information of the terminal device includes the status information of the terminal device itself, as well as the communication process status information of the terminal device;

[0014] Network-level status information includes global status information provided by the IoT control platform.

[0015] Preferably, the action space of the reinforcement learning decision model includes the following access parameter adjustment actions:

[0016] Channel selection: Selecting the communication channel for terminal devices to access in order to optimize their data transmission latency;

[0017] Power control: Adjusting the operating power or transmit power level of the terminal equipment to reduce energy consumption and processing latency while ensuring performance;

[0018] Bandwidth allocation: Allocating bandwidth reasonably to terminal devices under limited bandwidth resources in order to optimize their transmission latency;

[0019] Data compression rate adjustment: Adjustments are made to the data reporting rate of terminal devices to achieve a balance between increasing compression to reduce transmission volume and reducing compression to reduce latency, thereby optimizing energy consumption and transmission latency;

[0020] Computing resource allocation: Edge / cloud computing resource quotas allocated by the IoT control platform to terminal devices to meet their data processing needs and optimize end-to-end latency.

[0021] Preferably, the calculation of the reward value considers both global and local rewards. The global reward aims to improve overall system performance, including maximizing the proportion of successfully connected terminal devices, minimizing the average communication latency of all terminal devices, minimizing total system power consumption, and improving bandwidth and computing resource utilization. The local reward is performance feedback for individual terminal devices, including giving a positive reward when a terminal device successfully connects, and giving a penalty or additional reward based on the actual communication latency and energy consumption of the terminal device. The total reward R for each agent... t It is obtained by weighted combination of global and local rewards and is used to guide each agent to optimize its access parameter decisions.

[0022] Preferably, the reinforcement learning decision model is trained using a multi-agent near-end policy optimization algorithm and adopts a centralized training and distributed execution architecture, including: acquiring network-level state information during the training phase and using a centralized evaluator to evaluate the joint policy of the multi-agents and update the shared value function parameters; during the execution phase, each terminal device agent independently selects actions based on its own observed device-level state information using the trained policy, thereby enabling the acquisition of near-globally optimal access parameter configuration policies in large-scale terminal device scenarios and reducing the computational and communication overhead of real-time decision-making.

[0023] Preferably, during the training phase, the policy update of the reinforcement learning decision model employs a policy gradient truncation and target network soft update mechanism, specifically including:

[0024] Using the advantage function A t The merits of actions under the current policy are evaluated, and the advantage function is defined as the difference between the actual reward obtained by the current action and the benchmark value. Based on the advantage function, a shearing strategy is adopted to optimize the objective, and the policy probability ratio is restricted during policy updates. The variation range is within the preset range [1-∈, 1+∈], where π new (a t |s t ) indicates that in state s according to the current policy parameter θ t Select action a t The probability or probability density of . This represents the corresponding probability or probability density of the old policy before the update, where ∈ is the policy truncation threshold;

[0025] The target network parameters are updated using a soft update method, meaning that in each training iteration, the new parameters of the policy network are incorporated into the target network parameters proportionally τ: θ target ←τ,θ new +(1-τ),θ target , where θ target Let θ be the target network parameter. new Let τ be a new parameter for the policy network, where τ is a soft update coefficient less than 1.

[0026] Preferably, when a new terminal device is connected to the IoT control platform, the following steps are performed: identify the object model type to which the new terminal device belongs, initialize the corresponding intelligent agent according to the predefined parameters of the object model; incorporate the initial state information of the new terminal device into the environmental state of the reinforcement learning decision model, and generate the access parameter configuration scheme of the terminal device based on the optimal policy obtained from the current training, so that the new terminal device can access the network with the optimal parameters in an instant.

[0027] Secondly, a dynamic optimization system for access parameters of an object model based on reinforcement learning includes:

[0028] The object model intelligent agent construction module is used to construct object model intelligent agents based on the business needs, priorities and resource needs of terminal devices in the power Internet of Things. Each object model intelligent agent corresponds to a type of access terminal device.

[0029] The state monitoring and acquisition module is used to collect device-level and network-level state information from each terminal device, and model this state information as the current environment state S for reinforcement learning. t ;

[0030] The decision execution module is used to carry the reinforcement learning decision model. The reinforcement learning decision model makes decisions based on the current environment state S input. t Based on the multi-agent reinforcement learning algorithm, the system outputs the access parameter optimization decision for each terminal device and sends the decision to the corresponding terminal device to perform the access operation.

[0031] The strategy training module is used to calculate the reward value R based on the execution results of each terminal device. t According to the current environmental state S tAction decision and the reward value R obtained t Update the policy parameters of the reinforcement learning decision model and optimize the access parameter strategy of each terminal device under the principle of maximizing cumulative reward;

[0032] The control platform deployment module is used to integrate the trained reinforcement learning decision model into the IoT control platform. When a new terminal device is connected, it creates an intelligent agent instance of the corresponding object model, performs real-time parameter configuration, and adjusts the access strategy in real time according to changes in the terminal device's own state and network state.

[0033] Thirdly, an electronic device includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the reinforcement learning-based dynamic optimization method for object model access parameters as described in the first aspect of the invention.

[0034] Fourthly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the reinforcement learning-based method for dynamic optimization of object model access parameters as described in the first aspect of the present invention.

[0035] Beneficial effects:

[0036] (1) This invention provides a dynamic intelligent method for optimizing access parameters of IoT terminal devices. The IoT terminal devices are modeled as object model intelligent agents, and a multi-agent reinforcement learning training framework is established to enable the intelligent agents to dynamically adjust access parameters according to real-time status and adapt to device changes through dynamic training, thereby achieving efficient, low-latency, and low-power access. This effectively solves the problem of poor performance of traditional static access strategies in dynamic network environments. (2) Multi-agent collaborative optimization is achieved through a multi-agent near-end strategy optimization algorithm, which is suitable for large-scale heterogeneous terminal device scenarios and significantly improves the efficiency of global resource utilization. The centralized training and decentralized execution mode greatly reduces the computational load of terminal devices, enables real-time decision-making, and is suitable for edge computing environments. (3) Through this invention, the system can adapt to changes in the number of devices and dynamic changes in device status, has strong scalability and adaptability, ensures the stable operation of the power IoT, improves the access success rate of terminal devices, reduces power consumption and communication latency, and significantly improves the overall service quality of the power IoT. Attached Figure Description

[0037] Figure 1 A flowchart of a method and system for dynamic optimization of access parameters for object models based on reinforcement learning.

[0038] Figure 2 This is a schematic diagram of the training framework for a multi-agent proximal policy optimization algorithm.

[0039] Figure 3 It is a flowchart of the core functional interactions during the dynamic adjustment of access parameters.

[0040] Figure 4 This is the implementation architecture diagram of the object model access parameter dynamic optimization system. Detailed Implementation

[0041] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0042] like Figure 1 As shown, this embodiment of the invention provides a method for dynamically optimizing the access parameters of an object model based on reinforcement learning, including the following steps:

[0043] S1. Constructing Intelligent Agents for the Power Internet of Things (IoT) Object Model: Terminal devices in the power IoT are abstracted as intelligent agents, each corresponding to a pre-created device "object model" in the control platform. The object model defines the device's attributes and requirements in metadata form (e.g., a JSON Schema device description file), such as the device's service type and priority (e.g., latency-sensitive devices, energy-sensitive devices, bandwidth-sensitive devices, etc.) and other attributes, services, and events. The intelligent agent corresponds to the actual terminal device through this object model, used to optimize the network access parameters of the terminal device. In a multi-agent system, the access behaviors of different terminal devices may influence each other (e.g., interference caused by shared channels, competition for bandwidth resources, etc.), therefore, it is necessary to optimize the resource utilization of the entire system through multi-agent collaboration. Each object model intelligent agent corresponds to a type of access terminal device, allowing devices with the same business logic to share intelligent agent experience strategies, enabling rapid migration after new device access, suitable for power IoT environments with large-scale heterogeneous devices.

[0044] S2, Initialize the multi-agent reinforcement learning training framework: Before starting training, initialize the relevant parameters and model of the reinforcement learning algorithm. This invention uses the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm to solve the above-mentioned terminal access optimization problem. MAPPO adopts a Centralized Training, Decentralized Execution (CTDE) architecture. During the training phase, reinforcement learning training is centrally executed by an edge server or cloud control platform, and agents can utilize network-level global state information during training; during the execution phase, agents on each terminal device independently execute policy decisions based only on their own observations (device-level state). To implement the above training architecture, step S2 requires initializing the reinforcement learning training environment state space S and action space A, as well as the agent's policy network (Actor network) and value network (Critic network) parameters.

[0045] S21, State Space S: The state space includes device-level state and network-level global state. Device-level state refers to the state information of each terminal device, including but not limited to: current power consumption load, bandwidth usage, channel quality indicators; device service requirement parameters, such as service type, service priority, maximum tolerable latency, minimum allowable power, minimum required bandwidth, etc.; and communication process state, such as data processing latency and data transmission latency. Network-level state consists of global state information provided by the IoT control platform, including but not limited to: overall system computing resource utilization, bandwidth utilization of each wireless channel, total power consumption of all access devices; number of currently accessed object model devices, overall quality indicators of each channel, and global service latency statistics, etc. Through the above device-level and network-level state information, the state s of terminal access in the power IoT environment can be completely described.

[0046] S22, Action Space A: The action space includes multiple controllable parameter dimensions for dynamically optimizing terminal access. Specific actions may include: channel selection (discrete action), which involves selecting a suitable wireless channel for each agent to reduce transmission latency and interference; power control (discrete action), which involves adjusting the transmit power or operating power level of the terminal device to reduce energy consumption and processing latency while meeting communication requirements; bandwidth allocation (continuous action), which involves rationally allocating bandwidth to each terminal within limited total bandwidth resources to reduce transmission congestion and optimize transmission latency; data compression ratio adjustment (continuous action), which involves compressing data by a certain proportion before the terminal sends data to balance data transmission latency and device energy consumption; and computing resource allocation (continuous action), which involves the control platform allocating edge computing or cloud computing resources to the terminal device to assist in terminal data processing, thereby reducing the processing burden and latency on the terminal side. These actions can be selected and executed by the agent based on its current state, thus forming a dynamic optimization decision for the terminal access parameters.

[0047] S3, Determine the optimization objective and reward function: The reward function is designed to simultaneously reflect global network performance and the access effectiveness of individual devices, thereby guiding agents to optimize strategies towards improving overall resource utilization, successful access rate, and reducing latency and power consumption. Global performance metrics (such as the percentage of successful accesses among all devices, average latency, total power consumption, and resource utilization) can be combined with the performance of individual devices to form the immediate reward value for each agent. For example, the reward function R for agent i can be defined. (i) As shown below:

[0048]

[0049] R global This is a global performance indicator, also known as a global reward. Let θ1 and θ2 be the performance of a single terminal device corresponding to agent i, i.e., the local reward, and θ1 and θ2 be the weight parameters.

[0050] S31, Global Reward aims to improve overall system performance, and defines the global reward R. global The system comprehensively reflects the following indicators: terminal device access success rate, average terminal communication latency, total terminal power consumption, bandwidth, and computing resource utilization. Therefore, the global reward objective is to maximize the proportion of successfully accessed terminal devices, minimize the average communication latency of all terminal devices, minimize total system power consumption, and improve bandwidth and computing resource utilization. For example, if the total number of terminals in the system at a certain moment is N, and the priority of the terminal devices is P... i The access quality of the terminal device is Q. i The latency of the terminal device is T. i Power consumption is W i The bandwidth occupied is B iGiven the utilization rate of communication and computing resources, U, the global reward is calculated as follows:

[0051]

[0052] Where S is the set of terminal devices that have successfully connected, α, β, and η are weighting coefficients, and T... lim U is the maximum tolerable latency for terminal devices. safe U is the safe threshold for resource utilization. max k represents the maximum resource utilization rate. U An exponential adjustment term is used to control the growth curve of the utilization rate penalty.

[0053] In S32, local rewards are performance feedback for a single terminal device, and each agent can also receive local rewards based on its own local conditions. If the i-th terminal device successfully connects and meets its service QoS requirements, a positive reward is given; if the connection fails or the requirements are not met, a penalty is given; and the latency and power consumption of the terminal device are negatively evaluated (the higher the latency or power consumption, the lower the reward). As an example, the local reward calculation for the terminal device is as follows:

[0054]

[0055] The access quality of the terminal device is calculated using the following indicator function:

[0056] Q i =σ(α1(B) i -B lim ))·σ(α2(W i -W lim ))·σ(α3(T lim -T i ))

[0057] σ(·) is the Sigmoid smoothing function that makes the reward differentiable, B lim W lim These represent the minimum bandwidth and power consumption required for terminal equipment services, respectively; α1, α2, and α3 are smoothing control adjustment terms, with larger values ​​indicating a closer approximation to a hard indicator function; λ i and ω i These are the weighting coefficients.

[0058] Through this reward design, the agent not only focuses on its own access performance during training but also takes responsibility for the overall system performance to a certain extent, encouraging cooperation to avoid performance degradation caused by resource contention. It should be noted that the reward function form described above can be adjusted according to actual needs, and the weight coefficients and specific function form can be determined experimentally to ensure stable convergence during training and achieve the expected optimization goals.

[0059] S4, Training the MAPPO agent. During training, the agents collectively interact in a simulation environment according to the current policy to collect state transition and reward samples. (Refer to...) Figure 2 Each agent's Actor network outputs access action decisions for a given state. Multiple agents can share the same Actor network parameter initialization scheme, but with their own input states. The Critic network evaluates the value of the current policy for a given global state. Multiple agents share a global Critic network during training to assess overall performance. Reinforcement learning hyperparameters such as discount factor γ, learning rate ν, and policy cutoff threshold ∈ are set, and data storage structures such as the experience replay pool are initialized to prepare for the training process.

[0060] S41, Environment Interaction and Experience Storage: The interaction experience data collected by each agent in step S2 is stored in the Experience Replay Buffer. The experience buffer caches a large number of samples generated by multiple agents during several rounds of environment interaction, allowing for random sampling during training, thereby improving learning stability and sample utilization efficiency. Before each policy update, a batch of samples (mini-batch) is randomly selected from the experience replay buffer to form a training batch. This batch of samples contains data such as states, actions, rewards, and the corresponding next state for multiple time steps, covering multiple agents and various environmental situations. By introducing the experience buffer and random sampling mechanism, the correlation between sequence samples in reinforcement learning can be reduced, improving the convergence stability of training.

[0061] S42, Calculate the advantage function and evaluate the current policy: Calculate the advantage value for each sample in the batch. The advantage function A(s,a) is used to evaluate how good or bad an action is relative to the average level in a specific state, and can reduce the variance of the policy gradient algorithm. First, the actual reward accumulated from each experience sequence in the batch (e.g., the total reward obtained by summing and discounting immediate rewards) is used as the target value. Let G be the target value. t The cumulative return for discounts starting at time t is calculated using the following formula:

[0062] G t =r t +γr t+1 +γ 2 r t+2 +…+γ n r t+n

[0063] Where γ is the discount factor, r tLet be the reward for the agent at time t, and n represent the round step size starting from time t. The sequence terminates at time t+n or reaches a predetermined length. Then, the Critic value network is applied to state s. t The estimated value is denoted as V(s) t If ), then the advantage function can be expressed as A. t =G t -V(s t ). When A t A positive odds value indicates that the actual return is higher than the expected value, and the action outperforms the average strategy; conversely, a negative odds value indicates that the action is less effective. The calculated odds value is used to evaluate each state-action sample in the batch, providing guidance for strategy updates.

[0064] S43, Update the Actor Policy Network: Based on the calculated advantage value, update the agent's policy network parameters using a policy gradient algorithm to increase the probability of executing high-quality actions. To ensure the stability of the policy update, this invention employs the policy clipping technique from Proximal Policy Optimization (PPO) to constrain the policy gradient.

[0065] First, define the probability ratio between the old and new strategies:

[0066]

[0067] Where, π θ (a t |s t ) indicates that in state s according to the current policy parameter θ t Select action a t The probability or probability density of . This represents the probability of the old policy before the update. Then, the objective function after policy pruning is expressed as:

[0068] L CLIP (θ)=E[min(ρ t (θ)A t ,clip(ρ t (θ), 1-∈, 1+∈)A t )]

[0069] Where clip[,r,,1-∈,,1+∈,] represents the ratio ρ t (θ) is restricted to between (1-∈) and (1+∈), when ρ t When it exceeds this range, it is truncated to the boundary value. By maximizing L clip(Equivalent to minimizing its negative value), the policy network parameters θ can be updated. The policy pruning mechanism ensures that the magnitude of each policy update does not deviate too much from the original policy; that is, when the difference between the old and new policies is too large, only the smaller advantage after pruning is adopted, thereby avoiding drastic policy oscillations and improving training stability. The Actor network parameters are iteratively updated using the stochastic gradient ascent (or descent) method, so that the agent's policy gradually tends towards the direction with higher cumulative rewards.

[0070] S44, Update the Critic Value Network (Value Function Optimization): Train the Critic network using batch samples and corresponding cumulative target rewards to achieve more accurate and reliable estimation of state values. The loss function of the Critic network can be in the form of Mean Squared Error (MSE), comparing the predicted state value with the actual discounted cumulative reward.

[0071]

[0072] Wherein, V(s) t ;w) represents the state s under Critic network parameters w. t Value estimate, G t The cumulative discount reward calculated in S42 is used. The Critic network parameters w are updated by minimizing the above loss function, for example, for L... V The gradient of w is calculated iteratively using stochastic gradient descent. Optimizing the Critic network allows it to more accurately approximate the environment's value function, thus providing a more precise benchmark for subsequent policy updates and improving the effectiveness of the policy gradient.

[0073] S45, Introducing Soft Update of the Target Network: To further enhance training stability, in this embodiment of the invention, corresponding target networks are maintained for the policy network and value network, respectively, and the target network parameters are synchronized using a soft update strategy. Soft update refers to gradually approximating the target network with small step sizes, rather than directly copying the current network parameters to the target network each time (hard update). Let θ ′ Let θ represent the target network parameters of the Actor, and θ represent the current network parameters of the Actor. Then, a soft update is performed in each iteration.

[0074] θ ′ ←τθ+(1-τ)θ ′

[0075] Similarly, for the Critic network, we execute ,w′←τ,w+(1-τ),w′, where τ is the soft update coefficient, taking a positive number much less than 1 (e.g., 0.005). Through incremental updates, only a small portion of the current network parameters is injected into the target network each time, smoothly tracking the latest policy changes. Soft updates avoid the network instability that can be caused by hard updates, helping to mitigate oscillations during training and making policy convergence more stable.

[0076] After iterative training from S41 to S45, when the agent's policy network converges, meaning the policy optimization objective (e.g., cumulative reward) stabilizes, the offline learning of the optimal access policy is complete. At this point, the Actor network parameters of each agent correspond to the optimal policy model for optimizing the object model's access parameters.

[0077] S5, Model Deployment and Dynamic Adjustment of Access Parameters: The trained and optimized strategy model is integrated into the IoT control platform to perform real-time parameter configuration for newly connected devices and dynamic adjustment of access strategies for existing devices to adapt to changing environments. Specifically, refer to... Figure 3 Step S5 may include:

[0078] (1) New Device Access Identification and Agent Initialization: When a new terminal device is accessed by the power Internet of Things, the control platform first identifies the type of the device (e.g., identified as a smart meter, sensor, distribution terminal, or other specific category), loads the corresponding object model description, and assigns or creates an agent instance for it. The agent can be initialized based on pre-trained policy parameters (e.g., loading a general initial policy for known device types), enabling it to immediately participate in network resource allocation and collaborative decision-making. After the new agent is initialized, its device state is incorporated into the global state space for management.

[0079] (2) Automatic Configuration of Optimal Access Parameters: Based on the trained agent strategy, the control platform automatically selects the optimal access method and parameter configuration combination for new access devices. For example, based on the current network state, the agent strategy may decide to select the channel with the least interference and allocate appropriate bandwidth and transmit power to the device, or prioritize scheduling computing resources for devices with low latency requirements to shorten processing time. By applying the agent's decision-making strategy, the platform can adaptively allocate communication and computing resources to each device, achieving plug-and-play optimal access configuration without manual intervention.

[0080] (3) Real-time adjustment during operation: During the operation of the device, if changes in the device's status or network environment are detected (e.g., increased device traffic load, decreased channel quality, or increased competition due to the addition of other devices), the control platform will dynamically adjust the device's access strategy based on the feedback reward signal and new status information obtained by the agent. Specifically, the agent reselects or adjusts its actions based on the changed status (e.g., switching to a better channel, increasing bandwidth allocation or power, etc.), and the new decision will immediately apply to the device's access parameters, thereby maintaining optimal overall performance. As time goes on and the system scales up, the control platform can add new agents as needed to cover newly added object models and terminal devices. The multi-agent collaborative mechanism of the entire system can be extended to thousands or tens of thousands of devices while still maintaining efficient operation. The method described in this invention ensures that in the dynamically changing power Internet of Things environment, each terminal device always accesses the network with a near-globally optimal configuration, improving communication efficiency, reducing power consumption, and maintaining the stability and reliability of network services.

[0081] This invention also provides a dynamic optimization system for access parameters of an object model based on reinforcement learning, comprising:

[0082] The object model intelligent agent construction module is used to construct object model intelligent agents based on the business needs, priorities and resource needs of terminal devices in the power Internet of Things. Each object model intelligent agent corresponds to a type of access terminal device.

[0083] The state monitoring and acquisition module is used to collect device-level and network-level state information from each terminal device, and model this state information as the current environment state S for reinforcement learning. t ;

[0084] The decision execution module is used to carry the reinforcement learning decision model. The reinforcement learning decision model makes decisions based on the current environment state S input. t Based on the multi-agent reinforcement learning algorithm, the system outputs the access parameter optimization decision for each terminal device and sends the decision to the corresponding terminal device to perform the access operation.

[0085] The strategy training module is used to calculate the reward value R based on the execution results of each terminal device. t According to the current environmental state S t Action decision and the reward value R obtained t Update the policy parameters of the reinforcement learning decision model and optimize the access parameter strategy of each terminal device under the principle of maximizing cumulative reward;

[0086] The control platform deployment module is used to integrate the trained reinforcement learning decision model into the IoT control platform. When a new terminal device is connected, it creates an intelligent agent instance of the corresponding object model, performs real-time parameter configuration, and adjusts the access strategy in real time according to changes in the terminal device's own state and network state.

[0087] It should be understood that the reinforcement learning-based object model access parameter dynamic optimization system can implement all the technical solutions in the above method embodiments. The functions of each functional module can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above embodiments, which will not be repeated here.

[0088] In embodiments of the present invention, the system implementation layer includes a terminal device layer, an intelligent agent layer, and a control platform layer, and the relationships between them are as follows: Figure 4 As shown, the terminal device layer supports dynamic registration and identification of devices, enhancing the system's adaptive capabilities; it implements a lightweight policy execution mechanism to adapt to the operating requirements of low-power terminals; and it provides accurate and real-time operating status data to support policy training and optimization.

[0089] In terms of functional implementation, the device identification function at the terminal device layer reports the device's object model metadata (such as JSON Schema) and specific instance parameters to the control platform when the terminal device connects, thereby achieving device registration and type identification. Initial modeling of the object model is completed by collecting parameters such as service type, latency / power consumption / bandwidth sensitivity.

[0090] Device Interaction Terminal Device Layer: Interacts with terminal devices, including receiving optimization strategy parameters (such as channel selection, power level, bandwidth allocation, etc.) issued by the intelligent agent, and collecting and reporting device operating status (such as channel quality, power consumption, communication latency, etc.) in real time to support the reinforcement learning training process.

[0091] The control platform layer enables global management and training control of the reinforcement learning system; supports parallel optimization tasks for multiple object models and multiple devices; provides the ability to dynamically adapt to network load and terminal heterogeneity; and constructs a training-deployment closed loop to improve policy generalization capabilities.

[0092] In terms of functionality, the control platform layer's agent management function stores and manages the TSL (Thing Specification Language) description files of all object models in the system. Based on the object model type corresponding to the terminal device, it assigns or creates a dedicated agent for it and establishes a mapping relationship between state observation and behavior.

[0093] Training control function: Executes a centralized training process during model training and dynamic optimization phases, organizing parallel interactions among multiple agents and batch policy update tasks. Supports policy convergence control and training round management, and combines PPO policy pruning and target network soft update mechanisms to improve training stability.

[0094] Experience replay pool function: It is used to centrally store the state-action-reward sequence (SARS) generated by the interaction between each agent and the environment, and integrate global system information and local observation data of the device to calculate its global reward function (such as the overall power consumption of the system, access rate, latency, etc.), so as to provide high-quality samples for the training process.

[0095] The Critic network is responsible for calculating the Advantage value (advantage function) based on the sequences stored in the experience pool, which guides the gradient optimization process of the Actor policy network. By training the Critic network to evaluate the state-value function, the merits of the current policy can be accurately characterized.

[0096] The intelligent agent layer supports parallel reinforcement learning tasks across multiple devices and scenarios; implements a high-precision policy update and deployment mechanism; dynamically senses environmental changes and optimizes terminal device access policies in real time; and improves the overall system access success rate, resource utilization, and quality of service (QoS).

[0097] In terms of functionality, the Actor network of the agent layer updates the policy network based on the Advantage evaluation results, determining the optimal access action to be taken in the current state. Clip-PPO technology is used for policy pruning, combined with target network soft update technology, to improve the stability and efficiency of the learning process.

[0098] Policy execution function: Apply the trained policy to specific device scenarios and adjust the access parameters of terminal devices (such as channel, power, bandwidth, compression rate, etc.) in real time to ensure efficient and low-power device access in dynamic environments.

[0099] Environmental interaction function: Each agent establishes a binding relationship with its corresponding device, and forms a local environmental interaction process by sensing the device status and obtaining reward signals. During training, local reward experience at the device level is uploaded, and during deployment, optimization parameters are obtained from the policy execution function module and sent to the device.

[0100] Another embodiment of the present invention provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement the steps of the reinforcement learning-based object model access parameter dynamic optimization method as described above.

[0101] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the dynamic optimization method for access parameters of an object model based on reinforcement learning as described above.

[0102] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus (systems), electronic devices, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This invention is described with reference to a flowchart of a method according to embodiments of the invention. It should be understood that each step in the flowchart and combinations thereof can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A device for a function specified in one or more processes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

Claims

1. A method for dynamic optimization of access parameters of an object model based on reinforcement learning, characterized in that, Includes the following steps: Based on the business needs, priorities, and resource needs of terminal devices in the power Internet of Things, construct object model intelligent agents, with each object model intelligent agent corresponding to a type of access terminal device. Collect device-level and network-level state information from each terminal device, and model this state information as the current environment state S for reinforcement learning. t ; Set the current environmental state S t Input a pre-built reinforcement learning decision model, which outputs access parameter optimization decisions for each terminal device based on a multi-agent reinforcement learning algorithm. Adjust the access parameters of each terminal device according to the access parameter optimization decisions and execute device access operations. Calculate the reward value R based on the access execution results of each terminal device. t According to the current environmental state S t Action decision and the reward value R obtained t Update the policy parameters of the reinforcement learning decision model and optimize the access parameter strategy of each terminal device under the principle of maximizing cumulative reward; The trained reinforcement learning decision model is integrated into the IoT control platform. When a new terminal device is connected, a corresponding agent instance of the object model is created, parameters are configured in real time, and the access strategy is adjusted in real time according to changes in the terminal device's own state and network state.

2. The method according to claim 1, characterized in that, The device-level status information of the terminal device includes the status information of the terminal device itself, as well as the communication process status information of the terminal device; Network-level status information includes global status information provided by the IoT control platform.

3. The method according to claim 1, characterized in that, The action space of a reinforcement learning decision model includes the following access parameter adjustment actions: Channel selection: Selecting the communication channel for terminal devices to access in order to optimize their data transmission latency; Power control: Adjusting the operating power or transmit power level of the terminal equipment to reduce energy consumption and processing latency while ensuring performance; Bandwidth allocation: Allocating bandwidth reasonably to terminal devices under limited bandwidth resources in order to optimize their transmission latency; Data compression rate adjustment: Adjustments are made to the data reporting rate of terminal devices to achieve a balance between increasing compression to reduce transmission volume and reducing compression to reduce latency, thereby optimizing energy consumption and transmission latency; Computing resource allocation: Edge / cloud computing resource quotas allocated by the IoT control platform to terminal devices to meet their data processing needs and optimize end-to-end latency.

4. The method according to claim 1, characterized in that, The calculation of the reward value takes into account both global and local rewards. The global reward aims to improve the overall system performance, including maximizing the proportion of successfully connected terminal devices, minimizing the average communication latency of all terminal devices, minimizing the total power consumption of the system, and improving the utilization of bandwidth and computing resources. The local reward is a performance feedback for a single terminal device, including a positive reward when the terminal device successfully connects, and a penalty or additional reward based on the actual communication latency and energy consumption of the terminal device; the total reward R for all agents is... t It is obtained by weighted combination of global and local rewards and is used to guide each agent to optimize its access parameter decisions.

5. The method according to claim 1, characterized in that, The reinforcement learning decision model is trained using a multi-agent near-end policy optimization algorithm and adopts a centralized training and distributed execution architecture. This includes: acquiring network-level state information during the training phase and using a centralized evaluator to evaluate the joint policy of the multi-agents and update the shared value function parameters; during the execution phase, each agent on a terminal device independently selects actions based on its own observed device-level state information using the trained policy, thereby achieving a near-globally optimal access parameter configuration policy in large-scale terminal device scenarios and reducing the computational and communication overhead of real-time decision-making.

6. The method according to claim 5, characterized in that, During the training phase, the policy update of the reinforcement learning decision model employs a policy gradient truncation and target network soft update mechanism, specifically including: Using the advantage function A t The merits of actions under the current policy are evaluated, and the advantage function is defined as the difference between the actual reward obtained by the current action and the benchmark value. Based on the advantage function, a shearing strategy is adopted to optimize the objective, and the policy probability ratio is restricted during policy updates. The variation range is within the preset range [1-∈, 1+∈], where π new (a t |s t ) indicates that in state s according to the current policy parameter θ t Select action a t The probability or probability density of . This represents the corresponding probability or probability density of the old policy before the update, where ∈ is the policy truncation threshold; The target network parameters are updated using a soft update method, meaning that in each training iteration, the new parameters of the policy network are incorporated into the target network parameters proportionally τ: θ target ←τ,θ new +(1-τ),θ target , where θ target Let θ be the target network parameter. new Let τ be a new parameter for the policy network, where τ is a soft update coefficient less than 1.

7. The method according to claim 1, characterized in that, When a new terminal device is connected to the IoT control platform, the following steps are performed: identify the object model type to which the new terminal device belongs, initialize the corresponding intelligent agent according to the predefined parameters of the object model; incorporate the initial state information of the new terminal device into the environmental state of the reinforcement learning decision model, and generate the access parameter configuration scheme of the terminal device based on the optimal policy obtained from the current training, so that the new terminal device can connect to the network with the optimal parameters in an instant.

8. A dynamic optimization system for access parameters of an object model based on reinforcement learning, characterized in that, The system includes: The object model intelligent agent construction module is used to construct object model intelligent agents based on the business needs, priorities and resource needs of terminal devices in the power Internet of Things. Each object model intelligent agent corresponds to a type of access terminal device. The state monitoring and acquisition module is used to collect device-level and network-level state information from each terminal device, and model this state information as the current environment state S for reinforcement learning. t ; The decision execution module is used to carry the reinforcement learning decision model. The reinforcement learning decision model makes decisions based on the current environment state S input. t Based on the multi-agent reinforcement learning algorithm, the system outputs the access parameter optimization decision for each terminal device and sends the decision to the corresponding terminal device to perform the access operation. The strategy training module is used to calculate the reward value R based on the execution results of each terminal device. t According to the current environmental state S t Action decision and the reward value R obtained t Update the policy parameters of the reinforcement learning decision model and optimize the access parameter strategy of each terminal device under the principle of maximizing cumulative reward; The control platform deployment module is used to integrate the trained reinforcement learning decision model into the IoT control platform. When a new terminal device is connected, it creates an intelligent agent instance of the corresponding object model, performs real-time parameter configuration, and adjusts the access strategy in real time according to changes in the terminal device's own state and network state.

9. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the reinforcement learning-based dynamic optimization method for object model access parameters as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dynamic optimization method for access parameters of object models based on reinforcement learning as described in any one of claims 1-7.

Images