Intelligent agent cooperation intelligent decision-making method, system and equipment for substation environment control, and medium

Abstract

The invention discloses an intelligent agent cooperation intelligent decision-making method, system, equipment and medium for transformer substation environment control, and belongs to the technical field of intelligent operation and maintenance, and the method comprises the steps: collecting transformer substation historical data, building an intelligent agent interaction environment, carrying out the modeling of a decision-making problem, and carrying out the state estimation according to local observation and the transformer substation historical data, the method comprises the following steps: constructing a global objective function, predicting an individual contribution degree through a neural network model, defining an individual reward signal according to the individual contribution degree, calculating a return index through a primary algorithm, carrying out strategy training and optimization according to the return index, executing a decision, carrying out normalization processing on the individual reward signal, and introducing a regularization item into the global objective function. And calculating a convergence index and monitoring the convergence. According to the method, state sensing and decision making are carried out, the problems of low operation and maintenance efficiency and poor collaboration in substation control are solved, automatic decomposition and efficient decision making of a global optimization target are achieved, and the self-adaptive capacity and the overall energy efficiency of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology, specifically to a method, system, device, and medium for intelligent agent collaborative intelligent decision-making in substation environmental control. Background Technology

[0002] With the development of the energy internet and smart grids, substation operation and maintenance are gradually transforming towards digitalization, networking, and intelligence. However, in the current substation operation and management, the air conditioning system, as an important component for maintaining stable room temperature and humidity, still suffers from relatively outdated management methods. Traditional substation air conditioning systems mostly rely on local control panels or remote controls operated manually on-site to start / stop, switch modes, and set temperatures, lacking remote monitoring and intelligent adjustment capabilities.

[0003] Traditional methods of operation and maintenance are inefficient and have slow response times. When abnormal room temperature and humidity or air conditioning failures occur, maintenance personnel need to go to the site for inspection and handling, which makes it impossible to achieve remote sensing and real-time control. This results in prolonged periods of environmental abnormalities, affecting the safe operation of power equipment.

[0004] Traditional air conditioners lack network monitoring mechanisms, making it difficult to collect and upload operational data in real time. This prevents managers from understanding current temperature and humidity changes and the air conditioner's operating status. When abnormal operation or decreased cooling performance occurs, it cannot be detected and addressed promptly, easily leading to problems such as overheating of electrical equipment and insulation aging, increasing safety risks.

[0005] Traditional air conditioning control systems are limited by their single control method and lack intelligent scheduling. They typically only have temperature threshold control and lack intelligent adjustment mechanisms based on load forecasting, energy consumption optimization, and multi-device collaboration. This makes it difficult to meet the development needs of modern substations for "unmanned operation, centralized monitoring, and energy-saving operation".

[0006] Therefore, upgrading existing substation air conditioning systems to be intelligent, enabling remote monitoring, intelligent decision-making, and adaptive control, has become an important direction in the current power operation and maintenance field. By introducing high-precision wireless temperature and humidity sensors to collect environmental parameters in real time and utilizing the secure and reliable WAPI wireless communication protocol to transmit the data to the backend micro-environment control system in real time, the stability and security of data interaction can be significantly improved. Combined with large-model-assisted decision-making algorithms and set temperature and humidity threshold strategies, the system automatically analyzes environmental change trends and dynamically issues air conditioning control commands, enabling remote start / stop, mode switching, and parameter adjustment of the air conditioning, thereby achieving intelligent and automated micro-environment control. Summary of the Invention

[0007] In view of the above-mentioned problems, the present invention is proposed.

[0008] Therefore, the technical problem solved by the present invention is that the existing technology in the operation and maintenance management of substation air conditioning has low operation and maintenance efficiency, slow response, invisible status and difficult fault early warning, single control means and lack of intelligent scheduling.

[0009] To address the aforementioned technical problems, this invention provides the following technical solution: a method for agent-based collaborative intelligent decision-making in substation environmental control, comprising, Collect historical data from substations, build an intelligent agent interaction environment, model the decision problem, estimate the state based on local observations and historical data from substations, and construct a global objective function; Based on the global objective function, the individual contribution is predicted by a neural network model, and the individual reward signal is defined according to the individual contribution. The return index is calculated by a first-level algorithm, and the strategy is trained and optimized and the decision is executed based on the return index. The individual reward signal is normalized, the regularization term is introduced into the global objective function, the convergence index is calculated, and the convergence is monitored.

[0010] As a preferred embodiment of the agent-based collaborative intelligent decision-making method for substation environmental control described in this invention, the steps include: collecting historical substation data, building an agent interaction environment, modeling the decision problem, performing state estimation based on local observations and historical substation data, and constructing a global objective function. Collect historical data from substations, build an intelligent agent interaction environment, and model the decision-making problem through random game theory; Based on the agent interaction environment, state estimation is performed according to local observations and historical data of the substation, and the state representation vector of the agent is output. A global objective function is constructed based on the agent's state representation vector.

[0011] As a preferred embodiment of the agent-based collaborative intelligent decision-making method for substation environmental control described in this invention, the steps include: predicting individual contributions using a neural network model based on a global objective function, defining individual reward signals based on individual contributions, calculating reward indicators using a first-level algorithm, and training and optimizing strategies based on the reward indicators before executing decisions. Based on the global objective function, the individual contribution is defined and predicted by a neural network model, and the individual contribution is output. Individual reward signals are defined based on individual contribution, and return indicators are calculated using a first-level algorithm. The agent trains and optimizes the strategy based on the reward metrics, and then executes the strategy.

[0012] As a preferred embodiment of the agent-based collaborative intelligent decision-making method for substation environmental control described in this invention, the steps of normalizing individual reward signals, introducing a regularization term into the global objective function, calculating a convergence index, and monitoring convergence include: The individual reward signal is normalized, and the regularization term is introduced into the global objective function; Based on the normalized individual reward signal, a convergence index is defined and calculated, and the convergence index is monitored. Adjust the parameters for strategy training based on the relationship between the convergence metric and the set convergence metric threshold.

[0013] As a preferred embodiment of the agent-based collaborative intelligent decision-making method for substation environmental control described in this invention, the construction of the global objective function based on the agent's state representation vector includes: The goal of a swarm of agents is to learn a joint policy. To maximize the long-term cumulative global return, the global objective function is expressed as follows: in, The global objective function is... To track Expectations Discount factor Power-decay weights for time step t The desired time constant, For the actual time interval, For global instant rewards, For the task time range, For trajectory and orbital distribution.

[0014] As a preferred embodiment of the agent-based collaborative intelligent decision-making method for substation environmental control described in this invention, the calculation of the return index through a first-level algorithm includes: intelligent agent The optimization objective is formalized as maximizing the expected return, expressed as: in, For agent i, when adopting the policy function The expected long-term cumulative return Let i be the policy function of agent i. To track Expectations Discount factor Power-decay weights for time step t For the task time range, For trajectory and orbital distribution, The individualized reward for agent i at time t.

[0015] This invention explicitly formalizes the agent's optimization objective as maximizing the expected long-term discounted return based on individualized rewards. The discount factor balances the impact of immediate and long-term returns on current decisions, enabling the agent to learn a strategy that balances short-term efficiency and long-term system stability. This ensures that each agent's strategy updates directly target improving the value it contributes, guaranteeing the practical operability of the training process.

[0016] As a preferred embodiment of the agent-based collaborative intelligent decision-making method for substation environmental control described in this invention, the step of normalizing individual reward signals and introducing a regularization term into the global objective function includes: Introducing a reward normalization operator: in, To normalize individual reward signals, To reward the normalization operator, For agent i, the individualized reward at time t. and These are the mean and standard deviation of the individual reward during the most recent training window, respectively. To prevent constants with a denominator of zero; An entropy regularization term is introduced into the policy optimization objective, with the expression: in, Let the objective function be the policy objective function after entropy regularization. The entropy of the agent's policy. For agent i, when adopting the policy function The expected long-term cumulative return To track Expectations Let i be the policy distribution of agent i. This is the adjustment coefficient.

[0017] This invention introduces reward normalization and entropy regularization mechanisms. The reward normalization operation eliminates the differences in the dimensions and scales of individual reward signals, ensuring the stability and comparability of policy updates for different agents and avoiding training oscillations caused by uneven distribution of reward values. The entropy regularization term directly incentivizes the randomness of policy distribution, preventing agents from prematurely converging to local optimal deterministic policies.

[0018] This invention provides a system for intelligent agent collaborative decision-making for substation environmental control.

[0019] To address the aforementioned technical problems, this invention provides the following technical solution: a system for agent-based collaborative intelligent decision-making in substation environmental control, comprising: a decision problem modeling module, an individual contribution prediction module and a strategy training module, and a convergence monitoring module. The decision problem modeling module collects historical data from substations, builds an intelligent agent interaction environment, models the decision problem, estimates the state based on local observations and historical data from substations, and constructs a global objective function. The individual contribution prediction module and strategy training module are based on a global objective function, use a neural network model to predict individual contributions, define individual reward signals based on individual contributions, calculate reward indicators through a first-level algorithm, and perform strategy training and optimization based on reward indicators before executing decisions. The convergence monitoring module normalizes the individual reward signal, introduces a regularization term into the global objective function, calculates the convergence index, and monitors the convergence.

[0020] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the method for intelligent agent collaborative intelligent decision-making for substation environmental control.

[0021] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of a method for intelligent agent collaborative intelligent decision-making for substation environmental control.

[0022] The beneficial effects of this invention are as follows: This invention constructs an interactive environment and a global objective function based on historical data, and designs a reward allocation mechanism based on individual contribution prediction, combining reward calculation and strategy optimization to achieve collaborative decision-making; it introduces reward normalization and entropy regularization to suppress premature convergence during training and enhance strategy exploration capabilities; it models the substation control problem as a stochastic game, uses local observations and historical data for state estimation, and through hierarchical objective functions and reward design, enables the agent to promote the achievement of the global objective while optimizing individual rewards, thereby improving the adaptive and collaborative control performance of the system in complex environments. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating an overall process for an agent-based collaborative intelligent decision-making method for substation environmental control, as provided in one embodiment of the present invention. Detailed Implementation

[0025] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0026] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a method for agent-based collaborative intelligent decision-making for substation environmental control, comprising: To address the shortcomings of existing technologies in substation air conditioning operation and maintenance management, such as low efficiency, delayed response, lack of visibility of status and difficulty in early warning of faults, limited control methods, and lack of intelligent scheduling, this invention provides a method for intelligent agent collaborative intelligent decision-making for substation environmental control.

[0027] S1: Collect historical data of substations, build an intelligent agent interaction environment, model the decision problem, estimate the state based on local observations and historical data of substations, and construct a global objective function.

[0028] S2: Based on the global objective function, predict individual contribution through a neural network model, define individual reward signals based on individual contribution, calculate reward indicators through a first-level algorithm, train and optimize strategies based on reward indicators, and execute decisions.

[0029] S3: Normalize the individual reward signal, introduce the regularization term into the global objective function, calculate the convergence index, and monitor the convergence.

[0030] Therefore, a precise system state perception and global optimization objective are established through historical data and environmental modeling; an individual reward mechanism based on contribution prediction is used to reasonably decompose the global objective and guide individual strategy optimization, ensuring the consistency between local decision-making and overall effectiveness; normalization and regularization techniques are introduced to improve the stability of the training process and avoid premature convergence.

[0031] Example 2, an embodiment of the present invention, provides a method for agent-based collaborative intelligent decision-making for substation environmental control based on the previous embodiment, comprising: In this embodiment of the application, step S1 involves collecting historical data of the substation, building an intelligent agent interaction environment, modeling the decision problem, estimating the state based on local observations and historical data of the substation, and constructing a global objective function, including the following steps A1-A3: A1: Collect historical data from substations, build an intelligent agent interaction environment, and model the decision-making problem through random game theory.

[0032] A2: Based on the agent interaction environment, state estimation is performed according to local observations and historical data of the substation, and the state representation vector of the agent is output.

[0033] Since most real-world environments are partially observable scenarios, intelligent agents The global state cannot be obtained directly. Based on its own local observation With historical information State estimation is performed using the following expression: in, For feature extraction function, parameters It can be implemented using deep neural networks. For historical trajectories or communication information from neighbors; For intelligent agents The state representation vector belongs to a high-dimensional continuous space. .

[0034] A3: Construct a global objective function based on the agent's state representation vector.

[0035] The goal of a swarm of agents is to learn a joint policy. To maximize the long-term cumulative global return, the global objective function is expressed as follows: in, The global objective function is... To track Expectations Discount factor Power-decay weights for time step t The desired time constant, For the actual time interval, For global instant rewards, For the task time range, For trajectory and orbital distribution, The system employs a joint strategy, comprising N agents. The policy function of the i-th agent is: joint strategy It is a combination of all agent policies. Accumulated rewards for discounts.

[0036] In this embodiment of the application, the modeling of the decision problem through random game in step A1 specifically employs partially observable Markov games: The system includes There are 3 intelligent agents, each interacting with the environment at any given time. The overall state of the environment is denoted as . ,in Representing the state space. Since the environment is often partially observable, the agent... It cannot directly obtain the global state, but can only receive local observations. Based on available information (Generally, it is an observation-action history; in special cases, it can degenerate into real-time observation.) ), intelligent agent From the action space Select Action The joint actions of multiple agents constitute a vector. Under the combined action, the environment follows the transfer function. Evolve to the next state, while providing a global instant reward. .

[0037] In an alternative implementation, the decision problem modeled by random games in step A1 can also be modeled by a fully observable Markov game. This assumes that the agent can directly observe the global state without needing to estimate the state through historical information or local observations. The decision-making process is based on the global state, and joint actions affect state transitions and generate global rewards.

[0038] In another alternative implementation, the decision problem modeled by random game in step A1 can also be modeled using a distributed partially observable Markov decision process, which models the multi-agent system as a joint decision-maker. Each agent makes decisions based on its own local observation history, and there is no central controller in the system to perform information fusion or issue instructions. The goal is still to maximize the long-term global reward, but policy learning and execution are entirely based on local information flow.

[0039] In the implementation of this application, the decision problem is modeled by random game in step A1, and the multi-agent decision problem of substation is modeled by partially observable Markov game to ensure the consistency between algorithm design and real application scenario.

[0040] In this embodiment of the application, step S2 predicts individual contribution based on a global objective function using a neural network model, defines an individual reward signal based on the individual contribution, calculates a reward index using a first-level algorithm, trains and optimizes the strategy based on the reward index, and executes the decision, including the following steps B1-B3: B1: Based on the global objective function, define the individual contribution and predict the individual contribution through a neural network model, and output the individual contribution.

[0041] Introducing individual contribution function Used to characterize intelligent agents In global state With joint actions The marginal effect under [a certain condition]. Intuitively, this function measures the marginal effect of an agent. Changes in the actions of the system will affect its overall performance.

[0042] In mathematical terms, an individual's contribution can be defined as: in, For individual contribution, Indicates the state and joint actions The overall value of the following Indicates except for intelligent agents External combined actions set Represents intelligent agents The alternative benchmark action.

[0043] B2: Define individual reward signals based on individual contribution and calculate return indicators through a first-level algorithm.

[0044] B3: The agent trains and optimizes the strategy based on the reward metrics and then executes the strategy.

[0045] In policy optimization, this invention employs a gradient-based optimization approach. For the agent... Its strategy parameters are denoted as The gradient update direction is given by the expectation, and its expression is: in, Let i be the policy objective function. Regarding its strategy parameters gradient, To track Expectations To take the logarithm of the policy probability and then adjust the parameters Find the gradient. Indicates time Advantage function estimation is used to characterize the action. The degree of superiority or inferiority compared to the average level.

[0046] An improved individual reward is introduced into the definition of the advantage function, expressed as: in, Indicates time Dominance function estimation under the following conditions For intelligent agents Value function estimator, For agent i, the individualized reward at time t. As a discount factor, , This represents the state.

[0047] In this way, the agent can utilize both immediate contribution signals and estimates of future rewards when updating its policy.

[0048] After policy training is completed, each agent enters the execution phase. To maintain the system's scalability and robustness, this invention employs a centralized training, distributed execution (CTDE) mechanism: During the training phase, all agents share global information (including state) within a centralized training framework. Joint actions (and improved credit allocation results) to enhance learning efficiency; During the execution phase, each agent operates independently, relying only on its local observations. With historical trajectory According to the learned policy function Make action decisions.

[0049] This ensures that efficient training can be achieved by utilizing global information, while maintaining the feasibility and flexibility of distributed deployment in actual situations.

[0050] In this embodiment of the application, the individual contribution predicted by the neural network model in step S2 specifically uses a lightweight prediction network: Since directly calculating the above function can incur high computational overhead in large-scale systems, especially when comparing different combinations of actions, its complexity increases exponentially with the number of agents. Therefore, this invention further introduces a lightweight prediction network. Approximated by parameterization : in, For individual contribution, For intelligent agents The state representation, For state, For joint operations, It is a lightweight prediction network. This is the set of learnable parameters for the contribution prediction network. Their values ​​are derived from the training process and automatically obtained by the optimization algorithm through gradient updates. Their values ​​fall within the real number space and do not require manual setting of fixed values.

[0051] In an optional implementation, the prediction of individual contribution through a neural network model in step S2 can also employ a tree-based regression model. The system state, state representation of each agent, joint action, and true contribution label collected during the training phase are used as training samples. Gradient boosting trees or random forests are used for training, enabling the tree model to learn the mapping between contribution and input. During the online phase, the current state and joint action are input, and the tree model outputs the contribution estimate of each agent.

[0052] In another alternative implementation, the prediction of individual contribution through a neural network model in step S2 can also be based on a graph neural network, where each agent is treated as a node in the graph, and edges are established between nodes based on communication and cooperation relationships. Through the message passing structure of the graph neural network, neighborhood information is aggregated, and finally, the contribution prediction of each agent is output at the node layer.

[0053] In the embodiments of this application, in step S2, the individual contribution is predicted by a neural network model, and the marginal contribution of the agent under different states and action combinations is automatically fitted by a set of learnable parameters. While ensuring reasonable modeling of the individual contribution, the computational efficiency and feasibility of the algorithm in large-scale multi-agent systems are improved.

[0054] In this way, the present invention can significantly reduce computational complexity while ensuring the rationality of contribution modeling.

[0055] This invention achieves optimal parameter settings through a reasonable initialization strategy, supervised learning objectives, and regularization constraints. It can effectively fit the individual contribution of agents under different state and action combinations, thus ensuring the stability and efficiency of the prediction network in large-scale multi-agent systems.

[0056] In this embodiment of the application, the calculation of the return index using a first-level algorithm in step S2 is specifically manifested as follows: After obtaining the contribution estimate, this invention combines it with the global reward to define an individualized reward signal for each agent, expressed as: in, Individualized reward signals for each agent The global instant reward returned by the environment. For intelligent agents The predicted contribution This indicates the removal of the agent. In the case of [missing information], the average contribution estimate of the remaining agents is used as a baseline correction term.

[0057] Individual rewards as defined intelligent agent The optimization objective can be formalized as maximizing the expected return expression: in, For agent i, when adopting the policy function The expected long-term cumulative return Let i be the policy function of agent i. To track Expectations Discount factor Power-decay weights for time step t For the task time range, For trajectory and orbital distribution, The individualized reward for agent i at time t.

[0058] In an optional implementation, the reward allocation based on contribution ranking can also be used in step S2 to calculate the reward index through the first-level algorithm. The predicted contribution of all agents at the current time is ranked, and the reward level is divided according to the ranking position. For example, agents with high contribution are given higher rewards, and agents with low contribution are given basic rewards. Agents receive their own reward signals according to the corresponding level.

[0059] In another optional implementation, the reward index calculated by the first-level algorithm in step S2 can also be based on the reward correction of contrastive learning. During the training phase, experience pairs of positive and negative behaviors are constructed. The contrast module compares the current behavior with the historical best behavior. The higher the similarity, the higher the reward. Each agent receives a contrast score based on its own behavior as an individual reward.

[0060] In the embodiments of this application, step S2 uses a first-level algorithm to calculate the reward index, decomposing the abstract global collaborative goal into specific goals that can be directly perceived and optimized by individuals. This enables the policy updates of each agent to be closely aligned and effectively drive the improvement of overall performance, thus solving the problem of ambiguous credit allocation caused by global reward sharing in multi-agent collaboration.

[0061] It should be noted that by combining gradient-based policy learning with a centralized training and distributed execution mechanism, the agent can learn using global information and make independent decisions based solely on local observations in actual deployment, ensuring the algorithm's scalability and practicality. By constructing an optimization objective driven by individualized rewards, the agent is guided to improve its own contribution while promoting the achievement of the global objective, thus achieving stable and efficient collaborative control in complex dynamic environments.

[0062] In this embodiment of the application, step S3 involves normalizing the individual reward signal, introducing a regularization term into the global objective function, calculating the convergence index, and monitoring the convergence, including the following steps C1-C3: C1: Normalize the individual reward signal and introduce the regularization term into the global objective function.

[0063] Introducing a reward normalization operator: in, To normalize individual reward signals, To reward the normalization operator, For agent i, the individualized reward at time t. and These are the mean and standard deviation of the individual reward during the most recent training window, respectively. To prevent constants with a denominator of zero.

[0064] C2: Based on the normalized individual reward signal, a convergence index is defined and calculated, and the convergence index is monitored.

[0065] In actual training, an excessively large learning rate may lead to training oscillations, while an excessively small learning rate may result in slow convergence. Let time... The global value function is estimated as follows: The convergence index is then defined as: in, As a convergence index, For a moment The global value function under, For a moment The global value function under -1.

[0066] C3: Adjust the parameters for strategy training based on the relationship between the convergence index and the set convergence index threshold.

[0067] when Persistently below the convergence index threshold When the learning rate is high, it automatically decreases; conversely, if the learning rate is low, it automatically decreases. If the learning rate remains above the convergence metric threshold for an extended period, it should be appropriately increased to encourage escape from local optima. The convergence metric threshold is... The settings are based on experience and actual application scenarios.

[0068] In this embodiment of the application, the introduction of the regularization term into the global objective function in step C1 is specifically manifested as follows: An entropy regularization term is introduced into the policy optimization objective, with the expression: in, Let the objective function be the policy objective function after entropy regularization. Used to measure the state of agent i The randomness of the action distribution. For agent i, when adopting the policy function The expected long-term cumulative return To track Expectations Let i be the policy distribution of agent i. This is an adjustment coefficient used to adjust the weight of the policy entropy in the overall optimization objective. Its value is typically in the range [0.01, 0.2] and can be dynamically adjusted according to task complexity and training stage: a larger value is used in the early stages of training to maintain sufficient exploration, and it is gradually decreased in the later stages of training to promote policy convergence. For continuous action spaces, The typical value range is 0.001-0.02, in order to avoid the entropy term causing excessive interference to the profit optimization.

[0069] In an alternative implementation, the regularization term introduced into the global objective function in step C1 can also be a behavior constraint regularization term based on policy smoothness. During each policy update, the difference between the action distribution of the current policy and the previous policy is compared. When the difference exceeds a set threshold, an additional penalty is added to the global objective function. The updated policy is constrained to not deviate too far from the previous version.

[0070] In another alternative implementation, the regularization term introduced into the global objective function in step C1 can also be a reference policy-based offset constraint regularization term. Each agent maintains a reference policy, which can be the historical best policy or the policy at the previous moment. A constraint term that reflects the difference between the current policy and the reference policy is added to the optimization objective. The update process is restricted to a safe range, thereby avoiding abnormal behavior.

[0071] In this embodiment of the application, a regularization term is introduced into the global objective function in step C1. By taking the randomness of the strategy as part of the optimization objective and combining it with dynamically adjustable coefficients, the relationship between strategy exploration and exploitation is balanced.

[0072] In summary, by formalizing the decision problem as a partially observable Markov game, a contribution estimation method based on a lightweight prediction network is designed to map the global objective to individual rewards. Utilizing a centralized training and distributed execution framework, combined with advantage functions based on individual contributions and gradient policy optimization, the efficient learning and distributed robust execution of agent policies under collaborative objectives are ensured. Through reward normalization and entropy regularization mechanisms, the training process is stabilized and the policy exploration capability is maintained. At the same time, convergence monitoring and adaptive learning rate adjustment are introduced to improve the performance and convergence efficiency of multi-agent collaborative decision-making.

[0073] Example 3 is an embodiment of the present invention, which provides a system for agent-based collaborative intelligent decision-making in substation environmental control, including a decision problem modeling module, an individual contribution prediction module and a strategy training module, as well as a convergence monitoring module. The decision problem modeling module collects historical data from substations, builds an intelligent agent interaction environment, models the decision problem, estimates the state based on local observations and historical substation data, and constructs a global objective function. The individual contribution prediction module and the strategy training module are based on a global objective function. They predict individual contributions through a neural network model, define individual reward signals based on individual contributions, calculate reward indicators through a first-level algorithm, and train and optimize strategies based on reward indicators before executing decisions. The convergence monitoring module normalizes individual reward signals, introduces a regularization term into the global objective function, calculates the convergence index, and monitors the convergence.

[0074] This embodiment also provides an electronic device applicable to a method for intelligent agent collaborative decision-making in substation environmental control, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method for intelligent agent collaborative decision-making in substation environmental control as proposed in the above embodiment.

[0075] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements a method for intelligent agent collaborative decision-making for substation environmental control as proposed in the above embodiments.

[0076] The storage medium proposed in this embodiment and the method for implementing intelligent agent collaborative decision-making for substation environmental control proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0077] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0078] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for agent-based collaborative intelligent decision-making in substation environmental control, characterized in that: include, Collect historical data from substations, build an intelligent agent interaction environment, model the decision problem, estimate the state based on local observations and historical data from substations, and construct a global objective function; Based on the global objective function, the individual contribution is predicted by a neural network model, and the individual reward signal is defined according to the individual contribution. The return index is calculated by a first-level algorithm, and the strategy is trained and optimized and the decision is executed based on the return index. The individual reward signal is normalized, the regularization term is introduced into the global objective function, the convergence index is calculated, and the convergence is monitored.

2. The method for agent-based collaborative intelligent decision-making in substation environmental control as described in claim 1, characterized in that: The process involves collecting historical data from substations, building an intelligent agent interaction environment, modeling the decision-making problem, estimating the state based on local observations and historical substation data, and constructing a global objective function. Collect historical data from substations, build an intelligent agent interaction environment, and model the decision-making problem through random game theory; Based on the agent interaction environment, state estimation is performed according to local observations and historical data of the substation, and the state representation vector of the agent is output. A global objective function is constructed based on the agent's state representation vector.

3. The method for agent-based collaborative intelligent decision-making in substation environmental control as described in claim 2, characterized in that: The process of predicting individual contributions using a neural network model based on a global objective function, defining individual reward signals based on individual contributions, calculating reward indicators using a first-level algorithm, and training and optimizing strategies based on reward indicators before executing decisions includes... Based on the global objective function, the individual contribution is defined and predicted by a neural network model, and the individual contribution is output. Individual reward signals are defined based on individual contribution, and return indicators are calculated using a first-level algorithm. The agent trains and optimizes the strategy based on the reward metrics, and then executes the strategy.

4. The method for agent-based collaborative intelligent decision-making in substation environmental control as described in claim 3, characterized in that: The process of normalizing individual reward signals, incorporating a regularization term into the global objective function, calculating a convergence index, and monitoring convergence includes... The individual reward signal is normalized, and the regularization term is introduced into the global objective function; Based on the normalized individual reward signal, a convergence index is defined and calculated, and the convergence index is monitored. Adjust the parameters for strategy training based on the relationship between the convergence metric and the set convergence metric threshold.

5. The method for agent-based collaborative intelligent decision-making in substation environmental control as described in claim 4, characterized in that: The construction of the global objective function based on the agent's state representation vector includes, The goal of a swarm of agents is to learn a joint policy. To maximize the long-term cumulative global return, the global objective function is expressed as follows: in, The global objective function is... To track Expectations Discount factor Power-decay weights for time step t The desired time constant, For the actual time interval, For global instant rewards, For the task time range, For trajectory and orbital distribution.

6. The method for agent-based collaborative intelligent decision-making in substation environmental control as described in claim 5, characterized in that: The calculation of return metrics using a first-level algorithm includes, intelligent agent The optimization objective is formalized as maximizing the expected return, expressed as: in, For agent i, when adopting the policy function The expected long-term cumulative return Let i be the policy function of agent i. To track Expectations Discount factor Power-decay weights for time step t For the task time range, For trajectory and orbital distribution, The individualized reward for agent i at time t.

7. The method for agent-based collaborative intelligent decision-making in substation environmental control as described in claim 6, characterized in that: The normalization process for individual reward signals, including the incorporation of a regularization term into the global objective function, includes... Introducing a reward normalization operator: in, To normalize individual reward signals, To reward the normalization operator, For agent i, the individualized reward at time t. and These are the mean and standard deviation of the individual reward during the most recent training window, respectively. To prevent constants with a denominator of zero; An entropy regularization term is introduced into the policy optimization objective, with the expression: in, Let the objective function be the policy objective function after entropy regularization. The entropy of the agent's policy. For agent i, when adopting the policy function The expected long-term cumulative return To track Expectations Let i be the policy distribution of agent i. This is the adjustment coefficient.

8. A system for agent-based collaborative intelligent decision-making in substation environmental control, employing the agent-based collaborative intelligent decision-making method for substation environmental control as described in any one of claims 1 to 7, characterized in that... It includes a decision problem modeling module, an individual contribution prediction module and a strategy training module, as well as a convergence monitoring module. The decision problem modeling module collects historical data from substations, builds an intelligent agent interaction environment, models the decision problem, estimates the state based on local observations and historical data from substations, and constructs a global objective function. The individual contribution prediction module and strategy training module are based on a global objective function, use a neural network model to predict individual contributions, define individual reward signals based on individual contributions, calculate reward indicators through a first-level algorithm, and perform strategy training and optimization based on reward indicators before executing decisions. The convergence monitoring module normalizes the individual reward signal, introduces a regularization term into the global objective function, calculates the convergence index, and monitors the convergence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for intelligent agent collaborative decision-making for substation environmental control as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for intelligent agent collaborative decision-making for substation environmental control as described in any one of claims 1 to 7.

Images