Intelligent agent data interaction method, system and equipment based on reflective memory network

By adopting the reflex memory network-based agent data interaction method in the new power system, the data incompatibility and transmission lag problems are solved, the immediacy and accuracy of agent data interaction are realized, and the system efficiency and agent's Adaptability.

CN119004998APending Publication Date: 2024-11-22CHINA SOUTHERN POWER GRID COMPANY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411182163.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The prior art has data incompatibility and data transmission lag in the process of data interaction of the agent in the new power system, resulting in low system data interaction efficiency.

Method used

Using the interactive method of the agent data based on the reflective memory network, by constructing the agent and the target machine ID, the real-time node card maps the observation values ​​and action decisions of the local node and the target node, and passes the observation measurements stored in the real-time node card. , action decisions and reward values ​​train the agent to ensure the immediacy and accuracy of data exchange.

Benefits of technology

It improves the system data interaction efficiency, simplifies the interaction process, ensures the immediacy and accuracy of data exchange, and improves the adaptability and reliability of the agent.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004998A_ABST
    Figure CN119004998A_ABST
Patent Text Reader

Abstract

The invention provides an agent data interaction method, system and device based on a reflective memory network, and the method comprises the steps: generating an agent and a target machine ID based on a preset global parameter configuration file; the method comprises the following steps: reading a real-time node card of a local node to obtain a first observed quantity corresponding to the local node; a corresponding action decision is obtained by reading a real-time node card of a target node; updating a real-time node card of the local node to obtain a second observed quantity corresponding to the local node and a corresponding action reward value; performing reinforcement learning training on the intelligent agent to obtain a decision-making intelligent agent; and receiving a control signal output by the local node, and returning the control signal to the target node of the corresponding target machine ID for data interaction. According to the method, the problem of low system data interaction efficiency caused by data incompatibility and hysteresis of data transmission in a multi-platform agent data interaction process in the prior art is solved. The interaction process is simplified, and the system data interaction efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data interaction, and in particular to an intelligent agent data interaction method, system and device based on a reflective memory network. Background Art

[0002] Reinforcement learning, with its unique advantages, has demonstrated significant application potential and prospects in a wide range of scenarios in new power systems. Most current studies on the application of deep reinforcement learning in new power systems tend to use advanced digital simulation software as the interactive environment for intelligent agents to achieve efficient and safe model training and testing. Some cutting-edge research has expanded the interactive environment to rapid prototypes and even real physical experimental platforms. This move not only enriches the diversity and authenticity of the data, but also greatly enhances the effectiveness and credibility of the developed models.

[0003] At present, the reinforcement learning interactive environment constructed by pure digital simulation software, although powerful, is still difficult to completely bridge the gap with the real physical world, resulting in the trained model facing adaptability challenges in practical applications and being unable to be directly deployed; when exploring semi-physical simulation platforms and physical experimental platforms as interactive environments, researchers encountered two difficulties: first, the incompatibility of data file formats and the lag in data transmission, these technical barriers limit the widespread application of this method; second, in the face of massive and complex data interaction requirements in new power systems, the use of traditional direct end-to-end data cable connections between control units and controlled units not only increases the risk of misconnection but also makes it difficult to quickly locate problems, seriously restricting research efficiency and progress; it can be seen that the existing technology currently has data incompatibility and data transmission lags during multi-platform intelligent agent data interaction, resulting in low system data interaction efficiency. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes an intelligent agent data interaction method, system and device based on reflective memory network, which realizes the immediacy and accuracy of data exchange, simplifies the interaction process and improves the system data interaction efficiency.

[0005] To achieve the above object, an embodiment of the present invention provides an agent data interaction method based on a reflective memory network, comprising:

[0006] Based on a preset global parameter configuration file, an agent and a target machine ID are generated; wherein the target machine ID includes a number of node IDs;

[0007] Based on the agent and the target machine ID, obtaining a first observation value corresponding to the local node by reading a real-time node card of the local node;

[0008] Based on the first observation value corresponding to the local node and the target machine ID, a corresponding action decision is obtained by reading the real-time node card of the target node;

[0009] Based on the corresponding action decision and the target machine ID, the real-time node card of the local node is updated to obtain a second observation value and a corresponding action reward value corresponding to the local node;

[0010] Based on the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node, and the corresponding action reward value, the agent is subjected to reinforcement learning training until the agent meets the preset training requirements, thereby obtaining a decision-making agent;

[0011] Based on the decision-making agent, a control signal output by a local node is received, and the control signal is returned to a target node corresponding to the target machine ID for data interaction.

[0012] The embodiment of the present invention proposes an intelligent agent data interaction method based on a reflective memory network, by constructing an intelligent agent and a target machine ID, performing data mapping on the observation value of the local node and the action decision of the target node through a real-time node card, and then training the intelligent agent through the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node and the corresponding action reward value stored in the real-time node card of each node, so as to improve the adaptability and reliability of the intelligent agent, and finally deploying the trained intelligent agent to perform data interaction of each node through the target machine ID; based on the above innovative data interaction mode, the immediacy and accuracy of data exchange are ensured, while simplifying the interaction process and improving the system data interaction efficiency.

[0013] Furthermore, based on the preset global parameter configuration file, the agent and target machine ID are generated, specifically:

[0014] Based on the preset global parameter configuration file, a reinforcement learning assisted decision-making intelligent agent model is constructed to obtain an intelligent agent;

[0015] Based on the preset global parameter configuration file, unique IDs are set for the global local node and target node to obtain the target machine ID.

[0016] Further, based on the agent and the target machine ID, the first observation corresponding to the local node is obtained by reading the real-time node card of the local node, specifically:

[0017] Read the target machine ID and match it with the corresponding local node ID;

[0018] The local node ID is transmitted to the agent, and the first observation value corresponding to the local node is obtained by reading the real-time node card of the local node.

[0019] Furthermore, based on the first observation value corresponding to the local node and the target machine ID, the corresponding action decision is obtained by reading the real-time node card of the target node, specifically:

[0020] Based on the first observation value corresponding to the local node, output a corresponding action;

[0021] Based on the target machine ID, match the corresponding target node ID;

[0022] The target node ID is transmitted to the intelligent agent, and the corresponding action is updated to the real-time node card of the target node through the real-time network HUB and executed to obtain the corresponding action decision.

[0023] Further, based on the corresponding action decision and the target machine ID, the real-time node card of the local node is updated to obtain the second observation value and the corresponding action reward value corresponding to the local node, specifically:

[0024] Based on the target machine ID, read the corresponding local node ID;

[0025] Transmitting the local node ID to the agent, and recording the observation value and action reward value in real time by executing the corresponding action decision;

[0026] By updating the observation value and the action reward value to the real-time node card of the local node, a second observation value and a corresponding action reward value corresponding to the local node are obtained.

[0027] Further, based on the first observation corresponding to the local node, the corresponding action decision, the second observation corresponding to the local node and the corresponding action reward value, the agent is subjected to reinforcement learning training until the agent meets the preset training requirements to obtain a decision-making agent, specifically:

[0028] The first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node and the corresponding action reward value form a set of experience samples;

[0029] Obtaining a number of experience samples by continuously updating the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node, and the corresponding action reward value;

[0030] Based on the several experience samples, the agent is subjected to reinforcement learning training until the number of training times converges to obtain a decision-making agent.

[0031] Further, based on the real-time node card of the local node and the real-time node card of the target node, a unique target machine ID and its corresponding unique memory address are set;

[0032] Based on the target machine ID and the number of states that each node needs to observe, determine the space size of the corresponding unique memory address;

[0033] The target machine ID cannot be modified, but the data stored in the unique memory address can be modified.

[0034] An embodiment of the present invention proposes an intelligent agent data interaction method based on a reflective memory network, which ensures the consistency of the unique identification of the intelligent agent and each node by setting a unique target machine ID and limiting the ID to be unmodifiable, thereby ensuring the accuracy and reliability of data interaction.

[0035] Furthermore, it also includes:

[0036] Get the second observation value and the corresponding action reward value corresponding to the local node;

[0037] Based on the real-time network HUB, the second observation value and the corresponding action reward value corresponding to the local node are updated to the memory address corresponding to the real-time node card of the global node.

[0038] The embodiment of the present invention also provides an agent data interaction system based on a reflective memory network, comprising: a basic configuration module, a first processing module, a second processing module, a third processing module, a model training module and a data interaction module;

[0039] The basic configuration module is used to generate an agent and a target machine ID based on a preset global parameter configuration file; wherein the target machine ID includes a number of node IDs;

[0040] The first processing module is used to obtain a first observation value corresponding to the local node by reading a real-time node card of the local node based on the intelligent agent and the target machine ID;

[0041] The second processing module is used to obtain a corresponding action decision by reading a real-time node card of a target node based on the first observation value corresponding to the local node and the target machine ID;

[0042] The third processing module is used to obtain the second observation value and the corresponding action reward value corresponding to the local node by updating the real-time node card of the local node based on the corresponding action decision and the target machine ID;

[0043] The model training module is used to perform reinforcement learning training on the agent based on the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node and the corresponding action reward value until the agent meets the preset training requirements to obtain a decision-making agent;

[0044] The data interaction module is used to receive the control signal output by the local node based on the decision-making agent, and return the control signal to the corresponding target machine ID for data interaction.

[0045] The embodiment of the present invention proposes an intelligent agent data interaction system based on a reflective memory network, which constructs an intelligent agent and a target machine ID through a basic configuration module, and uses a first processing module, a second processing module and a third processing module to perform data mapping on the observation value of the local node and the action decision of the target node through a real-time node card, and then uses a model training module to train the intelligent agent through the first observation value, the corresponding action decision, the second observation value and the corresponding action reward value corresponding to the local node stored in the real-time node card of each node, so as to improve the adaptability and reliability of the intelligent agent, and finally the data interaction module deploys the trained intelligent agent and performs data interaction of each node through the target machine ID; based on the above innovative data interaction mode, the immediacy and accuracy of data exchange are ensured, while the interaction process is simplified and the system data interaction efficiency is improved.

[0046] The embodiment of the present invention also provides an intelligent agent data interaction device based on a reflective memory network, comprising: a control unit, an interaction system and a controlled unit;

[0047] The control unit generates an agent and a target machine ID by receiving the preset global parameter configuration file; wherein the agent is deployed in the control unit, the interactive system and the controlled unit;

[0048] The controlled unit receives the control signal from the control unit, executes the corresponding control action and updates the real-time node card corresponding to each node;

[0049] The interactive system receives a control signal from a control unit and transmits it to a corresponding controlled unit, and receives a signal returned by the controlled unit to update the real-time node card memory address data of each node.

[0050] The embodiment of the present invention proposes an intelligent data interaction device based on a reflective memory network. Through a data interaction mode composed of a control unit, an interaction system and a controlled unit, the control unit generates an intelligent agent and a target machine ID, and then outputs a control signal to be sent to the corresponding controlled unit via the interaction system. The real-time node card memory address data of each node is then updated by the controlled unit, thereby simplifying the data interaction process and updating the real-time node card memory address data of each node according to the control signal, thereby realizing a mapping mechanism for data interaction and reducing the lag of data transmission. Therefore, based on the above-mentioned innovative data interaction mode, the immediacy and accuracy of data exchange are ensured, while the interaction process is simplified and the system data interaction efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the steps of a method for interacting with agent data based on a reflective memory network provided by an embodiment of the present invention;

[0052] Figure 2 A diagram showing an application example of an agent data interaction method based on a reflective memory network provided by a certain embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the module structure of an agent data interaction system based on a reflective memory network provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of the module structure of an intelligent agent data interaction device based on a reflective memory network provided in one embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the working steps of an intelligent agent data interaction device based on a reflective memory network is provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Reinforcement learning, with its unique advantages, has shown significant application potential and prospects in a wide range of scenarios in new power systems, attracting a large number of researchers to explore its application strategies in this field. In view of the outstanding performance of reinforcement learning in controllability, safety and cost-effectiveness, most current studies on the application of deep reinforcement learning in new power systems tend to use advanced digital simulation software as the interactive environment of intelligent agents to achieve efficient and safe model training and testing. In order to solve the problem of data incompatibility and data transmission lag in the process of multi-platform intelligent agent data interaction in the prior art, resulting in low system data interaction efficiency, the present invention proposes an intelligent agent data interaction method and system integration equipment based on reflective memory network.

[0058] Example 1

[0059] See also Figure 2 , Figure 2 FIG. 1 is an application example diagram of an agent data interaction method based on a reflective memory network provided by a certain embodiment of the present invention; Figure 2As shown, a multi-platform collaborative simulation system of a new power system constructed by an application example of this embodiment includes: a host computer system 201, a semi-physical simulation platform 202, a physical experiment platform 203 and a distributed interactive system 204, wherein the host computer system 201 is used to control the semi-physical simulation platform 202 and the physical experiment platform 203, and specifically includes two parts: a computer and a hardware-in-the-loop device, wherein the computer is used to construct and train deep reinforcement learning and multi-agent deep reinforcement learning algorithms, and the hardware-in-the-loop device is used to deploy deep reinforcement learning and multi-agent deep reinforcement learning algorithms; the semi-physical simulation platform 202 and the physical experiment platform 203 are collectively referred to as simulation platforms, wherein the semi-physical simulation platform 202 is various types of rapid prototyping machines; the physical experiment platform 203 is various types of renewable energy generators contained in the new power system, such as photovoltaic generators, wind turbines, etc.; the distributed interactive system 204 is a distributed real-time interactive system constructed using a reflective memory network card (real-time node card). The distributed interactive system 204 adopts a star structure as a system connection mode, which is composed of a real-time network HUB and multiple real-time node cards. Various devices in the network are connected to each other using optical fibers.

[0060] In this embodiment, an agent data interaction method based on a reflective memory network is applied to the multi-platform collaborative simulation system of the novel power system for explanation, which will not be described in detail below.

[0061] See also Figure 1 , Figure 1 A schematic flow chart of a method for interacting with agent data based on a reflective memory network is provided in accordance with an embodiment of the present invention. Figure 1 As shown, the embodiment of the present invention proposes an agent data interaction method based on a reflective memory network, including steps 101 to 106, each of which is specifically as follows:

[0062] Step 101, based on a preset global parameter configuration file, generating an agent and a target machine ID; wherein the target machine ID includes a number of node IDs;

[0063] As an example of this embodiment, based on the preset global parameter configuration file, a reinforcement learning assisted decision-making agent model is constructed to obtain an agent; based on the preset global parameter configuration file, a unique ID is set for the global local node and the target node to obtain the target machine ID. A specific explanation is that in the host computer system, the global parameter configuration file is imported, the deep reinforcement learning assisted decision-making agent model is constructed and deployed, and the ID of each agent and the target machine is generated; according to the target machine ID and the parameter configuration file, a renewable power generation unit such as a photovoltaic power generation unit, a wind power generation unit, an energy storage system, etc. is constructed in the semi-physical simulation platform; according to the target machine ID and the parameter configuration file, a node model that meets the research object of the actual new power system is constructed in the semi-physical simulation platform, as well as other power generation units that are not built in the physical experiment platform. In a specific embodiment, the target machine ID is determined according to the type and number of nodes in the new power system. For example, the host computer system uses ID000, the first real-time simulation machine in the semi-physical simulation platform 202 uses ID100, and the first power generation unit in the physical experiment platform 203 uses ID200, and so on to obtain the global target machine ID. The global parameter configuration file is a data structure that records all the information of the experimental scene, including the types and numbers of intelligent agents participating in the simulation, the simulation information in the simulation platform, the model topology, the types and numbers of power generation units, and the assignment of corresponding parameters; the target machine ID is determined according to the type and number of nodes in the new power system; the simulation information includes system time, simulation step, sampling period and communication period, wherein the system time refers to the time from the start of the simulation to the current moment, and in order to reduce the time delay of each unit, the central node of the communication optical fiber is used as the flag of the system time; the simulation step is the calculation period of the semi-physical simulation platform; the sampling period is the time period for the host computer system 201 or the semi-physical simulation platform 202 and the physical experiment platform 203 to write or read data from the real-time node card; the communication period is the time period for the reflective memory network to synchronize information. In general, the simulation step should be an integer multiple of the communication step, and the sampling period should be an integer multiple of the simulation step.

[0064] Step 102, based on the agent and the target machine ID, obtain a first observation value corresponding to the local node by reading the real-time node card of the local node;

[0065] As an example of this embodiment, the target machine ID is read and matched with the corresponding local node ID; the local node ID is transmitted to the agent, and the first observation corresponding to the local node is obtained by reading the real-time node card of the local node. A specific implementation method is that according to the target machine ID, the reinforcement learning agent in the host system reads the corresponding observation from the real-time node card of the local node.

[0066] Step 103, based on the first observation value corresponding to the local node and the target machine ID, obtain the corresponding action decision by reading the real-time node card of the target node;

[0067] As an example of this embodiment, based on the first observation quantity corresponding to the local node, the corresponding action is output; based on the target machine ID, the corresponding target node ID is matched; the target node ID is transmitted to the intelligent agent, and the corresponding action is updated to the real-time node card of the target node through the real-time network HUB and executed to obtain the corresponding action decision. A specific implementation method is based on the target mapping mechanism. After reading and obtaining the corresponding observation quantity, the corresponding action is output and input into the real-time node card of the target node; through the real-time network HUB, the real-time node updates the action decision to the designated real-time node card. The target mapping mechanism adopts different data interaction methods according to the data flow situation. The first data interaction method is to send from the output node to the receiving node without passing through the memory space of the real-time node card, which becomes a direct transmission method. The second data interaction method is to map the memory space of the local real-time node card to the memory space of the global node card, which becomes a memory reflection method. Specifically, the first category is the data type with a clear data flow relationship or fewer data receiving ends (such as one-to-one, one-to-two, etc.). This type of data is mainly the control signal sent from the host computer system 201 to the semi-physical simulation platform 202 and the physical experiment platform 203. These data will be sent directly from the output node to the receiving node without passing through the memory space of the real-time node card, so the address space will not be allocated; the second category is the data type with unclear data flow or more data receiving ends. This type of data is mainly the state quantity of the semi-physical simulation platform 202 and the physical experiment platform 203. These data will be reflected to the memory space of the global node card through the memory space of the local real-time node card to complete the real-time transmission of data. Therefore, it is also necessary to determine the space size of the memory address according to the target machine ID and the number of states that the node unit needs to observe.

[0068] Step 104, based on the corresponding action decision and the target machine ID, by updating the real-time node card of the local node, obtaining a second observation value and a corresponding action reward value corresponding to the local node;

[0069] As an example of this embodiment, based on the target machine ID, the corresponding local node ID is read; the local node ID is transmitted to the intelligent agent, and the observation amount and action reward value are recorded in real time by executing the corresponding action decision; by updating the observation amount and action reward value to the real-time node card of the local node, the second observation amount and the corresponding action reward value corresponding to the local node are obtained. A specific implementation method is that according to the target machine ID, each power generation unit of the semi-physical simulation platform 202 and the physical experiment platform 203 reads the corresponding action decision from the local real-time node card and executes it, and records the new local observation amount and the corresponding action reward value and inputs it into the local real-time node card, and then updates the new local observation amount and reward value to all global real-time node cards.

[0070] Step 105, based on the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node, and the corresponding action reward value, the agent is subjected to reinforcement learning training until the agent meets the preset training requirements, thereby obtaining a decision-making agent;

[0071] As an example of this embodiment, the first observation corresponding to the local node, the corresponding action decision, the second observation corresponding to the local node and the corresponding action reward value are formed into a group of experience samples; by continuously updating the first observation corresponding to the local node, the corresponding action decision, the second observation corresponding to the local node and the corresponding action reward value, a number of experience samples are obtained; based on the several experience samples, the agent is subjected to reinforcement learning training until the number of training times converges to obtain a decision agent. A specific implementation method is that the experience library in the host computer system 201 reads the new local observation and reward value from the real-time node card, and then the computer in the host computer system 201 forms a group of experience samples with the local observation, action decision, reward value and new local observation and saves them in the experience library. According to the experience saved in the experience library, deep reinforcement learning performs a reinforcement learning training in the computer, and the decision agent obtained by training will be deployed in the hardware-in-the-loop device. Among them, by repeating steps 102 and 104 until the training is completed. This needs to be designed according to different task scenarios, generally determined according to the convergence degree of the model or whether the number of training times converges.

[0072] As another example of this embodiment, based on the target machine ID and the number of states that each node needs to observe, determine the space size of the corresponding unique memory address; wherein, the target machine ID cannot be modified, and the data stored in the unique memory address can be modified. Obtain the second observation value and the corresponding action reward value corresponding to the local node; based on the real-time network HUB, update the second observation value and the corresponding action reward value corresponding to the local node to the memory address corresponding to the real-time node card of the global node. Specifically, set a unique machine ID and a corresponding unique memory address for the real-time node card equipped with the host computer system and each simulation platform. The corresponding device of each node can read the data of any machine ID in the memory address of the local real-time node card, but can only modify the data of the memory address corresponding to its own machine ID in the local real-time node card to ensure the consistency of the uniqueness identification of the intelligent agent and the actual physical device.

[0073] Step 106: Based on the decision-making agent, a control signal output by a local node is received, and the control signal is returned to the target node of the corresponding target machine ID for data interaction.

[0074] As an example of this embodiment, the trained reinforcement learning decision-making agent is deployed in the hardware-in-the-loop device of the host computer system 201, and a control signal is output according to the local state transmitted by the distributed interactive system 204, and the signal is returned to the semi-physical simulation platform 202 and the physical experiment platform 203 corresponding to the target machine ID, so as to realize data interaction between renewable energy units such as photovoltaic power generation units and wind power generation units of the new power system, and optimize control can also be performed.

[0075] The embodiment of the present invention proposes an intelligent agent data interaction method based on a reflective memory network, which constructs an intelligent agent and a target machine ID, performs data mapping on the observation value of the local node and the action decision of the target node through a real-time node card, and then trains the intelligent agent through the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node and the corresponding action reward value stored in the real-time node card of each node to improve the adaptability and reliability of the intelligent agent, and finally deploys the trained intelligent agent to perform data interaction of each node through the target machine ID; based on the above-mentioned innovative data interaction mode, the immediacy and accuracy of data exchange are ensured, and the interaction process is simplified, thereby improving the system data interaction efficiency; by setting a unique target machine ID and limiting the ID to be unmodifiable, the consistency of the unique identification of the intelligent agent and each node is ensured, and the accuracy and reliability of data interaction are guaranteed.

[0076] Example 2

[0077] See also Figure 3 , Figure 3A schematic diagram of the module structure of an agent data interaction system based on a reflective memory network is provided in one embodiment of the present invention. Figure 3 As shown, the embodiment of the present invention proposes an agent data interaction system based on a reflective memory network, including: a basic configuration module 301, a first processing module 302, a second processing module 303, a third processing module 304, a model training module 305 and a data interaction module 306;

[0078] The basic configuration module 301 is used to generate an agent and a target machine ID based on a preset global parameter configuration file; wherein the target machine ID includes a number of node IDs;

[0079] The first processing module 302 is used to obtain a first observation value corresponding to the local node by reading the real-time node card of the local node based on the intelligent agent and the target machine ID;

[0080] The second processing module 303 is used to obtain a corresponding action decision by reading a real-time node card of a target node based on the first observation value corresponding to the local node and the target machine ID;

[0081] The third processing module 304 is used to obtain the second observation value and the corresponding action reward value corresponding to the local node by updating the real-time node card of the local node based on the corresponding action decision and the target machine ID;

[0082] The model training module 305 is used to perform reinforcement learning training on the agent based on the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node and the corresponding action reward value until the agent meets the preset training requirements to obtain a decision-making agent;

[0083] The data interaction module 306 is used to receive the control signal output by the local node based on the decision-making agent, and return the control signal to the corresponding target machine ID for data interaction.

[0084] The embodiment of the present invention proposes an intelligent agent data interaction system based on a reflective memory network, which constructs an intelligent agent and a target machine ID through a basic configuration module, and uses a first processing module, a second processing module and a third processing module to perform data mapping on the observation value of the local node and the action decision of the target node through a real-time node card, and then uses a model training module to train the intelligent agent through the first observation value, the corresponding action decision, the second observation value and the corresponding action reward value corresponding to the local node stored in the real-time node card of each node, so as to improve the adaptability and reliability of the intelligent agent, and finally the data interaction module deploys the trained intelligent agent and performs data interaction of each node through the target machine ID; based on the above innovative data interaction mode, the immediacy and accuracy of data exchange are ensured, while the interaction process is simplified and the system data interaction efficiency is improved.

[0085] Example 3

[0086] See also Figure 4 , Figure 4 A schematic diagram of the module structure of an intelligent agent data interaction device based on a reflective memory network provided by an embodiment of the present invention; Figure 4 As shown, an embodiment of the present invention proposes an intelligent agent data interaction device based on a reflective memory network, including: a control unit, an interactive system and a controlled unit; the control unit generates an intelligent agent and a target machine ID by receiving the preset global parameter configuration file; wherein the intelligent agent is deployed in the control unit, the interactive system and the controlled unit; the controlled unit executes corresponding control actions and updates the real-time node card corresponding to each node by receiving the control signal of the control unit; the interactive system receives the control signal of the control unit and transmits it to the corresponding controlled unit, and receives the signal returned by the controlled unit to update the memory address data of the real-time node card of each node. Specifically, the control unit is equivalent to the host computer system 201, the interactive system is equivalent to the semi-physical simulation platform 202 and the physical experiment platform 203, and the controlled unit is equivalent to the distributed interactive system 204. In this embodiment, based on Figure 2 The application example diagram of an intelligent agent data interaction method based on reflective memory network is explained.

[0087] As an example of this embodiment, see Figure 5 , Figure 5 A schematic diagram of the working steps of an intelligent agent data interaction device based on a reflective memory network is provided for one embodiment of the present invention. Figure 5 As shown, Figure 5The working principle of an intelligent agent data interaction device based on reflective memory network is demonstrated, and a specific explanation is given to build a multi-platform collaborative simulation system for a new power system, including a host computer system, a semi-physical simulation platform, a physical experiment platform, and a distributed interactive system; the basic information such as the system configuration of the research target, the network topology of the power system, and the requirements for various data flows are clarified, and a global parameter configuration file is generated. In the host computer system, the global parameter configuration file is imported, and a deep reinforcement learning-assisted decision-making intelligent agent model is built in the computer, and a unique representation ID for each intelligent agent and the target machine is generated, and the model is deployed in the hardware-in-the-loop device; according to the target machine ID and parameter configuration file, a renewable power generation unit such as a photovoltaic power generation unit, a wind power generation unit, and an energy storage system are built in the physical experiment platform; according to the target machine ID and parameter configuration file, a node model that meets the actual research object of the new power system is built in the semi-physical simulation platform, and a power generation unit model that cannot be built in the physical experiment platform due to various factors (such as cost factors, environmental factors, etc.) is established;

[0088] After the multi-platform collaborative simulation system of the new power system is constructed and each unit is instantiated, the configuration of the distributed real-time interactive system is performed, specifically: based on the target mapping mechanism, a distributed real-time interactive system is constructed. According to the global parameter configuration file, the data that needs to be observed in the multi-platform collaborative simulation system of the new power system is divided into two categories. The first category is the data type with a clear data flow relationship or fewer data receiving ends (such as one-to-one, one-to-two, etc.). This type of data is mainly the control signal sent from the control unit to the controlled unit. These data will be sent directly from the output node to the receiving node without passing through the memory space of the real-time node card, so the address space will not be allocated; the second category is the data type with unclear data flow or more data receiving ends. This type of data is mainly the state quantity of the controlled unit. These data will be reflected to the memory space of the global node card through the memory space of the local real-time node card to complete the real-time transmission of data. Finally, the space size of the memory address is determined according to the target machine ID and the number of states that the node unit needs to observe.

[0089] When the distributed real-time interactive system is configured, the agent training is carried out, specifically: according to the target machine ID, the reinforcement learning agent in the host system 201 (control unit) reads the corresponding observation amount from the specific memory address in the local real-time node card, and then the agent outputs the corresponding action and inputs it into the local real-time node card. Finally, through the real-time network HUB, the local real-time node card directly sends the action decision to the real-time node card of the target machine (controlled unit). The target machine reads the corresponding action decision from the real-time node card and executes it, and records the new local observation amount and the corresponding action reward value and inputs it into the specified memory address of the local real-time node card, and then reflects the new local observation amount and reward value to the specified memory address of all global real-time node cards through the real-time network HUB. The control unit reads the new local observation amount and reward value from the real-time node card and saves it in the experience library, and then the local observation amount, action decision, reward value and new local observation amount are combined into a group of experience samples and saved; according to the experience saved in the experience library, deep reinforcement learning performs a reinforcement learning training in the computer, and the trained agent will be deployed in the hardware-in-the-loop device.

[0090] The embodiment of the present invention proposes an intelligent agent data interaction device based on a reflective memory network. Through a data interaction mode composed of a control unit, an interactive system and a controlled unit, the control unit generates an intelligent agent and a target machine ID, and then outputs a control signal to send it to the corresponding controlled unit via the interactive system, and then the controlled unit updates the real-time node card memory address data of each node, thereby simplifying the data interaction process and updating the real-time node card memory address data of each node according to the control signal, realizing the mapping mechanism of data interaction and reducing the lag of data transmission; therefore, based on the above-mentioned innovative data interaction mode, the immediacy and accuracy of data exchange are ensured, while simplifying the interaction process and improving the system data interaction efficiency; the present invention also designs a distributed interaction device suitable for reinforcement learning intelligent agents and various power system simulation platforms, supporting the interaction requirements of multiple data and strong real-time with a distributed concept. Secondly, based on the target mapping mechanism, the requirements for memory size are reduced while solving the problem of inconsistent uniqueness marks between intelligent agents and actual physical devices. Finally, the data transmission mechanism based on the reflective memory network realizes the problem of multi-source data transmission.

[0091] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0092] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0093] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

Claims

1. An agent data interaction method based on reflective memory network, characterized in that: include: Based on a preset global parameter configuration file, an agent and a target machine ID are generated; wherein the target machine ID includes a number of node IDs; Based on the agent and the target machine ID, obtaining a first observation value corresponding to the local node by reading a real-time node card of the local node; Based on the first observation value corresponding to the local node and the target machine ID, a corresponding action decision is obtained by reading the real-time node card of the target node; Based on the corresponding action decision and the target machine ID, the real-time node card of the local node is updated to obtain a second observation value and a corresponding action reward value corresponding to the local node; Based on the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node, and the corresponding action reward value, the agent is subjected to reinforcement learning training until the agent meets the preset training requirements, thereby obtaining a decision-making agent; Based on the decision-making agent, a control signal output by a local node is received, and the control signal is returned to a target node corresponding to the target machine ID for data interaction.

2. The agent data interaction method based on reflective memory network as claimed in claim 1, characterized in that: The agent and target machine ID are generated based on the preset global parameter configuration file, specifically: Based on the preset global parameter configuration file, a reinforcement learning assisted decision-making intelligent agent model is constructed to obtain an intelligent agent; Based on the preset global parameter configuration file, unique IDs are set for the global local node and target node to obtain the target machine ID.

3. The agent data interaction method based on reflective memory network as claimed in claim 1, characterized in that: Based on the agent and the target machine ID, the first observation corresponding to the local node is obtained by reading the real-time node card of the local node, specifically: Read the target machine ID and match it with the corresponding local node ID; The local node ID is transmitted to the agent, and the first observation value corresponding to the local node is obtained by reading the real-time node card of the local node.

4. The agent data interaction method based on reflective memory network according to claim 1, characterized in that: Based on the first observation value corresponding to the local node and the target machine ID, the corresponding action decision is obtained by reading the real-time node card of the target node, specifically: Based on the first observation value corresponding to the local node, output a corresponding action; Based on the target machine ID, match the corresponding target node ID; The target node ID is transmitted to the intelligent agent, and the corresponding action is updated to the real-time node card of the target node through the real-time network HUB and executed to obtain the corresponding action decision.

5. The agent data interaction method based on reflective memory network according to claim 1, characterized in that: Based on the corresponding action decision and the target machine ID, the second observation value and the corresponding action reward value corresponding to the local node are obtained by updating the real-time node card of the local node, specifically: Based on the target machine ID, read the corresponding local node ID; Transmitting the local node ID to the agent, and recording the observation value and action reward value in real time by executing the corresponding action decision; By updating the observation value and the action reward value to the real-time node card of the local node, a second observation value and a corresponding action reward value corresponding to the local node are obtained.

6. The agent data interaction method based on reflective memory network according to claim 1, characterized in that: Based on the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node, and the corresponding action reward value, the agent is subjected to reinforcement learning training until the agent meets the preset training requirements to obtain a decision-making agent, specifically: The first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node and the corresponding action reward value form a set of experience samples; Obtaining a number of experience samples by continuously updating the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node, and the corresponding action reward value; Based on the several experience samples, the agent is subjected to reinforcement learning training until the number of training times converges to obtain a decision-making agent.

7. The agent data interaction method based on reflective memory network according to any one of claims 1 to 6, characterized in that: Based on the real-time node card of the local node and the real-time node card of the target node, setting a unique target machine ID and its corresponding unique memory address; Based on the target machine ID and the number of states that each node needs to observe, determine the space size of the corresponding unique memory address; The target machine ID cannot be modified, but the data stored in the unique memory address can be modified.

8. The agent data interaction method based on reflective memory network as claimed in claim 7, characterized in that: Also includes: Get the second observation value and the corresponding action reward value corresponding to the local node; Based on the real-time network HUB, the second observation value and the corresponding action reward value corresponding to the local node are updated to the memory address corresponding to the real-time node card of the global node.

9. An agent data interaction system based on reflective memory network, characterized in that: An agent data interaction method based on a reflective memory network as described in any one of claims 1 to 8, comprising: Basic configuration module, first processing module, second processing module, third processing module, model training module and data interaction module; The basic configuration module is used to generate an agent and a target machine ID based on a preset global parameter configuration file; wherein the target machine ID includes a number of node IDs; The first processing module is used to obtain a first observation value corresponding to the local node by reading a real-time node card of the local node based on the intelligent agent and the target machine ID; The second processing module is used to obtain a corresponding action decision by reading a real-time node card of a target node based on the first observation value corresponding to the local node and the target machine ID; The third processing module is used to obtain the second observation value and the corresponding action reward value corresponding to the local node by updating the real-time node card of the local node based on the corresponding action decision and the target machine ID; The model training module is used to perform reinforcement learning training on the agent based on the first observation value corresponding to the local node, the corresponding action decision, the second observation value corresponding to the local node and the corresponding action reward value until the agent meets the preset training requirements to obtain a decision-making agent; The data interaction module is used to receive the control signal output by the local node based on the decision-making agent, and return the control signal to the target node of the corresponding target machine ID for data interaction.

10. An intelligent agent data interaction device based on a reflective memory network, characterized in that: An agent data interaction method based on a reflective memory network as described in any one of claims 1 to 8, comprising: Control unit, interactive system and controlled unit; The control unit generates an agent and a target machine ID by receiving the preset global parameter configuration file; wherein the agent is deployed in the control unit, the interactive system and the controlled unit; The controlled unit receives the control signal from the control unit, executes the corresponding control action and updates the real-time node card corresponding to each node; The interactive system receives a control signal from a control unit and transmits it to a corresponding controlled unit, and receives a signal returned by the controlled unit to update the real-time node card memory address data of each node.