Control system radio resource allocation method and related equipment

Abstract

The invention provides a control system radio resource allocation method and related equipment. The method comprises the following steps: constructing a tree diagram according to a topological graph of a power distribution network; constructing a control model according to the tree diagram; all nodes in the control model communicate with one another through a wireless network; the control model is established in a network attack environment; and performing distributed resource regulation and control on the control model based on a first strategy generated by a pre-trained multi-agent. According to the embodiment of the invention, a control model is designed, a plurality of terminal nodes are covered, end-to-end information transmission is carried out through wireless communication, a resource optimization and voltage control coordinated regulation and control scheme is constructed, denial of service attacks are monitored and coped in real time, stable operation of a network is ensured, and a solution algorithm of multi-agent reinforcement learning is utilized, so that the network performance is improved. Therefore, each agent can perform cross-domain collaborative optimization in a distributed mode by means of accurate environmental information acquisition during decision making, and the control stability is guaranteed while the information transmission reliability is realized.

Description

Technical Field

[0001] The present application relates to the field of power resource technology, and in particular to a control system wireless resource allocation method and related equipment. Background Art

[0002] New power systems are developing in a larger and more complex direction, with large-scale access to distributed energy sources such as "source-grid-load-storage". Control services are also evolving from traditional centralized, master-slave control modes to ubiquitous dispatching and control. The power Internet of Things faces the challenge of accessing massive amounts of heterogeneous terminals and the pressure of explosive data growth. Energy and information flows between subsystems are becoming increasingly frequent. Dispatching targets cover all aspects of source, grid, load, and storage. The power supply, load, and power flow all have strong uncertainties, especially for the numerous and dispersed distributed renewable energy sources. For important production control services such as distributed photovoltaic regulation and new load control, uncertainties such as interaction delay and link dynamics seriously restrict control performance, requiring highly reliable transmission capabilities.

[0003] Furthermore, currently, existing technologies primarily focus on improving controller design or control strategies, without considering issues like actual communication network attacks or the unreliability of actual wireless links. Furthermore, most existing technologies focus solely on ensuring voltage stability or achieving optimal resource allocation, without comprehensively considering both communication domain resource allocation and control domain voltage regulation. In power system access scenarios involving a high proportion of renewable energy and highly randomized power control services, maximizing access security and reliability on a large scale while also ensuring control stability and ensuring real-time control and stable operation of the power system are crucial aspects of power wireless network control system design. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a control system wireless resource allocation method and related equipment.

[0005] Based on the above objectives, the present application provides a control system wireless resource allocation method, including:

[0006] Construct a tree diagram based on the topology of the power distribution network;

[0007] A control model is constructed according to the tree diagram; all nodes in the control model communicate with each other using a wireless network; and the control model is established under a network attack environment;

[0008] The control model is subjected to distributed resource regulation based on a first strategy generated by a pre-trained multi-agent.

[0009] In a possible implementation, the tree diagram is represented by the following formula:

[0010]

[0011] Among them, V represents the node set and E represents the edge set.

[0012] In a possible implementation, the tree diagram includes at least one region;

[0013] The step of constructing a control model according to the tree diagram includes:

[0014] A base station is set up in each of the areas; the area includes at least one terminal node and at least one solar photovoltaic node; the base station is connected to each of the solar photovoltaic nodes;

[0015] placing a sensor at each of the terminal nodes;

[0016] Setting a controller at each of the solar photovoltaic nodes;

[0017] A wireless communication connection is established between each of the terminal nodes and each of the solar photovoltaic nodes.

[0018] In a possible implementation, the network attack environment includes a denial of service attack environment;

[0019] The denial of service attack environment is constructed by the following method:

[0020] Determining the frequency and duration of the denial-of-service attack;

[0021] determining a signal-to-noise ratio and a packet loss rate received by the controller;

[0022] The denial-of-service attack environment is constructed based on the attack frequency, the attack duration, the signal-to-noise ratio, and the packet loss rate.

[0023] In a possible implementation, determining the attack frequency and attack duration of the denial of service attack includes:

[0024] Determining upper and lower limits on the duration of the denial of service attack;

[0025] Determining upper and lower bounds on the duration of time during which the denial of service attack does not occur;

[0026] Determine the attack frequency of the denial of service attack according to the lower limit of the duration of occurrence and the lower limit of the duration of non-occurrence;

[0027] The attack duration of the denial of service attack is determined according to the lower limit of the duration of occurrence, the lower limit of the duration of non-occurrence, and the upper limit of the duration of non-occurrence.

[0028] In a possible implementation, the signal-to-noise ratio is expressed by the following formula:

[0029]

[0030] Among them, γ m,n represents the signal-to-noise ratio received by the controller at the mth sensor at time point n, h m,n represents the small-scale fading of the mth sensor at time point n, g m represents the path loss, p m,n represents the transmission power spectral density assigned to the mth sensor at time point n, and N0 represents the one-sided noise spectral density.

[0031] In one possible implementation,

[0032] The first strategy generated based on the pre-trained multi-agent performs distributed resource regulation on the control model, including:

[0033] Determine the reward function;

[0034] Setting a second strategy of the control model; the second strategy is a random strategy;

[0035] performing gradient iteration on the second strategy with the goal of maximizing the reward function; obtaining the first strategy in response to completion of the gradient iteration;

[0036] Distributed resource regulation is performed on the control model according to the first strategy.

[0037] Based on the same inventive concept, an embodiment of the present application further provides a control system wireless resource allocation device, comprising:

[0038] A first building module is configured to build a tree diagram according to a topology diagram of a power distribution network;

[0039] A second building module is configured to build a control model based on the tree diagram; all nodes in the control model communicate with each other using a wireless network; and the control model is built under a network attack environment;

[0040] An allocation module is configured to perform distributed resource regulation on the control model based on a first strategy generated by a pre-trained multi-agent.

[0041] In a possible implementation, the tree diagram is represented by the following formula:

[0042]

[0043] Among them, V represents the node set and E represents the edge set.

[0044] In a possible implementation, the tree diagram includes at least one region;

[0045] The second building block is further configured to:

[0046] A base station is set up in each of the areas; the area includes at least one terminal node and at least one solar photovoltaic node; the base station is connected to each of the solar photovoltaic nodes;

[0047] placing a sensor at each of the terminal nodes;

[0048] Setting a controller at each of the solar photovoltaic nodes;

[0049] A wireless communication connection is established between each of the terminal nodes and each of the solar photovoltaic nodes.

[0050] In a possible implementation, the network attack environment includes a denial of service attack environment;

[0051] The device further comprises:

[0052] A first determining module is configured to determine the attack frequency and attack duration of the denial of service attack;

[0053] a second determining module, configured to determine a signal-to-noise ratio and a packet loss rate received by the controller;

[0054] The third building module is configured to build the denial of service attack environment based on the attack frequency, the attack duration, the signal-to-noise ratio, and the packet loss rate.

[0055] In a possible implementation, the first determining module is further configured to:

[0056] Determining upper and lower limits on the duration of the denial of service attack;

[0057] Determining upper and lower bounds on the duration of time during which the denial of service attack does not occur;

[0058] Determine the attack frequency of the denial of service attack according to the lower limit of the duration of occurrence and the lower limit of the duration of non-occurrence;

[0059] The attack duration of the denial of service attack is determined according to the lower limit of the duration of occurrence, the lower limit of the duration of non-occurrence, and the upper limit of the duration of non-occurrence.

[0060] In a possible implementation, the signal-to-noise ratio is expressed by the following formula:

[0061]

[0062] Among them, γ m,n represents the signal-to-noise ratio received by the controller at the mth sensor at time point n, h m,n represents the small-scale fading of the mth sensor at time point n, g m represents the path loss, p m,n represents the transmission power spectral density assigned to the mth sensor at time point n, and N0 represents the one-sided noise spectral density.

[0063] In a possible implementation, the allocation module is further configured to:

[0064] Determine the reward function;

[0065] Setting a second strategy of the control model; the second strategy is a random strategy;

[0066] performing gradient iteration on the second strategy with the goal of maximizing the reward function; obtaining the first strategy in response to completion of the gradient iteration;

[0067] Distributed resource regulation is performed on the control model according to the first strategy.

[0068] Based on the same inventive concept, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, a control system wireless resource allocation method as described in any one of the above items is implemented.

[0069] Based on the same inventive concept, an embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute any of the above-mentioned control system wireless resource allocation methods.

[0070] Based on the same inventive concept, an embodiment of the present application further provides a computer program product, which includes computer program instructions, and the computer instructions are used to enable the computer program product to execute any of the above-mentioned control system wireless resource allocation methods.

[0071] From the above, it can be seen that the control system wireless resource allocation method and related equipment provided by the present application construct a tree diagram according to the topology diagram of the power distribution network; construct a control model according to the tree diagram; all nodes in the control model communicate using a wireless network; the control model is established under a network attack environment; and the control model is distributedly regulated by a first strategy generated by a pre-trained multi-agent. The embodiment of the present application designs a 5G wireless network coverage and control model, covering multiple terminal nodes, each node is equipped with a sensor and a controller, and end-to-end information transmission is performed through wireless communication to ensure coordination and cooperation between nodes. A cross-domain multi-objective optimization problem is established based on the voltage control of the 5G wireless network distribution network, and a collaborative control scheme for resource optimization and voltage control is constructed. By introducing binary variables and control strategies, real-time monitoring and response to Denial of Service (DoS) attacks are carried out to ensure the stable operation of the network, and the duration range of the occurrence and non-occurrence of DoS attacks is set, and the optimization model is used to limit the impact of packet loss rate. A solution algorithm based on multi-agent reinforcement learning was proposed, which enables each agent to rely on accurate environmental information collection when making decisions, perform cross-domain collaborative optimization in a distributed manner, and achieve information transmission reliability while ensuring control stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0073] Figure 1 This is a flow chart of a method for allocating wireless resources in a control system according to an embodiment of the present application;

[0074] Figure 2 A schematic diagram of the power distribution network topology according to an embodiment of the present application;

[0075] Figure 3 This is a schematic diagram of a control model according to an embodiment of the present application;

[0076] Figure 4 This is a structural diagram of a wireless resource allocation device for a control system according to an embodiment of the present application;

[0077] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0079] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0080] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0081] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0082] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0083] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0084] As mentioned in the background technology section, the new power system is developing in the direction of large-scale and complex, with large-scale access to distributed energy such as "source-grid-load-storage", and the control business is also developing from the traditional centralized, master-slave control mode to ubiquitous dispatching control. The power Internet of Things faces the challenge of massive heterogeneous terminal access and the pressure of explosive growth of data. The flow of energy and information between subsystems is becoming more frequent. The dispatching objects cover all aspects of source, grid, load and storage. The power supply end, load end and power flow all have strong uncertainty, especially the large number of distributed new energy sources with scattered locations. For important production control services such as distributed photovoltaic regulation and new load control, uncertainties such as interaction delay and link dynamics seriously restrict the control performance, and highly reliable transmission capabilities are required.

[0085] Furthermore, currently, existing technologies primarily focus on improving controller design or control strategies, without considering issues like actual communication network attacks or the unreliability of actual wireless links. Furthermore, most existing technologies focus solely on ensuring voltage stability or achieving optimal resource allocation, without comprehensively considering both communication domain resource allocation and control domain voltage regulation. In power system access scenarios involving a high proportion of renewable energy and highly randomized power control services, maximizing access security and reliability on a large scale while also ensuring control stability and ensuring real-time control and stable operation of the power system are crucial aspects of power wireless network control system design.

[0086] Taking the above into consideration, the embodiment of the present application proposes a control system wireless resource allocation method, which constructs a tree diagram according to the topology diagram of the power distribution network; constructs a control model according to the tree diagram; all nodes in the control model communicate with each other using a wireless network; the control model is established under a network attack environment; and the control model is distributedly regulated by a first strategy generated by a pre-trained multi-agent. The embodiment of the present application designs a 5G wireless network coverage and control model, covering multiple terminal nodes, each node is equipped with a sensor and a controller, and end-to-end information transmission is performed through wireless communication to ensure coordination and cooperation between nodes. A cross-domain multi-objective optimization problem is established based on the voltage control of the 5G wireless network distribution network, and a collaborative control scheme for resource optimization and voltage control is constructed. By introducing binary variables and control strategies, real-time monitoring and response to Denial of Service (DoS) attacks are carried out to ensure the stable operation of the network, and the duration range of the occurrence and non-occurrence of DoS attacks is set, and the optimization model is used to limit the impact of packet loss rate. A solution algorithm based on multi-agent reinforcement learning was proposed, which enables each agent to rely on accurate environmental information collection when making decisions, perform cross-domain collaborative optimization in a distributed manner, and achieve information transmission reliability while ensuring control stability.

[0087] The technical solutions of the embodiments of the present application are described in detail below through specific examples.

[0088] refer to Figure 1 The control system wireless resource allocation method of the embodiment of the present application includes the following steps:

[0089] Step S101, constructing a tree diagram according to the topology diagram of the power distribution network;

[0090] Step S102, constructing a control model based on the tree diagram; all nodes in the control model communicate with each other using a wireless network; the control model is established under a network attack environment;

[0091] Step S103: performing distributed resource regulation on the control model based on a first strategy generated by the pre-trained multi-agent.

[0092] With respect to step S101 , a tree diagram is constructed according to the topology diagram of the power distribution network.

[0093] refer to Figure 2 , is a schematic diagram of the power distribution network topology of an embodiment of the present application.

[0094] like Figure 2 As shown, G represents the traditional generator, L represents the load, and V represents the photovoltaic power generation location.

[0095] Considering the medium- and low-voltage residential distribution networks where photovoltaic power generation is highly prevalent, the power system consists of a power plant, a transformer distribution network, and multiple users. Power plants generate electricity and transmit it through transmission lines. Multi-stage transformers are used to gradually reduce the voltage level in the distribution network, ultimately consuming it for users such as homes and factories. For ease of operation, large power grids are typically divided into multiple regions, each with several photovoltaic power plants installed and managed by the distribution network owner. Each photovoltaic plant generates reactive power to control the voltage at a fixed value, v ref Nearby. When both the load and photovoltaic generation system (PV) are operating normally, PV power generation efficiency fluctuates with factors such as weather and time of day, necessitating a control system to manage different PV systems and mitigate voltage fluctuations. However, since a random number of PVs exist in a power grid, the agents controlling each PV system involve a significant amount of collaboration or competition. Each PV-controlling agent relies on accurate collection of environmental information to make decisions. If this observational information becomes unavailable or delayed, the effectiveness of PV control will be compromised, threatening voltage stability and the security of the distribution network.

[0096] In some embodiments, the tree diagram is represented by the following formula:

[0097]

[0098] Among them, V represents the node set and E represents the edge set.

[0099] In this embodiment, Figure 2 The distribution network described in is modeled as a tree graph Where V = {0, 1, ..., N}, E = {1, 2, ..., N} represents nodes and edges respectively. Nodes 0 and 1 are considered to be connected to the main grid with constant voltage and infinite active and reactive power capacity to balance the active and reactive power in the distribution network. For each node i∈V, let v i and θ i are the amplitude and phase angle of the complex voltage, s i =p i +jq i is the complex power injection, where p i represents the active power of the i-th node, q i represents the reactive power of the i-th node, then the active power injection and reactive power injection can be defined as

[0100]

[0101] Among them, v i :={j|(i,j)∈E} is the set of node indices connected to i, g ij and b ij is the conductance and susceptance on branch (i, j), θ ij =θ i -θ j is the phase difference between nodes i and j. and are the active power and reactive power of node i respectively, Indicates the active power of the load, Indicates the reactive power of the load.

[0102] Furthermore, with respect to step S102, a control model is constructed based on the tree diagram; all nodes in the control model communicate with each other using a wireless network; and the control model is established under a network attack environment.

[0103] In some embodiments, the tree diagram includes at least one area; the control model is constructed based on the tree diagram, including: setting up a base station in each of the areas; the area includes at least one terminal node and at least one solar photovoltaic node; the base station is connected to each of the solar photovoltaic nodes; a sensor is set at each of the terminal nodes; a controller is set at each of the solar photovoltaic nodes; and a wireless communication connection is established between each of the terminal nodes and each of the solar photovoltaic nodes.

[0104] based on Figure 2 For the power system described in , this application establishes a system model based on 5G wireless network control.

[0105] refer to Figure 3 , which is a schematic diagram of the control model of an embodiment of the present application.

[0106] like Figure 3 As shown, first of all, it can be seen from the above that the distribution network can be divided into several areas, and a base station is set up in each area. Considering that the area contains m terminal nodes, since each node is connected to the load, each node is equipped with a sensor, and the controller is set at the PV node. All nodes can perform end-to-end wireless communication to transmit observation information.

[0107] Furthermore, among the various possible targets of network attacks discussed in the network system, wireless network control systems are susceptible to DoS attacks or deception attacks due to their certain non-cooperative characteristics. Therefore, we further consider that the most common and effective attacks are DoS attacks that may attack the perception layer and data interaction layer. DoS attacks are more common because they are easy to implement and are the most common type of network attack that needs to be dealt with. This application mainly establishes a model for the DoS attack environment.

[0108] In some embodiments, the network attack environment includes a denial of service attack environment; the denial of service attack environment is constructed by the following method: determining the attack frequency and attack duration of the denial of service attack; determining the signal-to-noise ratio and packet loss rate received by the controller; and constructing the denial of service attack environment based on the attack frequency, the attack duration, the signal-to-noise ratio and the packet loss rate.

[0109] In some embodiments, determining the attack frequency and attack duration of the denial of service attack includes: determining the upper and lower limits of the duration for which the denial of service attack occurs; determining the upper and lower limits of the duration for which the denial of service attack does not occur; determining the attack frequency of the denial of service attack based on the lower limit of the duration for which the attack occurs and the lower limit of the duration for which the attack does not occur; and determining the attack duration of the denial of service attack based on the lower limit of the duration for which the attack occurs, the lower limit of the duration for which the attack does not occur, and the upper limit of the duration for which the attack does not occur.

[0110] To describe the characteristics of DoS attacks, define the binary variable ζ(k) as

[0111]

[0112] In addition, the present invention defines two variables α n and β nis the duration of the DoS attack and the duration of the DoS attack not occurring. And there exists a positive scalar α min ,α max and β min ,β max Satisfy α min ≤α n ≤α max and β min ≤β n ≤β max α min and β min Represents α n and β n The uniform lower bound of α max and β max Represents α n and β n The consistent upper bound of .

[0113] Since in real scenarios, the attacker's energy is necessarily limited rather than infinite, the DoS attack must have an upper and lower time limit, so we can make the following assumptions:

[0114] Assumption 1 (DoS attack frequency): The number of DoS attack end / occurrence transitions n(k0,k) in the interval [k0,k) satisfies:

[0115]

[0116] Assumption 2 (DoS attack duration): The duration of the DoS attack Π(k0,k) satisfies the following in the interval [k0,k):

[0117]

[0118] In addition to man-made network attacks, control systems based on wireless networks are bound to face noise, natural interference, and various electromagnetic interferences. Therefore, this application assumes that the bandwidth allocated to the mth sensor at time point n and the transmission time are N respectively. m,n With T m,n , then for the mth sensor at time point n, the signal-to-noise ratio γ received by the controller m,n It can be expressed as:

[0119]

[0120] Among them, h m,n is the small-scale fading of the mth sensor at time n, g m is the path loss. According to research, it is only related to the path length. N m,n is the amount of allocated bandwidth, n mis the transmission power spectral density assigned to the mth sensor, B0 is the spacing between subcarriers, and N0 is the unilateral noise spectral density. Then the bandwidth B used by the mth sensor is m,n =B0N m,n The Shannon capacity of this channel can be expressed as:

[0121] C m,n =log(1+γ m,n )

[0122] The present invention assumes that the payload size of each sensor is m are the same, then λ m can be rewritten as λ, then for a given λ there are N m,n T m,n B0R m,n =λ, the packet loss rate of the controller for the mth node at time point n is expressed as follows:

[0123]

[0124] Among them, f Q represents the Q function, T m,n Represents the time domain resources for the mth node at time point n.

[0125] Furthermore, with respect to step S103, the control model is subjected to distributed resource regulation based on the first strategy generated by the pre-trained multi-agent.

[0126] In some embodiments, the first strategy generated based on pre-trained multi-agent performs distributed resource regulation on the control model, including: determining a reward function; setting a second strategy for the control model; the second strategy is a random strategy; performing gradient iterative processing on the second strategy with the goal of maximizing the reward function; obtaining the first strategy in response to the end of the gradient iteration; and performing distributed resource regulation on the control model according to the first strategy.

[0127] The reward function is determined by the following steps:

[0128] The ultimate optimization goal of the entire wireless network control system can be seen as requiring the voltage in the power system to be controlled near the rated voltage while minimizing reactive power, that is, maximizing the negative voltage barrier function. This invention mathematically expresses the control optimization problem as a constrained optimization problem, with the goal of maximizing long-term performance in terms of network load balancing and voltage control stability. The optimization goal for voltage control can then be established.

[0129]

[0130] Among them, l v (·) is the voltage barrier function, is the reactive power loss, and the constraint condition can be obtained from the above objectives And ∈>0. Through the KKT condition, we can use the Lagrange multiplier α∈(0,1) to convert the constrained reward into an unconstrained reward. Then we have

[0131]

[0132] Wherein, V represents voltage.

[0133] Based on the wireless network control system model, the spectrum efficiency is calculated using the following formula:

[0134]

[0135] This application achieves the optimal uplink wireless resource allocation with guaranteed reliability by establishing a problem of maximizing spectrum efficiency, which is expressed as follows:

[0136]

[0137] stC1:N m,n p m,n ≤p max ,

[0138] C2:ε m,n ≤ε th ,

[0139] C3:B m,n ≤B max ,

[0140]

[0141] Among them, p max is the maximum total power allocated to the communication network, ε th For a given reliability limit, B max is the maximum bandwidth allocated to the communication network. S is the successful transmission rate, B m,n is the bandwidth of the mth node at time n, γ m,n is the signal-to-noise ratio of the mth node at time n. is the reactive power loss, p m,n Assign the power spectral density of the mth node at time n.

[0142] This application addresses both voltage control and resource allocation issues under network attacks. Therefore, we decided to weight both issues equally to obtain the final overall reward function, which is expressed as follows:

[0143]

[0144] stC1:Nm,n p m,n ≤p max ,

[0145] C2:ε m,n ≤ε th ,

[0146] C3:B m,n ≤B max ,

[0147]

[0148] Furthermore, considering the complexity of the above decision-making problem, this application will choose the Multi-agent Reinforcement Learning (MARL) algorithm as a solution.

[0149] Specifically, the MARL algorithm excels at collaborative learning and action, and can effectively allocate tasks and resources among multiple agents, which is crucial for handling complex power wireless network systems. The algorithm has high robustness and fault tolerance, and can maintain stable operation of the overall system even when some agents fail. In addition, MARL supports flexible scalability and can adjust the number and functions of agents according to the actual needs of the system, which allows the algorithm to adapt to changing network conditions and external environments. These characteristics make multi-agent reinforcement learning a very suitable choice when facing the power wireless network system problem under complex external environments proposed in this application.

[0150] This application uses a multi-agent reinforcement learning algorithm to effectively solve the constructed multi-objective problem, transforming the weighted multi-objective problem into a multi-agent reinforcement learning strategy optimization problem. The MARL problem can be described as a non-central partially observable Markov decision process (Dec-POMDP), which is represented by the following tuple:

[0151] in, Is a set of agents, each of which controls a PV. At a node in , we define a function Indicates the node where the agent is located.

[0152] is the union of the entire power grid into m regions, where And when k1≠k2, And define a function Indicates the region where each node is located.

[0153] is the cross-domain fusion state set, where is the load power set, It is the active power collection of photovoltaic power generation. is the reactive power set generated by photovoltaic power generation in the previous step, is the set of voltages, where is the voltage magnitude vector, θ is the voltage phase vector measured in terms of radius, and the vector The components are expressed as Here we define a function Represents a subset of V and The mapping between the metrics in Represents a power set.

[0154] Joint Observation Set in represents the measurement of the area where agent i is located, Represents the set of all agents.

[0155] Joint Action Set Each of the intelligent Possesses a continuous action set in It represents the ratio of the maximum reactive power it generates. The reactive power generated by the kth PV is in represents the maximum apparent power, which is determined by the physical properties of the PV owned by the kth node. k >0, PV outputs reactive power to the distribution grid. k <0, the reactive power obtained by PV from the distribution network, the value of c is usually selected according to the actual load capacity of the grid to ensure the safe operation of the grid. Indicates the wireless link power allocation result.

[0156] is the transition probability function that describes the dynamics of the environment.

[0157] Is a global reward function that describes the reward for all agents based on their decisions. Given a weighted optimization problem, this application sets the reward function to

[0158] is the observation probability function, which describes the possible measurement error in the sensor and is determined by the physical properties of the sensor; is the probability function of the initial state; Υ∈(0,1) is a discount factor.

[0159] The goal of Dec-POMDP is to find the optimal joint policy So that there is

[0160]

[0161] in, represents the expected value under strategy v, Υ t represents the discount factor at time t, r t represents the reward at time t.

[0162] The embodiment of the present application constructs a multi-agent reinforcement learning based on the executor (Actor) and the evaluator (Critic) neural network, and the Actor parameters are updated as follows: Assume that θ = [θ1, θ2, ..., θ n ] represents the neural network parameters of n agents, π=[π1,π2,…,π n ] represents the strategy of n agents, then the expected return of each agent is:

[0163]

[0164] The policy gradient for the stochastic policy is:

[0165]

[0166] In the formula, o i represents the local information observed by the i-th agent; x = [o1, o2, ..., o n ] is global information; Represents the functional relationship between the state and action of the i-th agent; for a specific strategy Its gradient function is:

[0167]

[0168] Where D represents the experience array, Indicates that in the strategy μ i The action value function, μ i represents the strategy of the i-th agent. The update method of critic parameters is very similar to that of single-agent DDPG, and its loss function is:

[0169]

[0170] in, For the target network.

[0171] As can be seen from the above embodiments, the control system wireless resource allocation method described in the embodiments of the present application constructs a tree diagram based on the topology of the power distribution network; constructs a control model based on the tree diagram; all nodes in the control model communicate using a wireless network; the control model is established under a network attack environment; and the control model is distributedly controlled based on a first strategy generated by a pre-trained multi-agent. The embodiments of the present application design a 5G wireless network coverage and control model that covers multiple terminal nodes, each equipped with sensors and controllers, and performs end-to-end information transmission via wireless communication to ensure coordination and cooperation between nodes. A cross-domain multi-objective optimization problem is established based on the voltage control of the 5G wireless network distribution network. A coordinated resource optimization and voltage control scheme is constructed. By introducing binary variables and control strategies, real-time monitoring and response to denial of service attacks are achieved to ensure stable network operation. The duration range of DoS attacks is set, and the optimization model is used to limit the impact of packet loss rate. A solution algorithm based on multi-agent reinforcement learning is also proposed, enabling each agent to rely on accurate environmental information collection when making decisions, performing cross-domain collaborative optimization in a distributed manner, achieving information transmission reliability while ensuring control stability.

[0172] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0173] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0174] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a control system wireless resource allocation device.

[0175] refer to Figure 4 , the control system wireless resource allocation device includes:

[0176] A first building module 41 is configured to build a tree diagram according to a topology diagram of a power distribution network;

[0177] The second building module 42 is configured to build a control model based on the tree diagram; all nodes in the control model communicate with each other using a wireless network; and the control model is built under a network attack environment;

[0178] The allocation module 43 is configured to perform distributed resource regulation on the control model based on a first strategy generated by a pre-trained multi-agent.

[0179] In some embodiments, the tree diagram is represented by the following formula:

[0180]

[0181] Among them, V represents the node set and E represents the edge set.

[0182] In some embodiments, the tree diagram includes at least one region;

[0183] The second building block is further configured to:

[0184] A base station is set up in each of the areas; the area includes at least one terminal node and at least one solar photovoltaic node; the base station is connected to each of the solar photovoltaic nodes;

[0185] placing a sensor at each of the terminal nodes;

[0186] Setting a controller at each of the solar photovoltaic nodes;

[0187] A wireless communication connection is established between each of the terminal nodes and each of the solar photovoltaic nodes.

[0188] In some embodiments, the network attack environment includes a denial of service attack environment;

[0189] The device further comprises:

[0190] A first determining module is configured to determine the attack frequency and attack duration of the denial of service attack;

[0191] a second determining module, configured to determine a signal-to-noise ratio and a packet loss rate received by the controller;

[0192] The third building module is configured to build the denial of service attack environment based on the attack frequency, the attack duration, the signal-to-noise ratio, and the packet loss rate.

[0193] In some embodiments, the first determining module is further configured to:

[0194] Determining upper and lower limits on the duration of the denial of service attack;

[0195] Determining upper and lower bounds on the duration of time during which the denial of service attack does not occur;

[0196] Determine the attack frequency of the denial of service attack according to the lower limit of the duration of occurrence and the lower limit of the duration of non-occurrence;

[0197] The attack duration of the denial of service attack is determined according to the lower limit of the duration of occurrence, the lower limit of the duration of non-occurrence, and the upper limit of the duration of non-occurrence.

[0198] In some embodiments, the signal-to-noise ratio is represented by the following formula:

[0199]

[0200] Among them, γ m,n represents the signal-to-noise ratio received by the controller at the mth sensor at time point n, h m,n represents the small-scale fading of the mth sensor at time point n, g m represents the path loss, p m,n represents the transmission power spectral density assigned to the mth sensor at time point n, and N0 represents the one-sided noise spectral density.

[0201] In some embodiments, the allocation module is further configured to:

[0202] Determine the reward function;

[0203] Setting a second strategy of the control model; the second strategy is a random strategy;

[0204] performing gradient iteration on the second strategy with the goal of maximizing the reward function; obtaining the first strategy in response to completion of the gradient iteration;

[0205] Distributed resource regulation is performed on the control model according to the first strategy.

[0206] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0207] The apparatus of the above embodiment is used to implement the corresponding control system wireless resource allocation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0208] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the control system wireless resource allocation method described in any of the above embodiments is implemented.

[0209] Figure 5 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0210] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0211] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0212] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0213] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0214] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0215] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0216] The electronic device of the above embodiment is used to implement the corresponding control system wireless resource allocation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0217] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the control system wireless resource allocation method as described in any of the above embodiments.

[0218] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0219] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the control system wireless resource allocation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0220] Based on the same inventive concept, corresponding to the control system wireless resource allocation method described in any of the above embodiments, the present disclosure further provides a computer program product comprising computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the control system wireless resource allocation method. For the execution entities corresponding to the steps in each embodiment of the control system wireless resource allocation method, the processors executing the corresponding steps can belong to the corresponding execution entities.

[0221] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the control system wireless resource allocation method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0222] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0223] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0224] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0225] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A control system wireless resource allocation method, characterized in that: include: Construct a tree diagram based on the topology of the power distribution network; A control model is constructed according to the tree diagram; all nodes in the control model communicate with each other using a wireless network; The control model is established in the context of a cyber attack; The control model is subjected to distributed resource regulation based on a first strategy generated by a pre-trained multi-agent.

2. The method according to claim 1, characterized in that: The tree diagram is represented by the following formula: Among them, V represents the node set and E represents the edge set.

3. The method according to claim 1, characterized in that The tree diagram includes at least one region; The step of constructing a control model according to the tree diagram comprises: A base station is set in each of the areas; the area includes at least one terminal node and at least one solar photovoltaic node; the base station is connected to each of the solar photovoltaic nodes; Disposing a sensor at each of the terminal nodes; The controller is arranged at each of the solar photovoltaic nodes; A wireless communication connection is established between each of the terminal nodes and each of the solar photovoltaic nodes.

4. The method according to claim 3, characterized in that The network attack environment includes a denial of service attack environment; The denial of service attack environment is constructed by the following method: Determining the attack frequency and attack duration of the denial of service attack; determining a signal-to-noise ratio and a packet loss rate received by the controller; The environment of the denial of service attack is constructed based on the attack frequency, the attack duration, the signal-to-noise ratio and the packet loss rate.

5. The method according to claim 4, characterized in that Determining the attack frequency and attack duration of the denial of service attack includes: Determining upper and lower limits on the duration of the denial of service attack; determining upper and lower bounds on the duration during which the denial of service attack does not occur; Determine the attack frequency of the denial of service attack according to the lower limit of the duration of occurrence and the lower limit of the duration of non-occurrence; The attack duration of the denial of service attack is determined according to the lower limit of the duration of occurrence, the lower limit of the duration of non-occurrence, and the upper limit of the duration of non-occurrence.

6. The method according to claim 4, characterized in that The signal-to-noise ratio is expressed by the following formula: Among them, γ m,n represents the signal-to-noise ratio received by the controller at the mth sensor at time point n, h m,n represents the small-scale fading of the mth sensor at time point n, g m represents the path loss, p m,n represents the transmission power spectral density allocated to the mth sensor at time point n, and N0 represents the one-sided noise spectral density.

7. The method according to claim 1, characterized in that The first strategy generated based on the pre-trained multi-agent performs distributed resource regulation on the control model, including: Determine the reward function; Setting a second strategy of the control model; the second strategy is a random strategy; Performing gradient iteration processing on the second strategy with the goal of maximizing the reward function; obtaining the first strategy in response to the completion of the gradient iteration; Distributed resource regulation is performed on the control model according to the first strategy.

8. A control system wireless resource allocation device, characterized in that: include: A first building module is configured to build a tree diagram according to a topological diagram of a power distribution network; A second building module is configured to build a control model according to the tree diagram; all nodes in the control model communicate with each other using a wireless network; The control model is established in the context of a cyber attack; The allocation module is configured to perform distributed resource regulation on the control model based on a first strategy generated by a pre-trained multi-agent.

9. The device according to claim 8, characterized in that The tree diagram is represented by the following formula: Among them, V represents the node set and E represents the edge set.

10. The device according to claim 8, characterized in that The tree diagram includes at least one region; The second building block is further configured to: A base station is set in each of the areas; the area includes at least one terminal node and at least one solar photovoltaic node; the base station is connected to each of the solar photovoltaic nodes; Disposing a sensor at each of the terminal nodes; The controller is arranged at each of the solar photovoltaic nodes; A wireless communication connection is established between each of the terminal nodes and each of the solar photovoltaic nodes.

11. The device according to claim 10, characterized in that The network attack environment includes a denial of service attack environment; The device also includes: A first determination module is configured to determine the attack frequency and attack duration of the denial of service attack; A second determination module is configured to determine a signal-to-noise ratio and a packet loss rate received by the controller; The third building module is configured to build the environment of the denial of service attack based on the attack frequency, the attack duration, the signal-to-noise ratio and the packet loss rate.

12. The device according to claim 11, characterized in that The first determining module is further configured to: Determining upper and lower limits on the duration of the denial of service attack; determining upper and lower bounds on the duration during which the denial of service attack does not occur; Determine the attack frequency of the denial of service attack according to the lower limit of the duration of occurrence and the lower limit of the duration of non-occurrence; The attack duration of the denial of service attack is determined according to the lower limit of the duration of occurrence, the lower limit of the duration of non-occurrence, and the upper limit of the duration of non-occurrence.

13. The device according to claim 11, characterized in that The signal-to-noise ratio is expressed by the following formula: Among them, γ m,n represents the signal-to-noise ratio received by the controller at the mth sensor at time point n, h m,n represents the small-scale fading of the mth sensor at time point n, g m represents the path loss, p m,n represents the transmission power spectral density allocated to the mth sensor at time point n, and N0 represents the one-sided noise spectral density.

14. The device according to claim 8, characterized in that The allocation module is further configured to: Determine the reward function; Setting a second strategy of the control model; the second strategy is a random strategy; Performing gradient iteration processing on the second strategy with the goal of maximizing the reward function; obtaining the first strategy in response to the completion of the gradient iteration; Distributed resource regulation is performed on the control model according to the first strategy.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

17. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images