A demand response user side multi-agent collaborative decision-making method based on switchable preference double-map reinforcement learning

Abstract

This invention discloses a multi-agent collaborative decision-making method for demand response on the user side based on switchable preference dual-graph reinforcement learning. By introducing switchable preference parameters simultaneously at the reward and policy layers, it achieves continuous adjustability among multiple objectives such as economy, comfort, and energy storage device health. It constructs a dual-graph structure of homogeneous and heterogeneous graphs and filters cross-type key interaction objects based on a scale-matching key interaction object selection mechanism. Local attention vectors and global attention vectors are used to aggregate competitive and collaborative features respectively, forming a hybrid competitive-collaborative feature, which is then input into a preference-conditionalized policy network. Finally, a proximal policy optimization algorithm is used for centralized training and distributed execution, resulting in a hierarchical demand response collaborative decision-making scheme that allows for switching policy behaviors through preference parameters during runtime. This invention addresses the problem that multi-objective user-side agents struggle to effectively formulate decisions that align with their real-time preferences during demand response tasks.

Description

Technical Field

[0001] This invention relates to the field of demand response, and more specifically to a multi-agent collaborative decision-making method for demand response on the user side based on switchable preference dual-graph reinforcement learning. Background Technology

[0002] In power distribution systems with a high proportion of distributed resource access, the demand response involving user-side flexible resources is evolving from traditional single-entity load control to a multi-entity collaborative adjustment process. The system typically includes three core entities: aggregators, shared energy storage operators, and users with various flexible resources. User-side flexible resource operation is influenced by comfort constraints, equipment physical boundaries, and energy service demands. Shared energy storage operators assist users in energy time-shifting and supply-demand balance through charging and discharging power control, but are limited by state of charge and battery health constraints. Aggregators primarily use incentive signals to organize user participation in response, aiming to increase response scale and compliance probability while controlling supply-demand adjustment costs and user comfort. The interaction among these three entities on the same timescale leads to significant cross-entity coupling, conflicting objectives, and uncertainty in decision-making.

[0003] Existing research on the aforementioned hierarchical collaborative decision-making problem can be mainly divided into three categories. The first category is optimization-based centralized or hierarchical planning methods, which typically use user response quantities, power control, and guidance signals as decision variables, aiming to achieve a balance between system regulation objectives and energy efficiency through constrained optimization. The second category is game theory-based methods, which focus on characterizing the strategic interactions and interactive equilibria between agents (such as aggregators and users). The third category is data-driven methods based on MARL, which typically use graph representation learning to characterize the interaction structure of agents and use fixed-weight multi-objective rewards for training.

[0004] Based on the MARL-driven data approach, aggregators, shared energy storage operators, and users are modeled as intelligent agents that interact in discrete time series: aggregators output excitation signals based on observations, users schedule various flexible resources based on signals, and shared energy storage outputs charging and discharging power and supply and demand matching actions; the environment updates its state based on physical constraints and feeds back interactive data including response quantities, supply and demand adjustment costs, and equipment status.

[0005] However, the existing technology has the following shortcomings: (1) It is difficult for multi-target user-side subjects to make effective decisions that conform to their real-time preferences during the execution of demand response tasks; (2) It is difficult to characterize the decision migration of user-side subjects participating in demand response under similar collaboration and the behavioral restraint effect under heterogeneous competition; (3) It is difficult to accurately depict the marginal restraint relationship and resource competition behavior among multiple types of user-side subjects in demand response tasks; (4) When the scale of multiple subjects expands or the state changes rapidly, the existing methods are unstable in training and have insufficient generalization ability, making it difficult to adapt to demand response tasks in multiple time periods and multiple scenarios. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a multi-agent collaborative decision-making method for demand response on the user side based on switchable preference dual-graph reinforcement learning, which can solve the technical problem in the prior art that multi-objective user-side agents have difficulty in effectively making decisions that conform to their real-time preferences during the execution of demand response tasks.

[0007] Technical solution: The present invention provides a multi-agent collaborative decision-making method for demand response on the user side based on switchable preference dual-graph reinforcement learning, comprising the following steps:

[0008] Establish a hierarchical demand response system, which is a set of user-side entities consisting of aggregators, shared energy storage operators, and users. Based on the hierarchical demand response system, the behavior of each user-side entity is described as a partially observable Markov decision process. Establish a demand response behavior model for the user set for the partially observable Markov decision process of each user-side entity.

[0009] Establish a comfort characterization model for user sets to quantify the comfort loss of demand response behavior models; construct a shared energy storage model for shared energy storage operator sets; construct a time-of-use compensation pricing model for aggregator sets based on the demand response behavior model of user sets; and characterize the demand response peak-shaving target gap of hierarchical demand response systems based on the demand response behavior model of user sets.

[0010] Construct switchable preference parameters, and use these switchable preference parameters to define the preference-conditional multi-objective rewards for the established comfort characterization model, the shared energy storage model, the time-of-use compensation pricing model, and the demand response peak shaving target.

[0011] Based on the real-time physical state quantities of each subject output by the demand response behavior model, the shared energy storage model, and the time-of-use compensation pricing model, the interaction intensity weight is calculated and the key interaction object is selected by scale matching. Based on the selected key interaction objects, the heterogeneous key neighbors and homogeneous cooperative neighbors of each subject are screened from the user-side subject set to form a heterogeneous key neighbor set and a homogeneous cooperative neighbor set. Based on the heterogeneous key neighbor set and the homogeneous cooperative neighbor set, a dual-graph structure is constructed for each user-side subject.

[0012] Within the dual-graph structure of all user-side subjects, the hybrid competitive and cooperative relationship between adjacent user-side subjects is described based on the hybrid graph attention mechanism, resulting in hybrid competitive and cooperative relationship features. Subject preference embedding vectors are constructed based on the hybrid competitive and cooperative relationship features. The preference embedding vectors are input into the preference conditionalization policy network to obtain action distribution parameters, and the preference embedding vectors are input into the value network to obtain preference conditionalization state values.

[0013] By utilizing action distribution parameters, preference-conditional state values, and preference-conditional multi-objective rewards, combined with a proximal policy optimization reinforcement learning algorithm, preference-conditional policy networks and value networks are trained separately until convergence. This yields the final policy parameters of the preference-conditional policy network and the final policy parameters of the value network for each subject in the user-side subject set. The converged final policy parameters are then used to output preference switching and collaborative decision-making during the online operation phase.

[0014] Furthermore, the construction of switchable preference parameters, using these parameters to define the preference-conditional multi-objective rewards for the established comfort characterization model, the shared energy storage model, the time-of-use pricing model, and the demand response peak-shaving target, includes:

[0015] Introducing preference parameters It also allows preference parameters to be switched during runtime;

[0016] Preference parameters are set for aggregators, shared energy storage operators, and users respectively. These preference parameters are then used to define preference-conditional multi-objective rewards, as detailed below:

[0017] Aggregator The reward is defined as:

[0018] ;

[0019] in, Rewards for aggregators; These are the aggregator's preference parameters; Weighting for multi-objective rewards; This represents the total response volume within the aggregated region. For aggregators At any moment Compensation expenditures; In response to the gap; The average level of inappropriateness in the jurisdiction;

[0020] Shared energy storage operators The reward is defined as:

[0021] ;

[0022] in, Incentives for shared energy storage operators; These are the preference parameters for shared energy storage operators; Weighting for multi-objective rewards; As an indicator of energy storage profitability; It is in a charged state; For reference state of charge; Energy storage capacity; This represents the maximum energy storage capacity. To take the absolute value; This is a power stress penalty term;

[0023] User The reward is defined as:

[0024] ;

[0025] in, As a reward for users; For user preference parameters; Weighting for multi-objective rewards; To compensate for the losses; Electricity expenses; This is inappropriate.

[0026] Furthermore, the calculation of interaction intensity weights and the selection of key interaction objects based on the real-time physical state quantities of each entity output by the demand response behavior model, the shared energy storage model, and the time-of-use compensation pricing model includes:

[0027] A scale-matching key interaction object selection mechanism is introduced to define interaction intensity weights for different types of subjects:

[0028] ;

[0029] in, Indicates user At any moment Interaction strength weight; Indicates user Net load; Indicating shared energy storage operators Interaction strength weight; Indicating shared energy storage operators At any moment The charging and discharging power; Indicating shared energy storage operators The maximum charging and discharging power; Indicates aggregator At any moment Interaction strength weight; Indicates aggregator The sum of responses within the aggregated range;

[0030] For the target subject With cross-type candidate subjects Define the scale matching score:

[0031] ;

[0032] in, Indicates the target subject With cross-type candidate subjects Scale matching score between; For the target subject Interaction strength weight; For cross-type candidate subjects Interaction strength weight; For scale parameters; It is an exponential function;

[0033] Based on the target subject With cross-type candidate subjects Scale matching score between From heterogeneous candidate sets Select heterogeneous key neighbor sets from the data. To construct a heterogeneous graph edge set, the key interaction objects in the heterogeneous graph edge set. Defined as the representative object with the highest score:

[0034] ;

[0035] in, For the target subject Key heterogeneous interaction subjects; This represents the independent variable that corresponds to the maximum value.

[0036] The heterogeneous key neighbor set is defined according to the target subject. With cross-type candidate subjects Scale matching score between Select the highest score A cross-type object:

[0037] ;

[0038] in, Indicates before selection The set of elements corresponding to the maximum values; Given a preset number of neighbors, the edge set of a heterogeneous graph can be written as: ;

[0039] For homogeneous candidate sets Define the target subject With cross-type candidate subjects Similarity of state variables during the execution of demand response:

[0040] ;

[0041] in, For state quantity similarity; For the target subject The state variables during the execution of demand response; For cross-type candidate subjects State variables during the execution of demand response; It is the Euclidean norm; For similarity scale parameters;

[0042] Based on state quantity similarity From a homogeneous candidate set Select the most similar A set of homogeneous cooperative neighbors. And obtain the set of edges of the homogeneous graph. .

[0043] Furthermore, within the dual-graph structure of all user-side entities, the hybrid competitive-cooperative relationship between adjacent user-side entities is described based on a hybrid graph attention mechanism, resulting in hybrid competitive-cooperative relationship features, including:

[0044] Mapping the principal state variables to a unified latent space:

[0045] ;

[0046] in, For the target subject The state variables during demand response are mapped to vector representations in a latent space that can be recognized by the neural network; and All of these are learnable neural network parameters; For the target subject State variables during the execution of demand response;

[0047] target subject Key heterogeneous interaction subjects State variables are mapped to a unified latent space:

[0048] ;

[0049] For heterogeneous key neighbor sets Construct attention scores:

[0050] ;

[0051] in, For the target subject Cross-type candidate subjects Attention scoring, cross-type candidate subjects As the target entity Heterogeneous neighbor entities; and All of these are learnable parameters of the neural network; It is the hyperbolic tangent function; This involves concatenating vectors. For transpose; For the target subject Key heterogeneous interaction subjects Latent space representation; for heterogeneous key neighbor sets Normalization is performed to obtain the attention weights:

[0052] ;

[0053] in, Attention weights;

[0054] Competitive features are obtained by aggregating heterogeneous key neighbors based on local attention vectors.

[0055] ;

[0056] in, For the target subject The heterogeneous competition characteristic; For the target subject Key heterogeneous interaction subjects The implicit space representation;

[0057] homogeneous The set of nearest neighbors forms a homogeneous cooperative neighbor set. Based on global attention vectors for homogeneous cooperative neighbor sets Aggregation is performed to obtain collaborative features:

[0058] ;

[0059] in, For the target subject The homogeneous collaborative characteristics; The cardinality of the set;

[0060] target subject The latent space representation Target subject heterogeneous competition characteristics With the target subject Homogeneous collaboration characteristics By performing hierarchical splicing and fusion, a hybrid competitive and collaborative feature is obtained:

[0061] ;

[0062] in, It is characterized by a hybrid competitive and cooperative approach; For converged networks; For splicing operators.

[0063] Furthermore, the construction of subject preference embedding vectors based on hybrid competitive and cooperative relationship features, inputting the preference embedding vectors into the preference conditionalization policy network to obtain action distribution parameters, and inputting the preference embedding vectors into the value network to obtain preference conditionalization state values, includes:

[0064] For the target subject Preference parameters Constructing the embedding map:

[0065] ;

[0066] in, Embedded vectors for subject preferences; For learnable embedding functions; For the target subject Preference parameters, Pick , or ;

[0067] Embedding preferences into vectors Input the preference-conditional policy network, and the preference-conditional policy network outputs action distribution parameters:

[0068] ;

[0069] in, For parameters The strategy network; The mean of the action distribution; The standard deviation of the action distribution; The standard deviation is the logarithm. For the target subject The observation vector; It is characterized by a hybrid competitive and cooperative approach; Embedded vectors for subject preferences; To construct the splicing operator, a Gaussian strategy is employed, and the action range is constrained by hyperbolic tangent.

[0070] ;

[0071] in, The main action; It is the hyperbolic tangent function; For Hadamard product; For noise; A Gaussian distribution with zero mean and unit covariance; It is the identity matrix; and All are outputs of the policy network;

[0072] The preference-embedded vector is input into the value network, and the value network outputs preference-conditionalized state values:

[0073] ;

[0074] in, For parameters Value network; For preference-conditional state values; For the target subject State variables during the execution of demand response; It is characterized by a hybrid competitive and cooperative approach; The subject's preference embedding vector.

[0075] Furthermore, the establishment of a demand response behavior model for the user set based on the observable Markov decision process of each user-side subject includes:

[0076] For users Define user exist The net load power at time t is:

[0077] ;

[0078] in, For users exist Net load power at any given time; For users exist Fixed load power at any given time; For users exist Air conditioner power at any given time; For users exist Electric vehicle charging power at any given time; For users exist The load power that can be reduced at any time; For users exist The transferable load power at any given time; For users exist Distributed generation output power at any given time;

[0079] Set user exist The baseline net load when there is no response is Define the reduced demand response quantity to obtain the expression for the demand response behavior model of the user set:

[0080] ;

[0081] in, For users exist real-time demand response time; For users exist Baseline net load when there is no response participation at any given moment; For users exist Net load power at any given time; This is the operator for finding the maximum value.

[0082] Furthermore, the establishment of a comfort characterization model for the user set to quantify the comfort loss of the demand response behavior model includes:

[0083] The user's indoor temperature is updated using a first-order thermal inertia model:

[0084] ;

[0085] in, For users exist The indoor temperature at any given time; For users exist The indoor temperature at any given time; for outdoor temperature at all times; For users The heat exchange coefficient; For users The equivalent influence coefficient of air conditioner power on room temperature change; For users exist Air conditioner power at any given time;

[0086] Define user exist If the discomfort at any given moment is a non-negative quantity representing the temperature deviation from the comfortable range, then the expression for the comfort characterization model of the user set is as follows:

[0087] ;

[0088] in, For users exist Inappropriate timing; and These represent the lower and upper boundaries of the comfortable temperature range, respectively. For users exist The indoor temperature at any given time; To find the maximum value operator; use the comfort characterization model of the user set to quantify the comfort loss of the demand response behavior model.

[0089] Furthermore, the construction of a shared energy storage model for a group of shared energy storage operators includes:

[0090] For shared energy storage operators The energy storage state of charge update formula is as follows:

[0091] ;

[0092] in, For shared energy storage operators exist The state of charge at any given moment; For shared energy storage operators exist The state of charge at any given moment; To be Cut off in interval Inside; For shared energy storage operators The charging efficiency; For shared energy storage operators The discharge efficiency; For shared energy storage operators Rated capacity; For time step; This refers to the charging power component. This refers to the discharge power component. and They are shared energy storage operators exist The upper and lower bounds of the charged state at any given time; Energy storage power; positive values ​​represent charging, and negative values ​​represent discharging. The following constraints must be met:

[0093] ;

[0094] in, It is the absolute value; This represents the maximum permissible charge and discharge power.

[0095] Furthermore, the time-sharing compensation pricing model for the aggregator set, constructed based on the demand response behavior model of the user set, includes:

[0096] Aggregator At any moment The compensation electricity price is:

[0097] ;

[0098] in, To compensate for electricity prices; , , These are peak, flat, and valley compensation electricity prices, respectively. For indicator functions; Time period type;

[0099] Compensation electricity price The following boundaries must be satisfied:

[0100] ;

[0101] in, and These are the lower and upper limits of the compensation electricity price, respectively. To compensate for electricity prices;

[0102] The expression for the time-sharing compensation pricing model for the aggregator set is as follows:

[0103] ;

[0104] in, For aggregators At any moment Compensation expenditures; For summation operators; For user collection; User affiliation; To compensate for electricity prices; For users exist The demand response volume at any given moment.

[0105] Furthermore, the user set-based demand response behavior model characterizes the demand response peak-shaving target gap of the hierarchical demand response system, including:

[0106] Suppose the aggregator is in The target response at time t is The response gap of the demand response peak-shaving target of a hierarchical demand response system is defined as:

[0107] ;

[0108] in, In response to the gap; For aggregators exist The target response quantity at any given time; This is an operation that sums the responses within the aggregator's jurisdiction; For users exist real-time demand response time; This is the operator for finding the maximum value.

[0109] Beneficial effects: Compared with the prior art, the significant technical effects of the present invention are as follows:

[0110] (1) By introducing switchable preference parameters and subject preference embedding vectors, the present invention defines preference-conditional multi-objective rewards using switchable preference parameters and applies subject preference embedding vectors to both policy conditionation and reward scalarization processes. This solves the technical problem in the prior art that it is difficult for multi-objective user-side subjects to make decisions that conform to their real-time preferences when performing demand response tasks. The technical effects that can be achieved are as follows: enabling aggregators, shared energy storage operators and users to continuously switch policies based on real-time control preferences during the online operation phase, and improving the balance between controllability and adaptability of subject decisions in multi-objective tasks.

[0111] (2) This invention proposes a homogeneous-heterogeneous dual-graph structure and extracts homogeneous collaboration features and heterogeneous competition features respectively. Then, through hierarchical fusion, a hybrid competition and collaboration feature is formed. This can solve the technical problem that it is difficult to characterize the decision transfer between user-side subjects participating in demand response under the same type of collaboration and the behavioral restraint effect under heterogeneous competition in the existing technology. The technical effects that can be achieved are as follows: It significantly enhances the expressive ability of cross-subject coupling mechanism and the interpretability of strategy learning.

[0112] (3) The scale matching key interaction object selection mechanism proposed in this invention filters the heterogeneous candidate set and focuses attention on the cross-type objects with the most significant marginal mutual influence. It can solve the technical problem that the marginal restraint relationship and resource competition behavior between multiple types of users in the demand response task in the prior art is difficult to accurately depict. The technical effects that can be achieved are as follows: improve the utilization efficiency of training samples and reduce the training interference caused by irrelevant interactions between subjects.

[0113] (4) This invention constrains the scope of information propagation through dual-graph structured aggregation and attention weight normalization mechanism, and supports cross-scene sharing and migration with unified latent space representation. It can solve the technical problem that existing methods are unstable in training and have insufficient generalization ability when the scale of multiple subjects expands or the state changes rapidly. The technical effects that can be achieved are as follows: improve training convergence stability and enhance the generalization ability to different operating scenarios. Attached Figure Description

[0114] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0115] The technical solution of the present invention will now be described in detail with reference to specific embodiments and accompanying drawings.

[0116] This invention presents a multi-agent collaborative decision-making method for demand response on the user side based on switchable preference dual-graph reinforcement learning. This method is designed for multi-agent collaborative decision-making among aggregators, shared energy storage operators, and flexible users in power distribution system demand response scenarios. The method belongs to the interdisciplinary technical field of Multi-Agent Reinforcement Learning (MARL), graph representation learning, and power demand response decision optimization. Figure 1 As shown, the present invention provides a multi-agent collaborative decision-making method for demand response on the user side based on switchable preference dual-graph reinforcement learning, which specifically includes the following steps:

[0117] S1. Establish a hierarchical demand response system, which is a set of user-side entities consisting of an aggregator set, a shared energy storage operator set, and a user set; based on the hierarchical demand response system, describe the behavior of each user-side entity as a partially observable Markov decision process; establish a demand response behavior model for the user set for the partially observable Markov decision process of each user-side entity.

[0118] The specific implementation process of step S1 is as follows:

[0119] S1.1 Establish a hierarchical demand response system. This system is a set of user-side entities consisting of aggregators, shared energy storage operators, and users, used to support subsequent modeling and collaborative decision-making among these user-side entities. Specifically:

[0120] Construct a discrete-time hierarchical demand response system, with the time index as follows: Define the set of aggregators. Shared energy storage operators User set Define user affiliation mapping. Indicates user By aggregator Jurisdiction, defining interaction mapping Indicates user with shared energy storage operators Main interactions.

[0121] At every moment The power distribution system operator provides time-of-use pricing. With time period type ,in These correspond to the peak load period, the shoulder load period, and the valley load period, respectively.

[0122] S1.2. Based on the hierarchical demand response system, the behaviors of each user-side entity are represented as partially observable Markov decision processes. Specifically:

[0123] The behavior of each user-side entity is represented as a partially observable Markov process:

[0124]

[0125] in, These are the tuple variables that represent the established Markov decision process; This is a set of main types, representing aggregator type, shared energy storage operator type, and user type, respectively; This represents the global state space of a hierarchical demand response system. For type The observation space; For type The action space; This is the state transition probability matrix; For type The reward function; This is the discount factor. This definition normalizes the variable boundaries for the physical modeling in subsequent steps S1.3 to S2.3 and provides a mathematical framework for the policy network input in step S5.2.

[0126] S1.3. Establish a demand response behavior model for the user set based on the observable Markov decision process of each user-side subject. Specifically:

[0127] For users Define user exist The net load power at time t is:

[0128]

[0129] in, For users exist Net load power at any given time; For users exist Fixed load power at any given time; For users exist Air conditioner power at any given time; For users exist Electric vehicle charging power at any given time; For users exist The load power that can be reduced at any time; For users exist The transferable load power at any given time; For users exist The distributed generation output power at any given time; the above power can be measured by sensors and instruments such as power meters.

[0130] Set user exist The baseline net load when there is no response is Define the reduced demand response quantity to obtain the expression for the demand response behavior model of the user set:

[0131]

[0132] in, For users exist real-time demand response time; For users exist Baseline net load when there is no response participation at any given moment; For users exist Net load power at any given time; This step calculates the user's maximum value using the operator. exist demand response time This will serve as the basis for calculating the interaction intensity weight in step S4, and will also provide feedback input for calculating the aggregator regulation effect in step S2.

[0133] S2. Establish a comfort characterization model for the user set to quantify the comfort loss of the demand response behavior model; construct a shared energy storage model for the shared energy storage operator set; construct a time-sharing compensation pricing model for the aggregator set based on the demand response behavior model of the user set; characterize the demand response peak shaving target gap of the hierarchical demand response system based on the demand response behavior model of the user set.

[0134] The specific implementation process of step S2 is as follows:

[0135] S2.1 Establish a comfort characterization model for the user set to quantify the comfort loss of the demand-response behavior model in step S1. Specifically:

[0136] The user's indoor temperature is updated using a first-order thermal inertia model:

[0137]

[0138] in, For users exist The indoor temperature at any given time, which can be measured by a temperature sensor installed on the user's main body; For users exist The indoor temperature at any given time; for The outdoor temperature at any given time can be measured by an outdoor temperature sensor; For users The heat exchange coefficient; For users The equivalent influence coefficient of air conditioner power on room temperature change; For users exist The air conditioner power at any given time.

[0139] Define user exist If the discomfort at any given moment is a non-negative quantity representing the temperature deviation from the comfortable range, then the expression for the comfort characterization model of the user set is as follows:

[0140]

[0141] in, For users exist Inappropriate timing; and These represent the lower and upper boundaries of the comfortable temperature range, respectively. For users exist The indoor temperature at any given time; To obtain the maximum value operator, the comfort characterization model of the user set is used to quantify the comfort loss of the demand response behavior model in step S1.

[0142] S2.2, Construct a shared energy storage model for the set of shared energy storage operators in step S1. Specifically:

[0143] For the shared energy storage operators defined in step S1 The energy storage state of charge update formula is as follows:

[0144]

[0145] in, For shared energy storage operators exist The state of charge at any given time, which can be obtained by the operator through real-time measurement of the energy storage state of charge; For shared energy storage operators exist The state of charge at any given moment; To be Cut off in interval Inside; For shared energy storage operators The charging efficiency; For shared energy storage operators The discharge efficiency; For shared energy storage operators The rated capacity, charge / discharge efficiency, and rated capacity parameter values ​​are given by the actual properties of the energy storage. For time step; For charging power components, ; This is the discharge power component, and ; and They are shared energy storage operators exist The upper and lower bounds of the charged state at any given time; This represents energy storage power; positive values ​​indicate charging, and negative values ​​indicate discharging. Energy storage power The following constraints must be met:

[0146]

[0147] in, It is the absolute value; This represents the maximum permissible charge and discharge power.

[0148] S2.3. Construct a time-sharing compensation pricing model for the aggregator set based on the demand response behavior model of the user set in step S1. Specifically:

[0149] Aggregator At any moment The compensation electricity price is:

[0150]

[0151] in, To compensate for electricity prices; , , These are peak, flat, and valley compensation electricity prices, respectively. For indicator functions; This is a time period type.

[0152] Compensation electricity price The following boundaries must be satisfied:

[0153]

[0154] in, and These are the lower and upper limits of the compensation electricity price, respectively. To compensate for the electricity price.

[0155] The expression for the time-sharing compensation pricing model for the aggregator set is as follows:

[0156]

[0157] in, For aggregators At any moment Compensation expenditures; For summation operators; For user collection; For user ownership, ; To compensate for electricity prices; For users exist The demand response at any given moment. The output signal of this step (i.e., the aggregator). At any moment Compensation expenditure This will directly influence users' decisions as observed input, and the regulation cost will serve as a negative indicator for the aggregator reward calculation in step S3.

[0158] S2.4. Based on the user set demand response behavior model constructed in step S1, the demand response peak-shaving target gap of the hierarchical demand response system is characterized. Specifically:

[0159] Suppose that in step S1, the aggregator is... The target response at time t is The response gap of the demand response peak-shaving target of a hierarchical demand response system is defined as:

[0160]

[0161] in, In response to the gap; For aggregators exist The target response quantity at any given time; This is an operation that sums the responses within the aggregator's jurisdiction; For users exist real-time demand response time; This is the operator for finding the maximum value.

[0162] S3. Construct switchable preference parameters and use them to define the objective of the established comfort characterization model, the objective of the shared energy storage model, the objective of the time-sharing compensation pricing model, and the objective of the demand response peak shaving target, thereby achieving flexible trade-offs for various performance indicators (such as regulation costs, inadequacy, response gap, energy storage efficiency, etc.) in the three models established in step S2.

[0163] The specific implementation process of step S3 is as follows:

[0164] To achieve continuous adjustability of multi-objective tradeoffs, a preference parameter is introduced. It also allows preference parameters to be switched during runtime.

[0165] Preference parameters are set for aggregators, shared energy storage operators, and users respectively. Let the preference parameter for aggregators be... The preference parameters for shared energy storage operators are: User preference parameters are The preference-conditional multi-objective reward is defined using preference parameters, as follows:

[0166] Aggregator The reward is defined as:

[0167]

[0168] in, Rewards for aggregators; These are the aggregator's preference parameters; Weighting for multi-objective rewards; This represents the total response volume within the aggregated region. ; To compensate for expenses; In response to the gap; The average level of inappropriateness in the jurisdiction ; The set of users within the aggregated region. ; Let be the cardinality of the set.

[0169] Shared energy storage operators The reward is defined as:

[0170]

[0171] in, Incentives for shared energy storage operators; These are the preference parameters for shared energy storage operators; Weighting for multi-objective rewards; As an indicator of energy storage profitability; It is in a charged state; For reference state of charge; Energy storage capacity; This represents the maximum energy storage capacity. To take the absolute value; This is a power stress penalty term.

[0172] User The reward is defined as:

[0173]

[0174] in, As a reward for users; For user preference parameters; Weighting for multi-objective rewards; To compensate for the losses, ; For electricity expenses, ; Inappropriate; Time-of-use pricing; This step represents the time step. This step provides support for flexibly balancing various performance indicators (such as regulation costs, inadequacy, response gap, and energy storage efficiency) in the three models established in step S2 during the training and online operation in the subsequent step S6.

[0175] S4. Based on the real-time physical state quantities of each subject output by the demand response behavior model of the user set in step S1, the shared energy storage model of the shared energy storage operator set in step S2, and the time-sharing compensation pricing model of the aggregator set, calculate the interaction intensity weight and perform scale matching to select key interaction objects; according to the selected key interaction objects, filter out the heterogeneous key neighbors and homogeneous cooperative neighbors of each subject from the user-side subject set to form a heterogeneous key neighbor set and a homogeneous cooperative neighbor set, and construct a dual-graph structure for each user-side subject based on the heterogeneous key neighbor set and the homogeneous cooperative neighbor set.

[0176] Specifically, based on the selected key interaction objects, heterogeneous key neighbors and homogeneous collaborative neighbors are filtered from the user-side subject set to form a heterogeneous key neighbor set and a homogeneous collaborative neighbor set. A bi-graph structure for each user-side subject is then constructed based on these sets, as detailed below:

[0177] Based on the selected key interaction objects, heterogeneous key neighbors and homogeneous collaborative neighbors for each subject are selected from the user-side subject set. The heterogeneous key neighbors of all subjects form a heterogeneous key neighbor set, thus constructing a homogeneous graph for each subject. The homogeneous collaborative neighbors of all subjects form a homogeneous collaborative neighbor set, thus constructing a heterogeneous graph for each subject. The homogeneous and heterogeneous graphs of each subject form a dual-graph structure, thereby constructing a dual-graph structure. The specific calculation process is as follows:

[0178] In a homogeneous-heterogeneous dual-graph structure, the heterogeneous graph is used to characterize the competitive coupling relationships between cross-type agents, while the homogeneous graph is used to characterize the collaborative migration relationships between agents of the same type. To select the interaction objects with the most significant impact on the target agent's decision-making and similar scales from cross-type candidates, a Scale-Matched Key Counterpart (SMKC) mechanism is introduced. First, interaction strength weights are defined for different types of agents:

[0179]

[0180] in, Indicates user At any moment Interaction strength weight; Indicates user Net load; Indicating shared energy storage operators Interaction strength weight; Indicating shared energy storage operators At any moment The charging and discharging power; Indicating shared energy storage operators The maximum charging and discharging power; Indicates aggregator At any moment Interaction strength weight; Indicates aggregator The sum of responses within the aggregated range.

[0181] For the target subject With cross-type candidate subjects (Right now , (For a heterogeneous candidate set), define the scale matching score:

[0182]

[0183] in, Indicates the target subject With cross-type candidate subjects Scale matching score between; For the target subject Interaction strength weight; For cross-type candidate subjects Interaction strength weight; For scale parameters; It is an exponential function.

[0184] Based on the target subject With cross-type candidate subjects Scale matching score between From heterogeneous candidate sets Select heterogeneous key neighbor sets from the data. To construct a heterogeneous graph edge set, the key interaction objects in the heterogeneous graph edge set. Defined as the representative object with the highest score:

[0185]

[0186] in, For the target subject Key heterogeneous interaction subjects; This represents the independent variable that corresponds to the maximum value.

[0187] To ensure that subsequent local attention vector aggregation has non-degenerate neighborhood input and to reduce fully connected noise, the heterogeneous key neighbor set is further defined as the scale-matching score. Select the highest score A cross-type object:

[0188]

[0189] in, Indicates before selection The set of elements corresponding to the maximum values; The preset number of neighbors is used; the key interaction subjects are known from the definition. Must be included in Therefore, the edge set of a heterogeneous graph can be written as: .

[0190] For homogeneous candidate sets (and (A set of candidate subjects of the same type), defining the target subject. With cross-type candidate subjects Similarity of state variables during the execution of demand response:

[0191]

[0192] in, For state quantity similarity; For the target subject The state quantities during the execution of demand response can be measured by measuring instruments such as sensors installed on the user-side main body; For cross-type candidate subjects The state quantities during the execution of demand response can be measured by measuring instruments such as sensors installed on the user-side main body; It is the Euclidean norm; This is the similarity scale parameter.

[0193] Based on state quantity similarity From a homogeneous candidate set Select the most similar A set of homogeneous cooperative neighbors. And obtain the set of edges of the homogeneous graph. This step provides a concise and crucial graph structure input for feature aggregation in step S5.1.

[0194] It's important to note that bigraph structures include heterogeneous graphs and homogeneous graphs. The physical definition of heterogeneous and homogeneous graphs is a "set of neighbors," where neighbors form the vertices of the graph, and connections form the edges. Therefore, a bigraph structure is essentially... and heterogeneous key neighbor set As a heterogeneous graph, a set of homogeneous cooperative neighbors As a homogeneous graph.

[0195] S5. Within the dual-graph structure of all user-side subjects, the hybrid competitive and cooperative relationship between adjacent user-side subjects is described based on the hybrid graph attention mechanism to obtain the hybrid competitive and cooperative relationship features. Subject preference embedding vectors are constructed based on the hybrid competitive and cooperative relationship features. The preference embedding vectors are input into the preference conditionalization policy network to obtain action distribution parameters. The preference embedding vectors are input into the value network to obtain preference conditionalization state values.

[0196] The specific implementation process of step S5 is as follows:

[0197] S5.1, Dual-graph structure in all user-side main components and Internally, a hybrid competitive-cooperative relationship is described based on a hybrid graph attention mechanism, resulting in hybrid competitive-cooperative relationship features. Specifically:

[0198] Mapping the principal state variables to a unified latent space:

[0199]

[0200] in, For the target subject The state variables during demand response are mapped to vector representations in a latent space that can be recognized by the neural network; and All of these are learnable neural network parameters; For the target subject The state variables during the execution of demand response. The target entity. Key heterogeneous interaction subjects State variables are mapped to a unified latent space:

[0201]

[0202] For the heterogeneous key neighbor set determined in step S4 Construct attention scores:

[0203]

[0204] in, For the target subject Cross-type candidate subjects Attention scoring, cross-type candidate subjects As the target entity Heterogeneous neighbor entities; and All of these are learnable parameters of the neural network; It is the hyperbolic tangent function; This involves concatenating vectors. For transpose; For the target subject Key heterogeneous interaction subjects The latent space representation. For heterogeneous graphs in a bigraph structure, i.e., heterogeneous key neighbor sets. Normalization is performed to obtain the attention weights:

[0205]

[0206] in, This represents the attention weight.

[0207] Competitive features are obtained by aggregating heterogeneous key neighbors based on the Local Attention Vector (LAV).

[0208]

[0209] in, For the target subject The heterogeneous competition characteristic; For the target subject Key heterogeneous interaction subjects The implicit space representation.

[0210] homogeneous The set of nearest neighbors forms a homogeneous cooperative neighbor set. Based on the Global Attention Vector (GAV), homogeneous graphs in a bi-graph structure, i.e., sets of homogeneous cooperative neighbors, are analyzed. Aggregate the data to obtain collaborative features:

[0211]

[0212] in, For the target subject The homogeneous collaborative characteristics; Let be the cardinality of the set.

[0213] target subject The latent space representation Target subject heterogeneous competition characteristics With the target subject Homogeneous collaboration characteristics By performing hierarchical splicing and fusion, a hybrid competitive and collaborative feature is obtained:

[0214]

[0215] in, It is characterized by a hybrid competitive and cooperative approach; For converged networks; For the splicing operator. The resulting hybrid competitive-cooperative features. This serves as input features for subsequent policy and value networks. Hybrid competitive and cooperative features. The strategy and value network will be passed as the core input to step S5.2.

[0216] S5.2. Construct subject preference embedding vectors based on the characteristics of hybrid competition and cooperation relationships. Input the preference embedding vectors into the preference-conditionalized policy network to obtain action distribution parameters, and input the preference embedding vectors into the value network to obtain preference-conditionalized state values. Specifically:

[0217] For the target subject Preference parameters Constructing the embedding map:

[0218]

[0219] in, Embedded vectors for subject preferences; For learnable embedding functions; For the target subject Preference parameters, Pick , or .

[0220] Embedding preferences into vectors Input the preference-conditional policy network, and the preference-conditional policy network outputs action distribution parameters:

[0221]

[0222] in, For parameters The strategy network; The mean of the action distribution; The standard deviation of the action distribution; The standard deviation is the logarithm. For the target subject The observation vector; It is characterized by a hybrid competitive and cooperative approach; Embedded vectors for subject preferences; For the splicing operator, a Gaussian strategy is adopted, and the action range is constrained by hyperbolic tangent:

[0223]

[0224] in, The main action; It is the hyperbolic tangent function; For Hadamard product; For noise; A Gaussian distribution with zero mean and unit covariance; It is the identity matrix; and All are outputs of the policy network.

[0225] The preference-embedded vector is input into the value network, and the value network outputs preference-conditionalized state values:

[0226]

[0227] in, For parameters Value network; For preference-conditional state values; For the target subject State variables during the execution of demand response; It is characterized by a hybrid competitive and cooperative approach; The subject's preference embedding vector.

[0228] S6. Using the action distribution parameters and preference-conditional state values ​​obtained in step S5, and the preference-conditional multi-objective reward obtained in step S3, and combining the proximal policy optimization reinforcement learning algorithm, train the preference-conditional policy network and the value network respectively until convergence, and obtain the final policy parameters of the preference-conditional policy network and the final policy parameters of the value network corresponding to each subject in the user-side subject set; use the converged final policy parameters to perform preference switching and collaborative decision output in the online operation phase.

[0229] The specific implementation process of step S6 is as follows:

[0230] S6.1. Using the action distribution parameters and preference-conditional state values ​​obtained in step S5, and the preference-conditional multi-objective reward obtained in step S3, the preference-conditional policy network and the value network are trained separately using a proximal policy optimization reinforcement learning algorithm until convergence. This yields the final policy parameters of the preference-conditional policy network and the final policy parameters of the value network for each subject in the user-side subject set. Specifically:

[0231] First, calculate the timing difference error:

[0232]

[0233] in, This refers to timing difference error; For instant rewards, take , or ; Discount factor; value network output and ,in, To conditionalize state values ​​for preferences, for The preference-conditional state value at time step 1. This difference error is processed based on Generalized Advantage Estimation (GAE):

[0234]

[0235] in, The difference error is the difference after the advantage estimation; To cut off the step size; The attenuation coefficient of GAE; This refers to the time-series difference error. The importance ratio is defined as:

[0236]

[0237] in, Importance ratio; This is the current strategy; This is the old strategy; These are parameters for the old strategy; The main action; as the main body The observation vector; It is characterized by a hybrid competitive and cooperative approach; Let be the preference parameter. Further, the objective of the PPO pruning strategy is:

[0238]

[0239] in, For the cropping target; For expectation operators; For finding the minimum value operator; For truncation operators; This is the cutting factor; Importance ratio; This is the difference error after advantage estimation. The value function loss is defined as:

[0240]

[0241] in, For the loss of value; For expectation operators; For preference-conditional state values; For discount return estimates; It is a square operator.

[0242] During training, preference parameters are sampled for each round or time window. Based on the observations constructed in steps S1 to S5, the dual-map hybrid features, and preference embeddings are used to generate actions and obtain trajectories through interaction with the environment. And apply the strategy parameters according to equations (29) to (33). With value parameters Perform iterative updates until convergence.

[0243] S6.2. Utilize the final policy parameters obtained from training convergence to output preference switching and collaborative decision-making during the online execution phase. Specifically:

[0244] Fixed training parameters during the runtime phase According to the scheduling target, set the preference parameters in step S8. The above steps generate and execute the actions of each entity, thereby obtaining the hierarchical demand response collaborative decision-making results, including but not limited to: the peak-shaving-valley compensation pricing sequence of the aggregator. The charging and discharging power sequence of shared energy storage operators And its transaction-related decisions, as well as user load scheduling and response volume sequences. By changing The value can be set without changing the network parameters, thus achieving a switchable trade-off between economy, comfort and device health.

[0245] In summary, this invention presents a multi-agent collaborative decision-making method for demand response on the user side based on Switchable Preference Dual Graph Reinforcement Learning (SPDRL). This method targets aggregators, shared energy storage operators, and users. By introducing switchable preference parameters simultaneously at the reward and policy layers, this invention achieves continuous adjustability among multiple objectives, including economy, comfort, and energy storage device health. It constructs a dual-graph structure of homogeneous and heterogeneous graphs and uses a Scale Matched Key Counterpart Selection (SMKC) mechanism to filter cross-type key interaction objects among the user-side agents in demand response. Local Attention Vector (LAV) and Global Attention Vector (GAV) are used to aggregate the competitive and collaborative features among agents during demand response execution, forming a hybrid competitive-collaborative feature that is input into the preference-conditionalized policy network. Finally, a proximal policy optimization algorithm is employed. Policy Optimization (PPO) completes centralized training and implements distributed execution of demand response strategies for each user-side entity, thereby obtaining a hierarchical demand response collaborative decision-making scheme that can switch strategy behavior through preference parameters during the runtime phase.

[0246] The following describes the specific implementation of the switchable preference dual-graph reinforcement learning method described in this invention, using a reproducible hierarchical demand response simulation scenario. This implementation is geared towards a user-side collaborative decision-making scenario, with participating entities including aggregators, shared energy storage operators, and users possessing various flexible resources. These user-side flexible resources include at least one or more of the following: air conditioning load, electric vehicle charging load, load that can be reduced, load that can be transferred, and distributed power sources. The upstream power grid is not modeled as a decision-making entity in this invention; external signals such as time-of-use pricing are provided as exogenous environmental inputs.

[0247] (1) Establish a hierarchical demand response system consisting of an aggregator set, a shared energy storage operator set, and a user set, and describe the behavior of the subjects in each set as a partially observable Markov decision process to establish a behavioral model of each subject set.

[0248] In this embodiment, the time index is set as follows: ,in, This represents the number of time steps within a single runtime cycle. The duration of each time step is... Aggregator Collection Shared energy storage operators User set For any time step Aggregator The observation vector is denoted as Shared energy storage The observation vector is denoted as ,user The observation vector is denoted as Each main action is recorded as follows: , and Taking time-of-use electricity pricing as an example, the external input signal is denoted as... .

[0249] This invention is based on the collaborative interaction of multiple entities in the same simulation environment. It completes dynamic processes such as incentive pricing, energy consumption adjustment, shared energy storage charging and discharging, and transaction matching at each time step, and achieves controllable switching of multi-objective trade-offs through switchable preference parameters.

[0250] (2) Characterize the interaction behavior and differentiated goals between each set in the hierarchical demand response system, and construct the switchable preference parameters and preference-conditional multi-objective rewards for each set.

[0251] At time step Within, aggregators rely on their observations With preference parameters Provide incentive strategy actions This information is used to generate a compensation unit price or compensation coefficient for the users under its jurisdiction. Shared energy storage operators base their decisions on this information. and Provide information on charging / discharging actions related to transaction prices. User basis Time-of-use electricity pricing Incentives for affiliated aggregators and shared energy storage trading conditions, combined with Give the energy adjustment action .

[0252] To ensure the feasibility of the strategy output, this implementation method constrains key physical boundaries in the environmental dynamics, including but not limited to the upper and lower power limits of various user-side devices, the state of charge constraints and charging / discharging power constraints of shared energy storage, and the limitation of transaction volume by both user surplus / deficit power and available energy of energy storage. This ensures that the state update within any time step is always within the physically feasible region, thereby avoiding training instability and infeasibility caused by approximating hard constraints solely through rewards and penalties.

[0253] (3) Scale matching is performed on the subjects of each set in the hierarchical demand response system to select key interactive objects, so as to obtain a set of heterogeneous key neighbors consisting of heterogeneous key neighbors of all subjects and a set of homogeneous cooperative neighbors consisting of homogeneous cooperative neighbors of all subjects, thereby constructing a dual graph structure for each subject.

[0254] To characterize the experience transfer relationships among similar entities and the competitive coupling relationships among dissimilar entities in demand response collaborative decision-making, this implementation constructs a dual-graph structure consisting of a homogeneous graph and a heterogeneous graph. The heterogeneous graph is used to depict the goal conflicts and marginal constraints between cross-type entities in incentive pricing, transaction pricing, and load adjustment, while the homogeneous graph is used to depict the collaborative transfer and experience sharing among similar entities under similar conditions.

[0255] To select key interaction objects that are representative of the target subject in heterogeneous graphs, scale-matched key interaction object selection is introduced. First, interaction strength weights are defined. , and For any target subject With cross-type candidate subjects Define scale matching score Furthermore, in heterogeneous candidate sets The key interaction object with the best scale match is selected. To avoid missing cross-type information due to selecting only a single object, this implementation constructs a key interaction set by type grouping. Let the heterogeneous type set be... For each heterogeneous type candidate set Select: , ,in, Indicates heterogeneous type Below and target subject The key interactive subject with the most suitable scale; The operator that takes the maximum value of the corresponding independent variable; This is the set of heterogeneous key interactions after SMKC filtering. It is obtained through equation (18). It will be used as a heterogeneous neighbor set in subsequent attention aggregation, thereby achieving a closed-loop connection between the selection of key interaction objects and the extraction of competitive features.

[0256] For homogeneous graphs, select cooperative neighbors of the same type based on observed similarity. Define similar target subjects. With cross-type candidate subjects The observed similarity is in accordance with Select the one with the highest similarity A group of neighbors of the same type constitute This is used for subsequent collaborative feature aggregation.

[0257] (4) Based on the hybrid graph attention mechanism, the hybrid competitive and cooperative relationship between heterogeneous key neighbor sets and homogeneous cooperative neighbor sets is described.

[0258] To integrate self-information, heterogeneous competition information, and homogeneous cooperation information within a unified representation space, this implementation method performs a linear mapping on the observation vectors to obtain a latent space representation. In heterogeneous competitive aggregation, a Local Attention Vector (LAV) is introduced. This vector is used to target the main body. Its heterogeneous key interaction set Any cross-type candidate entity within Computational attention ,Will Normalization yields attention weights And aggregate heterogeneous competitive features based on attention weights. .

[0259] In homogeneous collaborative aggregation, a Global Average Vector (GAV) is introduced, based on a set of homogeneous neighbors. Characteristics of Convergent Cooperativeness By hierarchically splicing and fusing self-features, competitive features, and collaborative features, a hybrid competitive-collaborative feature is obtained. This enables the selection of heterogeneous key interaction objects and the extraction of competitive and collaborative features, i.e., the output of SMKC. It is directly used for LAV attention aggregation, thereby ensuring that the selection of key objects plays a practical role in subsequent representation learning.

[0260] (5) Construct subject preference embedding vectors based on the characteristics of hybrid competitive and cooperative relationships, and input the preference embedding vectors into the preference conditionalization policy network and the value network respectively to obtain action distribution parameters and preference conditionalization states.

[0261] To achieve switchable decision-making under different multi-objective trade-offs using the same strategy, this invention introduces SPDRL. For any objective subject... Define preference parameters And map it to a preference embedding vector :

[0262] Target subject The policy network and value network take observations, dual-graph hybrid features, and preference embeddings as joint inputs, and are represented as follows:

[0263]

[0264] Formula (35) is the expression for the policy network; where, as the main body The strategy function, These are the learnable parameters of the policy network; For the target subject The value function, These are the parameters of the policy network; The main action; symbol This indicates sampling according to a strategy distribution.

[0265] To ensure that preference switching is recognizable to policy learning, this implementation method also makes reward calculation recognizable to policy learning. Conditionalization. Based on any subject Taking the multi-objective component return as an example, let the economic correlation component be denoted as... The comfort or health-related components are Then, the conditionalized immediate reward is defined as:

[0266]

[0267] in, Conditional rewards based on preferences; and These are different target components, which can be selected according to the type of entity. For example, for aggregators, the components are response compliance and compensation costs; for shared energy storage, the components are arbitrage profits and battery health; and for users, the components are compensation profits and comfort loss.

[0268] Through the above steps, the policy output and reward evaluation are conditionally consistent with each other on the preference parameters, thus requiring only adjustments during the inference phase. You can switch policy preferences without retraining.

[0269] (6) Combine the training of the near-end policy optimization reinforcement learning algorithm and use the final policy parameters of the training convergence to perform preference switching and collaborative decision output in the online running stage.

[0270] This implementation uses the PPO algorithm to jointly train multi-agent strategies. For the target agent... trajectory Define the discount reward:

[0271]

[0272] in, For the target subject At any moment Discount rewards; The discount factor is used. Based on the truncated agent objective of PPO, the final set of network parameters for each agent's policy is obtained through training. and the corresponding value network parameter set During the inference execution phase, for a given preference parameter... With real-time observation It can compute hybrid features and preference embeddings of two graphs and output the subject's actions. This leads to a joint decision-making outcome encompassing aggregator incentive strategies, shared energy storage charging and discharging and trading strategies, and user-side energy consumption adjustment strategies. This joint decision-making outcome drives environmental state updates at each time step, ultimately resulting in a coordinated demand response adjustment effect and a multi-objective trade-off effect throughout the entire lifecycle.

Claims

1. A multi-agent collaborative decision-making method for demand response on the user side based on switchable preference dual-graph reinforcement learning, characterized in that, Includes the following steps: Establish a hierarchical demand response system, which is a set of user-side entities consisting of aggregators, shared energy storage operators, and users; Based on the hierarchical demand response system, the behavior of each user-side subject is described as a partially observable Markov decision process; a demand response behavior model of the user set is established for the partially observable Markov decision process of each user-side subject; Establish a comfort characterization model for user sets to quantify the comfort loss of demand response behavior models; construct a shared energy storage model for shared energy storage operator sets; construct a time-of-use compensation pricing model for aggregator sets based on the demand response behavior model of user sets; and characterize the demand response peak-shaving target gap of hierarchical demand response systems based on the demand response behavior model of user sets. Construct switchable preference parameters, and use these switchable preference parameters to define the preference-conditional multi-objective rewards for the established comfort characterization model, the shared energy storage model, the time-of-use compensation pricing model, and the demand response peak shaving target. Based on the real-time physical state quantities of each subject output by the demand response behavior model, the shared energy storage model, and the time-of-use compensation pricing model, the interaction intensity weight is calculated and the key interaction object is selected by scale matching. Based on the selected key interaction objects, the heterogeneous key neighbors and homogeneous cooperative neighbors of each subject are screened from the user-side subject set to form a heterogeneous key neighbor set and a homogeneous cooperative neighbor set. Based on the heterogeneous key neighbor set and the homogeneous cooperative neighbor set, a dual-graph structure is constructed for each user-side subject. Within the dual-graph structure of all user-side subjects, the hybrid competitive and cooperative relationship between adjacent user-side subjects is described based on the hybrid graph attention mechanism, resulting in hybrid competitive and cooperative relationship features; Based on the characteristics of hybrid competitive and cooperative relationships, a subject preference embedding vector is constructed. The preference embedding vector is then input into the preference conditionalization policy network to obtain action distribution parameters. The preference embedding vector is then input into the value network to obtain preference conditionalization state values. By utilizing action distribution parameters, preference-conditional state values, and preference-conditional multi-objective rewards, combined with a proximal policy optimization reinforcement learning algorithm, preference-conditional policy networks and value networks are trained separately until convergence. This yields the final policy parameters of the preference-conditional policy network and the final policy parameters of the value network for each subject in the user-side subject set. The converged final policy parameters are then used to output preference switching and collaborative decision-making during the online operation phase.

2. The demand response user-side multi-agent collaborative decision-making method based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, The construction of switchable preference parameters, using these parameters to define the preference-conditional multi-objective rewards for the established comfort characterization model, the shared energy storage model, the time-of-use pricing model, and the demand response peak-shaving target, includes: Introducing preference parameters It also allows preference parameters to be switched during runtime; Preference parameters are set for aggregators, shared energy storage operators, and users respectively. These preference parameters are then used to define preference-conditional multi-objective rewards, as detailed below: Aggregator The reward is defined as: ； in, Rewards for aggregators; These are the aggregator's preference parameters; Weighting for multi-objective rewards; This represents the total response volume within the aggregated region. For aggregators At any moment Compensation expenditures; In response to the gap; The average level of inappropriateness in the jurisdiction; Shared energy storage operators The reward is defined as: ； in, Incentives for shared energy storage operators; These are the preference parameters for shared energy storage operators; Weighting for multi-objective rewards; As an indicator of energy storage profitability; It is in a charged state; For reference state of charge; Energy storage capacity; This represents the maximum energy storage capacity. To take the absolute value; This is a power stress penalty term; users The reward is defined as: ； in, As a reward for users; For user preference parameters; Weighting for multi-objective rewards; To compensate for the losses; Electricity expenses; This is inappropriate.

3. The user-side multi-agent collaborative decision-making method for demand response based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, The calculation of interaction intensity weights and scale-matching selection of key interaction objects based on the real-time physical state quantities of each entity output by the demand response behavior model, shared energy storage model, and time-of-use compensation pricing model includes: A scale-matching key interaction object selection mechanism is introduced to define interaction intensity weights for different types of subjects: ； in, Indicates user At any moment Interaction strength weight; Indicates user Net load; Indicating shared energy storage operators Interaction strength weight; Indicating shared energy storage operators At any moment The charging and discharging power; Indicating shared energy storage operators The maximum charging and discharging power; Indicates aggregator At any moment Interaction strength weight; Indicates aggregator The sum of responses within the aggregated range; For the target subject With cross-type candidate subjects Define the scale matching score: ； in, Indicates the target subject With cross-type candidate subjects Scale matching score between; For the target subject Interaction strength weight; For cross-type candidate subjects Interaction strength weight; For scale parameters; It is an exponential function; Based on the target subject With cross-type candidate subjects Scale matching score between From heterogeneous candidate sets Select heterogeneous key neighbor sets from the data. To construct a heterogeneous graph edge set, the key interaction objects in the heterogeneous graph edge set. Defined as the representative object with the highest score: ； in, For the target subject Key heterogeneous interaction subjects; This represents the independent variable that corresponds to the maximum value. The heterogeneous key neighbor set is defined according to the target subject. With cross-type candidate subjects Scale matching score between Select the highest score A cross-type object: ； in, Indicates before selection The set of elements corresponding to the maximum values; Given a preset number of neighbors; the edge set of a heterogeneous graph can be written as: ; For homogeneous candidate sets Define the target entity With cross-type candidate subjects Similarity of state variables during the execution of demand response: ； in, For state quantity similarity; For the target subject State variables during the execution of demand response; For cross-type candidate subjects State variables during the execution of demand response; It is the Euclidean norm; For similarity scale parameters; Based on state quantity similarity From a homogeneous candidate set Select the most similar A set of homogeneous cooperative neighbors. And obtain the set of edges of the homogeneous graph. .

4. The user-side multi-agent collaborative decision-making method for demand response based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, Within the dual-graph structure of all user-side entities, the hybrid competitive-cooperative relationship between adjacent user-side entities is described based on a hybrid graph attention mechanism, resulting in hybrid competitive-cooperative relationship features, including: Mapping the principal state variables to a unified latent space: ； in, For the target subject The state variables during demand response are mapped to vector representations in a latent space that can be recognized by the neural network; and All of these are learnable neural network parameters; For the target subject State variables during the execution of demand response; target subject Key heterogeneous interaction subjects State variables are mapped to a unified latent space: ； For heterogeneous key neighbor sets Construct attention scores: ； in, For the target subject Cross-type candidate subjects Attention scoring, cross-type candidate subjects As the target entity Heterogeneous neighbor entities; and All of these are learnable parameters of the neural network; It is the hyperbolic tangent function; This involves concatenating vectors. For transpose; For the target subject Key heterogeneous interaction subjects Latent space representation; for heterogeneous key neighbor sets Normalization is performed to obtain the attention weights: ； in, Attention weights; Competitive features are obtained by aggregating heterogeneous key neighbors based on local attention vectors. ； in, For the target subject The heterogeneous competition characteristic; For the target subject Key heterogeneous interaction subjects The implicit space representation; homogeneous The set of nearest neighbors forms a homogeneous cooperative neighbor set. Based on global attention vectors for homogeneous cooperative neighbor sets Aggregation is performed to obtain collaborative features: ； in, For the target subject The homogeneous collaborative characteristics; The cardinality of the set; target subject The latent space representation Target subject heterogeneous competition characteristics With the target subject Homogeneous collaboration characteristics By performing hierarchical splicing and fusion, a hybrid competitive and collaborative feature is obtained: ； in, It is characterized by a hybrid competitive and cooperative approach; For converged networks; For splicing operators.

5. The user-side multi-agent collaborative decision-making method for demand response based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, The process of constructing a subject preference embedding vector based on the characteristics of a hybrid competitive and cooperative relationship, inputting the preference embedding vector into a preference-conditionalized policy network to obtain action distribution parameters, and inputting the preference embedding vector into a value network to obtain preference-conditionalized state values ​​includes: For the target subject Preference parameters Constructing the embedding map: ； in, Embedded vectors for subject preferences; For learnable embedding functions; For the target subject Preference parameters, Pick , or ; Embedding preferences into vectors Input the preference-conditional policy network, and the preference-conditional policy network outputs action distribution parameters: ； in, For parameters The strategy network; The mean of the action distribution; The standard deviation of the action distribution; The standard deviation is the logarithm. For the target subject The observation vector; It is characterized by a hybrid competitive and cooperative approach; Embedded vectors for subject preferences; To construct the splicing operator, a Gaussian strategy is employed, and the action range is constrained by hyperbolic tangent. ； in, The main action; It is the hyperbolic tangent function; For Hadamard product; For noise; A Gaussian distribution with zero mean and unit covariance; It is the identity matrix; and All are outputs of the policy network; The preference-embedded vector is input into the value network, and the value network outputs preference-conditionalized state values: ； in, For parameters Value network; For preference-conditional state values; For the target subject State variables during the execution of demand response; It is characterized by a hybrid competitive and cooperative approach; The subject's preference embedding vector.

6. The user-side multi-agent collaborative decision-making method for demand response based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, The establishment of a demand response behavior model for the user set based on the observable Markov decision process of each user-side subject includes: For users Define user exist The net load power at time t is: ； in, For users exist Net load power at any given time; For users exist Fixed load power at any given time; For users exist Air conditioner power at any given time; For users exist Electric vehicle charging power at any given time; For users exist The load power that can be reduced at any time; For users exist The transferable load power at any given time; For users exist Distributed generation output power at any given time; Set user exist The baseline net load when there is no response is Define the reduced demand response quantity to obtain the expression for the demand response behavior model of the user set: ； in, For users exist real-time demand response time; For users exist Baseline net load when there is no response participation at any given moment; For users exist Net load power at any given time; This is the operator for finding the maximum value.

7. The demand response user-side multi-agent collaborative decision-making method based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, The establishment of a comfort characterization model for the user set, to quantify the comfort loss of the demand response behavior model, includes: The user's indoor temperature is updated using a first-order thermal inertia model: ； in, For users exist The indoor temperature at any given time; For users exist The indoor temperature at any given time; for outdoor temperature at all times; For users The heat exchange coefficient; For users The equivalent influence coefficient of air conditioning power on room temperature change; For users exist Air conditioner power at any given time; Define user exist If the discomfort at any given moment is a non-negative quantity representing the temperature deviation from the comfortable range, then the expression for the comfort characterization model of the user set is as follows: ； in, For users exist Inappropriate timing; and These represent the lower and upper boundaries of the comfortable temperature range, respectively. For users exist The indoor temperature at any given time; To find the maximum value operator; use the comfort characterization model of the user set to quantify the comfort loss of the demand response behavior model.

8. The user-side multi-agent collaborative decision-making method for demand response based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, The aforementioned method for constructing a shared energy storage model for a group of shared energy storage operators includes: For shared energy storage operators The energy storage state of charge update formula is as follows: ； in, For shared energy storage operators exist The state of charge at any given moment; For shared energy storage operators exist The state of charge at any given moment; To be Cut off in interval Inside; For shared energy storage operators The charging efficiency; For shared energy storage operators The discharge efficiency; For shared energy storage operators Rated capacity; For time step; This refers to the charging power component. This refers to the discharge power component. and They are shared energy storage operators exist The upper and lower bounds of the charged state at any given time; Energy storage power; positive values ​​represent charging, and negative values ​​represent discharging. The following constraints must be met: ； in, It is the absolute value; This represents the maximum permissible charge and discharge power.

9. The user-side multi-agent collaborative decision-making method for demand response based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, The time-sharing compensation pricing model for aggregators based on the demand response behavior model of user sets includes: Aggregator At any moment The compensation electricity price is: ； in, To compensate for electricity prices; , , These are peak, flat, and valley compensation electricity prices, respectively. For indicator functions; Time period type; Compensation electricity price The following boundaries must be satisfied: ； in, and These are the lower and upper limits of the compensation electricity price, respectively. To compensate for electricity prices; The expression for the time-sharing compensation pricing model for the aggregator set is as follows: ； in, For aggregators At any moment Compensation expenditures; For summation operators; For user collection; User affiliation; To compensate for electricity prices; For users exist The demand response volume at any given moment.

10. The user-side multi-agent collaborative decision-making method for demand response based on switchable preference dual-graph reinforcement learning according to claim 1, characterized in that, The user set-based demand response behavior model characterizes the demand response peak-shaving target gap of the hierarchical demand response system, including: Suppose the aggregator is in The target response at time t is The response gap of the demand response peak-shaving target of a hierarchical demand response system is defined as: ； in, In response to the gap; For aggregators exist The target response quantity at any given time; This is an operation that sums the responses within the aggregator's jurisdiction; For users exist real-time demand response time; This is the operator for finding the maximum value.

Images