A method for multi-objective economic dispatch of cogeneration system based on deep reinforcement learning assisted operator selection

Abstract

The method for multi-objective economic dispatch of cogeneration system based on deep reinforcement learning assisted operator selection relates to the technical field of dispatch of cogeneration system. In order to solve the problem that the optimal dispatch scheme obtained by the existing method has poor accuracy, a target function is established to minimize the minimum comprehensive operation cost and environmental pollution penalty, a constraint function is established to constrain the two target functions, a plurality of solution sets satisfying the constraint conditions are obtained as a parent population, and each solution set is taken as an individual; the individual in the parent population is processed by using the deep reinforcement learning assisted operator method to generate a child population; the child population and the parent population are combined, and based on the total constraint violation priority principle and the Pareto non-dominated sorting, an individual is selected from the combined population to form a new parent population; according to whether the preset iteration number is met, it is determined whether an individual is selected from the individual in the new parent population as an optimal dispatch scheme or the parent population is updated and iteration is continued. The method is used for obtaining an optimal dispatch scheme.

Description

Technical Field

[0001] This invention relates to the field of scheduling technology for cogeneration systems, and specifically to a multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection. Background Technology

[0002] Against the backdrop of a global energy transition strategy, improving comprehensive energy utilization efficiency and reducing greenhouse gas and pollutant emissions have become core drivers for the development of power systems and integrated energy systems. Combined Heat and Power (CHP) systems, through the cascaded utilization of energy, can simultaneously provide users with electricity and heat. Their comprehensive energy utilization efficiency is far higher than that of traditional distributed energy systems, making them a key technology for achieving energy conservation and emission reduction. However, the economic dispatch of CHP systems is not simply a matter of minimizing costs. Dispatch centers often need to simultaneously consider two significantly conflicting optimization objectives: minimizing system operating costs and minimizing environmental costs associated with pollutant emissions.

[0003] The dual objectives of CHP scheduling transform it into a typical multi-objective optimization problem. Further complicating matters is the fact that, due to the physical binding of heat and electricity production processes, CHP units generally exhibit a strict thermoelectric coupling characteristic of "heat-driven power generation" or "electricity-driven heat generation." Coupled with stringent constraints such as the unit's valve point effect, ramp-up rate limitations, and network power flow, the system's multidimensional decision space (i.e., feasible region) typically exhibits high nonlinearity, nonconvexity, and even highly irregular discrete multi-connected systems. This highly challenging constrained multi-objective optimization problem (CMOP) renders traditional multi-objective solution methods (such as linear weighted methods based on empirical weights or maximum fuzzy satisfaction methods) ineffective. Such dimensionality reduction methods not only overly rely on the decision-maker's subjective preferences, failing to reveal the true trade-off between economic efficiency and environmental friendliness in a single instance, but also easily fall into local optima or miss the global Pareto front when facing complex nonconvex boundaries.

[0004] In recent years, multi-objective evolutionary algorithms (MOEAs) have shown great potential in power and energy system optimization due to their ability to search in parallel within a single cycle and provide a uniformly distributed Pareto optimal solution set. For constrained complex scheduling problems, researchers have further developed constrained multi-objective evolutionary algorithms (CMOEAs). However, a key underlying component determining the performance of CMOEAs—evolutionary operators—is often set to a fixed pattern in existing scheduling algorithm research (such as relying solely on crossover and mutation in genetic algorithms, or differential mutation in differential evolution). According to the "No Free Lunch" theorem, no fixed evolutionary operator can perfectly adapt to the search requirements of the CHP scheduling model at different evolutionary stages and within different feasible regions. For example, when crossing the irregular infeasibility boundaries of the CHP system, differential evolution (DE) operators are often more globally exploratory; while when approaching the true Pareto front, genetic algorithm (GA) operators are more conducive to local depth convergence. Therefore, the lack of an adaptive online operator selection (AOS) mechanism has become a core technical bottleneck restricting the performance breakthrough of existing CMOEAs in complex scheduling of cogeneration.

[0005] In summary, the multi-objective solution methods used in existing cogeneration multi-objective economic scheduling methods rely on subjective preferences and are prone to getting trapped in local optima. Furthermore, the evolutionary algorithm, which uses fixed operators, cannot adapt to the search requirements at different evolutionary stages, resulting in poor accuracy of the optimal scheduling scheme. That is, although the output scheduling scheme satisfies the physical constraints, the operating cost is too high and the emissions are too large, failing to achieve true Pareto optimality. Summary of the Invention

[0006] The purpose of this invention is to address the problem of poor accuracy in finding the optimal scheduling scheme in existing multi-objective economic scheduling methods for cogeneration systems. This invention proposes a multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection.

[0007] A multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection, the method comprising the following:

[0008] Step 1: Establish two objective functions with the goals of minimizing overall operating costs and minimizing environmental pollution penalties, and establish constraint functions to constrain the two objective functions. Obtain multiple sets of solutions that satisfy the constraints, and each set of solutions includes active power, heating power, and active power purchased and sold by the system from the external main grid.

[0009] Step 2: Treat each set of solutions as an individual, and treat multiple sets of solutions as the first set. Parental generation population, The initial value is 1;

[0010] Step 3: Employ deep reinforcement learning auxiliary operators to process the first... Processing individuals in the parent population to generate the second generation Offspring population;

[0011] Step 4, place the first The offspring and parent populations are merged, and a selection is made from the offspring and parent populations based on the principle of prioritizing total constraint violation and Pareto non-dominated ordering. Individuals form a new parental population;

[0012] Step 5, Judgment Does it equal the preset number of iterations? If so, start from the new parent population. Select one individual from the given individuals as the optimal scheduling scheme; otherwise, let... = +1, take the new parent population as the first For the parent generation, proceed to step 3.

[0013] Preferably, in step 3, a deep reinforcement learning auxiliary operator method is used to process the first... Processing individuals in the parent population to generate the second generation The specific process of generational populations is as follows:

[0014] Set the experience pool capacity to If the experience pool capacity is less than a preset value or the random numbers generated by the pseudo-random number generator are greater than the preset greedy utilization rate, an operator is randomly selected from the preset operator set. Based on the calculation method corresponding to the selected operator, the calculation is performed on the... Processing individuals in the parent population to generate the second generation The offspring population; if the experience pool capacity is greater than or equal to a preset value and the random numbers generated by the pseudo-random number generator are less than or equal to a preset greedy utilization rate, calculate the first generation. The state vector of the parent generation is input into a pre-trained deep Q-network, which outputs the cumulative reward value of each operator in the operator set. The operator corresponding to the maximum cumulative reward value is selected, and the calculation method corresponding to that operator is applied to the i-th generation. Processing individuals in the parent population to generate the second generation Offspring population.

[0015] The beneficial effects of this invention are:

[0016] A CHP multi-objective economic dispatch physical model involving complex thermoelectric coupling characteristics was constructed: it comprehensively optimizes the system's power generation operation cost and environmental pollution penalty cost, and strictly involves nonlinear constraints such as the polygonal thermoelectric operation boundary of the unit, ramp rate and network power flow, accurately characterizing the actual operating conditions of the cogeneration system.

[0017] Using deep reinforcement learning auxiliary operators to perform the first Individuals in the parent generation population are processed by this operator, which is not a fixed operator. Therefore, it achieves precise and intelligent scheduling of underlying evolutionary operators (such as GA and DE), breaking through the limitations of traditional fixed operators in finding the best under strong constraints.

[0018] Simulation results demonstrate that the online learning-driven algorithm of DQN can intelligently perceive the evolutionary environment and adaptively switch operators, effectively overcoming the challenges of narrow feasible regions and non-convex extremum traps in CHP scheduling. Compared with existing fixed-operator evolutionary algorithms, the method described in this invention can significantly improve the convergence accuracy and distribution uniformity of multi-objective scheduling solution sets, providing system operators with a high-quality Pareto optimal scheduling scheme that comprehensively balances economic and environmental benefits. Attached Figure Description

[0019] Figure 1 This is a flowchart of a multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0022] Example:

[0023] according to Figure 1 This embodiment describes a multi-objective economic scheduling method for a cogeneration system based on deep reinforcement learning-assisted operator selection. The method includes the following:

[0024] Step 1: Establish two objective functions with the goals of minimizing overall operating costs and minimizing environmental pollution penalties, and establish constraint functions to constrain the two objective functions. Obtain multiple sets of solutions that satisfy the constraints, and each set of solutions includes active power, heating power, and active power purchased and sold by the system from the external main grid.

[0025] Step 2: Treat each set of solutions as an individual, and treat multiple sets of solutions as the first set. Parental generation population, The initial value is 1;

[0026] Step 3: Employ deep reinforcement learning auxiliary operators to process the first... Processing individuals in the parent population to generate the second generation Offspring population;

[0027] Step 4, place the first The offspring and parent populations are merged, and a selection is made from the offspring and parent populations based on the principle of prioritizing total constraint violation and Pareto non-dominated ordering. Individuals form a new parental population. It is a positive integer;

[0028] Step 5, Judgment Does it equal the preset number of iterations? If so, start from the new parent population. Select one individual from the given individuals as the optimal scheduling scheme; otherwise, let... = +1, take the new parent population as the first For the parent generation, proceed to step 3.

[0029] The following defines the form of the objective function for minimizing overall operating cost in step 1:

[0030] (Formula 1),

[0031] In the formula, The system is Total comprehensive operating cost at any time , , These represent the total number of conventional pure generator sets, combined heat and power (CHP) units, and pure heat boilers in the system, respectively. , , They represent the first conventional generator, the first CHP unit, the first Fuel cost function of a pure heat boiler, For the system in The daily operation and maintenance costs of all operating equipment at all times. For the system in The economic settlement costs incurred from constantly exchanging electrical energy with the external main power grid (purchasing or selling electricity).

[0032] Further defining the comprehensive operating cost, it includes the fuel cost of conventional pure generator units, the fuel cost of CHP units, the fuel cost of pure thermal boilers, equipment operation and maintenance costs, and the cost of purchasing and selling electricity between the system and the external main grid.

[0033] Among them, 1. Fuel cost of conventional generator sets :

[0034] (Formula 2),

[0035] In the formula, For the first conventional generator sets in The active power output at all times. , , For the first Basic cost coefficients for fuel consumption of a conventional generator set (quadratic term, primary term, and constant term). , For the first The cost coefficient of the "valve-point effect" of the Taiwanese turbine unit is calculated by adding a sine term to the formula in order to accurately simulate the sudden change in energy consumption and wire pulling phenomenon caused by the sequential opening of the inlet valves of a multi-valve steam turbine. This makes the objective function exhibit strong non-convex characteristics. For the first The minimum active power limit for stable operation of a conventional generator set.

[0036] 2. Fuel cost of CHP unit (Joint quadratic surface function of thermoelectric bivariate):

[0037] (Formula 3),

[0038] In the formula, , For the first Taiwan CHP unit in The active electrical power and thermal power output at different times, , , These are the secondary, primary, and constant fuel cost coefficients for the pure power generation portion of the CHP unit. , The secondary and primary fuel cost coefficients for the pure heat production portion of the CHP unit. The cost factor for thermoelectric cross-coupling. This cross term reflects the physical law that fuel consumption is not independent and deeply coupled in the heat and power distribution when CHP units are simultaneously supplying heat and power.

[0039] 3. Fuel cost of pure heat boiler :

[0040] (Formula 4),

[0041] In the formula, For the first Taiwan pure heat boiler The heat output power at all times, , , For the first Fuel cost coefficients (quadratic, linear, and constant terms) for heat production from a single-heat boiler.

[0042] 4. External network interaction costs :

[0043] (Formula 5),

[0044] In the formula, , For the system in The active power that is constantly purchased and sold from the external main grid. , External mainnet in The time-of-use electricity purchase price and the electricity sales price at any given moment.

[0045] The following defines the form of the objective function for minimizing environmental pollution penalties in step 1:

[0046] Cogeneration systems inevitably emit pollutants when burning fossil fuels. , and Pollutants such as [list of pollutants]. This embodiment multiplies the emissions of multiple pollutants by their corresponding environmental value and penalty tax rate, converting them into a unified environmental penalty cost. :

[0047] (Formula 6),

[0048] In the formula, For the system in The total environmental pollution penalty cost generated at all times The total number of hazardous pollutants to be considered systematically (e.g.) represent , represent wait), , The first The economic value of environmental damage caused by various pollutants (yuan / kg) and the government's punitive environmental tax rate (yuan / kg). , , These are conventional generators, CHP units, and pure heat boilers, respectively, for the first... The emission function of a pollutant (which typically exhibits physical characteristics that increase exponentially or quadratically with the output power of the equipment).

[0049] The constraint functions in step 1 are further defined below to include system power and thermodynamic balance constraints, CHP unit thermoelectric coupling constraints, and conventional physical constraints.

[0050] To ensure the safety and feasibility of the scheduling scheme at the physical level, strict power balance and equipment operating boundaries must be met. Among these, the thermoelectric coupling feasible region constraint of the CHP unit is the biggest challenge in solving this model.

[0051] The system power and thermodynamic balance constraints include:

[0052] 1. Real-time balance between power supply and demand:

[0053] (Formula 7),

[0054] In the formula, For the system in Total electrical load demand at any given time For the system in Active power loss generated during power grid transmission at any given time (usually obtained from power flow calculations).

[0055] 2. Real-time balance between heat supply and demand:

[0056] (Formula 8),

[0057] In the formula, For the system in Total heat load demand at any given time (including industrial steam and residential heating, etc.).

[0058] The thermoelectric coupling constraints of the CHP unit include:

[0059] Unlike conventional generator sets with their independent "one-dimensional" upper and lower power limits, typical extraction-condensing CHP units have a strong physical coupling and constraint relationship between electrical and thermal power. During operation, the upper and lower limits of electrical power dynamically change with the extracted thermal power, manifesting geometrically as a highly irregular polygonal Feasible Operation Region (FOR).

[0060] Within the feasible operating domain, the thermoelectric physics of the unit are safe. Exceeding this domain will lead to turbine blade damage or boiler overpressure. This boundary consists of a set of extremely stringent joint linear inequalities:

[0061] 1. Maximum power boundary:

[0062] (Formula 9),

[0063] In the formula, For the unit operating under pure condensing conditions (zero extraction steam, i.e.) The maximum pure power generation capacity, , Let B be the boundary inflection point and its corresponding thermal and electrical power. This line segment represents the physical process in which electrical power decreases linearly with the increase of steam extraction (thermal power) when the main steam flow rate of the boiler reaches its rated upper limit.

[0064] 2. Back pressure operating boundary:

[0065] (Formula 10),

[0066] In the formula, , This represents the thermal and electrical power of the unit at its maximum permissible heating operating point C. This line segment represents the physical critical line where the unit is limited by the minimum condensate exhaust flow rate, and in order to ensure cooling of the last-stage blades, it is impossible to extract all the steam for heating.

[0067] 3. Minimum power boundary:

[0068] (Formula 11),

[0069] In the formula, The minimum power output required to maintain stable boiler combustion under pure condensing steam conditions (zero steam extraction). , This represents the thermal and electrical power corresponding to the minimum thermal equilibrium inflection point E. This line segment reflects the bottom-line constraint that, under the boiler's minimum stable combustion load condition, the amount of steam used for power generation is further sharply reduced in order to meet the external steam extraction for heating.

[0070] 4. Maximum condensation boundary:

[0071] (Formula 12),

[0072] In the formula, To ensure the unit is supplying heat at full load The minimum permissible electrical power under the corresponding operating condition. This line segment reflects the defense line constrained by the maximum condensing capacity of the exhaust port or the minimum steam inlet flow of the turbine.

[0073] 5. Absolute heat load constraint:

[0074] (Formula 13),

[0075] (Physical meaning: Thermal power cannot be negative and cannot exceed the unit's maximum steam extraction capacity) )

[0076] Traditional fixed-operator evolutionary algorithms (such as GA or DE) often involve mutations or crossovers along the coordinate axes when generating new solutions. However, when faced with the severely skewed and elongated non-convex boundary space described above, traditional operators are prone to throwing newly generated candidate scheduling schemes out of the feasible region, leading to a large number of infeasible solutions. This is the fundamental physical motivation for this embodiment to subsequently introduce deep reinforcement learning (DRL) to dynamically observe the feasibility state of the population and perform adaptive online operator scheduling and switching.

[0077] Among them, conventional physical constraints include:

[0078] In addition to the aforementioned strong coupling constraints of the CHP unit, the system also needs to meet the basic one-dimensional operational constraints of the equipment:

[0079] 1. Absolute output limits of conventional generators and pure heat boilers:

[0080] (Formula 14),

[0081] (Formula 15),

[0082] In the formula, For the first The nameplate of the conventional generator set shows the maximum permissible active power output. , For the first The minimum and maximum heat production power limits for pure heat boilers in order to maintain stable combustion and prevent overpressure explosions.

[0083] 2. Unit dynamic ramp-rate limits:

[0084] Due to the physical inertia of mechanical components such as rotors and the thermal inertia of the boiler furnace, the rate of change of output between adjacent time periods cannot be infinitely large.

[0085] (Formula 16),

[0086] In the formula, , For respectively the first The maximum uphill rate limit and the maximum downhill (slide) rate limit of the unit (applicable to conventional generators, CHP and boilers) are allowed by mechanical and physical characteristics.

[0087] 3. Power grid interconnection capacity limitations:

[0088] Considering the thermal stability limits of overhead lines and the apparent capacity of substation main transformers, the system cannot purchase and sell electricity from the main grid indefinitely.

[0089] (Formula 17),

[0090] (Formula 18),

[0091] In the formula, This is the maximum bidirectional active power capacity that is strictly limited by the thermal limit of the tie line and the capacity of the transformer.

[0092] The physical-mathematical model established in this embodiment clarifies the multidimensional objective space that the optimization algorithm needs to minimize, as well as the irregular constraint space that must be absolutely obeyed. In subsequent algorithms, a "constraint violation function" will be defined to quantify the severity of the scheduling scheme jumping out of the above feasible polygons, and a DRL agent-assisted evolutionary algorithm will be used to efficiently cross these complex boundaries.

[0093] The following further specifies that in step 3, a deep reinforcement learning auxiliary operator method is used for the first... Processing individuals in the parent population to generate the second generation The specific process of generational populations is as follows:

[0094] Set the experience pool capacity to If the experience pool capacity is less than a preset value or the random numbers generated by the pseudo-random number generator are greater than the preset greedy utilization rate, an operator is randomly selected from the preset operator set. Based on the calculation method corresponding to the selected operator, the calculation is performed on the... Processing individuals in the parent population to generate the second generation The offspring population; if the experience pool capacity is greater than or equal to a preset value and the random numbers generated by the pseudo-random number generator are less than or equal to a preset greedy utilization rate, calculate the first generation. The state vector of the parent generation is input into a pre-trained deep Q-network, which outputs the cumulative reward value of each operator in the operator set. The operator corresponding to the maximum cumulative reward value is selected, and the calculation method corresponding to that operator is applied to the i-th generation. Processing individuals in the parent population to generate the second generation Offspring population.

[0095] Among them, operator set for:

[0096] (Formula 19),

[0097] In the formula, It is a crossover and mutation operator for genetic algorithms, including simulated binary crossover (SBX) and polynomial mutation (PM). Its physical optimization significance lies in its strong local exploitation capability when approximating the optimal scheduling scheme. As a differential evolution mutation operator, it uses vector differences between individuals for recombination. Its physical optimization significance lies in the fact that the search step size it generates is highly adaptive and extremely good at dealing with the tilted and narrow feasible regions generated by CHP thermoelectric coupling, which can effectively lead the population to cross infeasible region faults.

[0098] In order for the neural network to accurately perceive the dynamic characteristics of the population in the multi-objective space and the constraint space, the extraction of... , and Construct the first The state vector of the parent generation population The specific process is as follows:

[0099] Calculate the first Average convergence state of the parent generation population Average physical limit exceedance and the average distribution concentration index in the target space Composition of the first The state vector of the parent generation population.

[0100] No. Average convergence state of the parent generation population for:

[0101] (Formula 20),

[0102] In the formula, The smaller the value, the lower the overall economic and environmental costs of the population, and the closer it is to the true Pareto optimal frontier. For the first The number of individuals included in the parent generation population (the total number of candidate scheduling schemes included). For the first The current population at the next iteration To optimize the number of targets, That is, economic costs and environmental costs. The normalized version of the first One objective function value is used to eliminate the influence of the dimensional difference between the amount and the amount of pollution discharged;

[0103] No. Average physical limit of the parent generation population :

[0104] (Formula 21),

[0105] In the formula, For individuals The total constraint violation rate is calculated when a large portion of the population falls outside the CHP thermoelectric polygon (infeasible region). The value is relatively large; it is 0 when all constraints are satisfied.

[0106] No. Average distribution concentration index of parent generation population in the target space :

[0107] (Formula 22),

[0108] In the formula, and The current population is at the th The maximum and minimum values ​​of each objective function. The difference between them represents the spread of the population in terms of economic or environmental dimensions. The wider the spread, the more uniform the distribution. The smaller the better. Therefore, this indicator also follows the logic of "the smaller the better".

[0109] Among them, the individual in Formula 21 Total constraint violation Calculated based on the constraint function:

[0110] (Formula 23),

[0111] In the formula, For each individual (decision variable vector) in the algorithm population, it represents the specific active / reactive and thermal power allocation and scheduling scheme for all units and tie lines within a certain time period. For individuals Total constraint violation, when This indicates that the scheduling scheme is physically absolutely safe and feasible; The larger the value, the more severe the power imbalance or equipment limitations caused by the scheme. For the first The constraint functions of the equations (such as the real-time balance equations for electric and heat power supply and demand in formulas 7 and 8) are obtained by subtracting the absolute values ​​of the differences between the left and right sides of formula 7 and formula 8, respectively. Then, the absolute values ​​of the differences from formulas 7 and 8 are added together. , For the first Inequality constraint functions (such as Equation 9-13). , This represents the total number of equality and inequality constraints in the CHP system operating model. The tolerance threshold (a very small positive real number) is used to determine the power balance because the algorithm is a floating-point search and is designed to tolerate small rounding errors.

[0112] Among them, in the first Before generating a new scheduling scheme, according to - Greedy strategy for selecting operators :

[0113] (Formula 24),

[0114] In the formula, Given the current state of the population, the neural network is in a given state. Below, predict the execution operator The long-term cumulative return that it can bring, This represents the weight and bias parameter matrix of the neural network within DQN. The [0,1] uniformly distributed random number is generated for each decision. This is the greedy utilization rate (a balance coefficient between exploration and utilization). It controls whether the algorithm tends to trust existing experience (choosing the operator with the largest Q value) or to conduct random exploration (testing the potential of other operators).

[0115] Specifically, in the aforementioned multi-objective economic dispatch model for combined heat and power (CHP), the polygonal feasible region of the CHP units and the strongly coupled constraints such as ramping make the decision space of the problem highly non-convex and discrete. Traditional evolutionary algorithms, which use fixed operators (such as single genetic algorithms or differential evolution), are prone to generating a large number of invalid solutions and getting stuck in evolutionary stagnation when traversing these narrow and sloping feasible region boundaries. To address this, this embodiment proposes a constrained multi-objective evolutionary algorithm (DQL-CMOEA) based on deep reinforcement learning-assisted online operator selection. It utilizes a deep Q-network (DQN) to perceive the population evolutionary state and intelligently switch evolutionary operators, breaking through the optimization bottleneck under strong constraints.

[0116] Further specifying below, step 4 also includes:

[0117] According to the Parent generation population calculation The state vector of the parent generation population is calculated based on the new parent generation population. The state vector of the new parent generation population is then calculated based on the first generation. The state vectors of the parent generation and the new parent generation are used to calculate the state vector of the first generation. The reward value of the generation will be the first generation. The state vector of the parent generation population, the first generation The corresponding operator, the first The reward value of the generation and the state vector of the new parent population form an interaction record tuple, which is stored in the experience pool.

[0118] In step 5, if no and let = The +1 range also includes:

[0119] judge If the number of iterations equals the number of network update cycles, then randomly select a preset number of interaction record tuples from the experience pool, calculate the loss function of the samples, and update the weights of the deep Q network using the backpropagation algorithm. If not, do not update the weights of the deep Q network.

[0120] Specifically, the algorithm records every scheduling interaction during its operation. Store in the experience replay pool Every certain number of iterations, a micro-batch of samples is randomly selected to calculate the loss and update the network.

[0121] (Formula 25),

[0122] In the formula, is the mean squared error loss function for neural networks, used to measure the deviation between the network's predicted value and the target value. Algorithms minimize this value through backpropagation (such as the Adam optimizer). , This refers to the set of micro-batches of training samples drawn from the experience replay pool and their size (e.g., 32 or 64 records). Reward Discount Factor This reflects the importance that intelligent agents place on the long-term evolutionary potential of the CHP system, rather than merely focusing on improvements in the current generation. These are the weight parameters of the target network. To ensure training stability, the target network parameters do not change with the current network. Instead of real-time updates, it performs periodic hard copies.

[0123] Specifically, this embodiment proposes an online intelligent operator selection framework based on DRL: for the evolutionary solution process of CHP scheduling, the convergence, feasibility (i.e., constraint violation degree) and diversity of the population are innovatively extracted into a continuous state space for reinforcement learning, and the cross-generational improvement of the overall state of the population is used as a reward signal to construct a DQN auxiliary decision model, which realizes accurate intelligent scheduling of the underlying evolutionary operators (such as GA and DE).

[0124] The following further specifies that in step 4, the first... The offspring and parent populations are merged, and a selection is made from the offspring and parent populations based on the principle of prioritizing total constraint violation and Pareto non-dominated ordering. For each individual, the specific process is as follows:

[0125] Using total constraint violation The formula calculates the total constraint violation rate for each individual after merging the offspring and parent populations, sorts the total constraint violation rates from smallest to largest, and selects the top... Individual.

[0126] In the multi-objective economic scheduling of a combined heat and power (CHP) system, the deep reinforcement learning (DRL) module acts as the agent making sequential decisions, while the constrained multi-objective evolutionary algorithm (CMOEA), which includes evolutionary operators, population evolution mechanisms, and a CHP physical evaluation model, constitutes the environment for agent interaction. The two form a closed loop through a "state-action-reward" mechanism, enabling online adaptive scheduling of the underlying evolutionary operators. The overall solution control flow is detailed below.

[0127] The specific solution process of the algorithm in this embodiment is divided into the following core steps: initialization, state awareness, operator decision-making, physical evolution and evaluation, experience storage and network update.

[0128] Step 1: Initialization & Initial State Detection

[0129] Input all physical parameters of the combined heat and power system, including the electrical and thermal load curves.

[0130] Initialize DQN network parameters and experience replay pool Population size and maximum number of iterations .

[0131] Randomly generate initial scheduling scheme population .

[0132] Preliminary population assessment Extract the initial state .

[0133] Step 2: Intelligent Operator Decision-Making (Action Selection):

[0134] Based on the current state Determine the experience pool Capacity and Greedy strategy:

[0135] like Insufficient or random numbers Random selection operator .

[0136] like Sufficient and If DQN has not yet been built, build it first; if it has already been built, use DQN to predict each operator. Value, select the optimal operator .

[0137] Step 3: Offspring Generation & Evaluation

[0138] Select the operator Effects on the parent population Generate a population of offspring scheduling schemes .

[0139] Substitute the offspring into the cogeneration physical model to calculate the cost of each option. and constraint over-limit .

[0140] Step 4: Environmental Selection

[0141] Merge parent and child generations, and based on the constraint priority principle and Pareto non-dominated sorting, eliminate inferior and severely out-of-bounds solutions, and retain the remaining ones. The optimal solution is to form a new generation of population. .

[0142] Step 5: Detect New State & Calculate Reward

[0143] Extracting new populations status .

[0144] Calculate the immediate reward of the action. .

[0145] Step 6: Experience Storage (Update EP):

[0146] Interaction record tuple Store in the experience replay pool middle.

[0147] Step 7: Termination Check and State Transition

[0148] Check if the maximum number of iterations has been reached. :

[0149] If the condition has been met: the algorithm terminates and outputs the Pareto optimal thermoelectric scheduling scheme set.

[0150] If the network update and loop are not completed:

[0151] Check if the network update cycle has been reached (e.g.) g%50=0 means that the training and parameter update of the Deep Q-Network (DQN) are triggered every 50 generations to ensure that the intelligent agent can regularly absorb new data from the experience pool.

[0152] If achieved, from Mini-batch samples are randomly selected from the DQN data, and backpropagation is used to update the DQN weights. .

[0153] Complete state transition: Let .

[0154] Return to Step 2 to continue with the next generation operator decision.

[0155] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection, characterized in that, The method includes the following: Step 1: Establish two objective functions with the goals of minimizing overall operating costs and minimizing environmental pollution penalties, and establish constraint functions to constrain the two objective functions. Obtain multiple sets of solutions that satisfy the constraints, and each set of solutions includes active power, heating power, and active power purchased and sold by the system from the external main grid. Step 2: Treat each set of solutions as an individual, and treat multiple sets of solutions as the first set. Parental generation population, The initial value is 1; Step 3: Employ deep reinforcement learning auxiliary operators to process the first... Processing individuals in the parent population to generate the second generation Offspring population; Step 4, place the first The offspring and parent populations are merged, and a selection is made from the offspring and parent populations based on the principle of prioritizing total constraint violation and Pareto non-dominated ordering. Individuals form a new parental population; Step 5, Judgment Does it equal the preset number of iterations? If so, start from the new parent population. Select one individual from the given individuals as the optimal scheduling scheme; otherwise, let... = +1, take the new parent population as the first For the parent generation, proceed to step 3.

2. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection as described in claim 1, characterized in that, In step 3, a deep reinforcement learning auxiliary operator method is used to process the first... Processing individuals in the parent population to generate the second generation The specific process of generational populations is as follows: Set the experience pool capacity to If the experience pool capacity is less than a preset value or the random numbers generated by the pseudo-random number generator are greater than the preset greedy utilization rate, an operator is randomly selected from the preset operator set. Based on the calculation method corresponding to the selected operator, the calculation is performed on the... Processing individuals in the parent population to generate the second generation The offspring population; if the experience pool capacity is greater than or equal to a preset value and the random numbers generated by the pseudo-random number generator are less than or equal to a preset greedy utilization rate, calculate the first generation. The state vector of the parent generation is input into a pre-trained deep Q-network, which outputs the cumulative reward value of each operator in the operator set. The operator corresponding to the maximum cumulative reward value is selected, and the calculation method corresponding to that operator is applied to the i-th generation. Processing individuals in the parent population to generate the second generation Offspring population.

3. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection as described in claim 2, characterized in that, Operator set for: ， In the formula, This refers to the crossover and mutation operator in a genetic algorithm. It is a differential evolution mutation operator.

4. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection according to claim 2 or 3, characterized in that, Calculate the first The state vector of the parent generation population is obtained through the following process: Calculate the first The average convergence state, average physical limit exceedance, and average distribution concentration in the target space of the parent population constitute the first... The state vector of the parent generation population.

5. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection according to claim 4, characterized in that, No. Average convergence state of the parent generation population for: ， In the formula, For the first The number of individuals included in the parent generation population. For the first The current population at the next iteration To optimize the number of targets, , The normalized version of the first One objective function value; No. Average physical limit of the parent generation population : ， In the formula, For individuals Total constraint violation rate; No. Average distribution concentration index of parent generation population in the target space : ， In the formula, and The current population is at the th The maximum and minimum values ​​of each objective function.

6. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection according to claim 5, characterized in that, individual Total constraint violation It is calculated based on the constraint function.

7. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection as described in claim 5, characterized in that, Step 4 also includes: According to the Parent generation population calculation The state vector of the parent generation population is calculated based on the new parent generation population. The state vector of the new parent generation population is then calculated based on the first generation. The state vectors of the parent generation and the new parent generation are used to calculate the state vector of the first generation. The reward value of the generation will be the first generation. The state vector of the parent generation population, the first generation The corresponding operator, the first The reward value of the generation and the state vector of the new parent population form an interaction record tuple, which is stored in the experience pool.

8. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection according to claim 7, characterized in that, In step 5, if no and let = The +1 range also includes: judge If the number of iterations equals the number of network update cycles, then randomly select a preset number of interaction record tuples from the experience pool, calculate the loss function of the samples, and update the weights of the deep Q network using the backpropagation algorithm. If not, do not update the weights of the deep Q network.

9. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection according to claim 1, characterized in that, The constraint functions include system power and thermodynamic balance constraints, CHP unit thermoelectric coupling constraints, and conventional physical constraints.

10. The multi-objective economic scheduling method for cogeneration systems based on deep reinforcement learning-assisted operator selection according to claim 9, characterized in that, System power and thermal balance constraints include real-time balance constraints on electrical power supply and demand and real-time balance constraints on thermal power supply and demand; Thermoelectric coupling constraints for CHP units include maximum electrical power boundary, back pressure operating boundary, minimum electrical power boundary, maximum condensation boundary, and absolute heat load constraint. Conventional physical constraints include the absolute limits of output of conventional generators and pure heat boilers, dynamic ramp rate constraints of units, and interactive capacity limits of grid interconnection lines.

Images