FL-DQN-AC fusion reinforcement learning-based multi-photovoltaic access transformer area power optimization distribution method

Abstract

The invention provides an FL-DQN-AC fusion reinforcement learning-based multi-photovoltaic access transformer area power optimal distribution method. The method comprises the following steps of S1, acquiring data of a multi-photovoltaic access transformer area; s2, data preprocessing; s3, constructing a power distribution mathematical model; s4, on the basis of the power distribution mathematical model, constructing a fuzzy logic + deep Q network + Actor-Critic fusion reinforcement learning model; s5, training the fusion reinforcement learning model; s6, verifying the fusion reinforcement learning model; and S7, performing real-time power distribution. According to the method, high precision, strong robustness and real-time performance are taken as targets, a fuzzy logic + deep Q network + Actor-Critic fusion reinforcement learning model is constructed, a closed-loop technical system from data to decision is formed, the problem of power distribution of multiple photovoltaic areas is effectively solved, new energy consumption and power grid energy efficiency improvement are supported, and the method is suitable for large-scale popularization and application. And a technical support is provided for refined dispatching of a transformer area under a novel power system.

Description

Technical Field

[0001] This invention relates to the field of power systems and their automation technology, and in particular to a power optimization allocation method for multi-photovoltaic access areas based on FL-DQN-AC fused reinforcement learning. Background Technology

[0002] In recent years, distributed photovoltaic (PV) power has rapidly increased its penetration rate in distribution network areas due to its clean and renewable characteristics, becoming an important part of energy transition. However, after multiple PV systems are centrally connected to distribution areas, existing power distribution technologies face three major pain points, which restrict PV consumption and grid operation efficiency: 1. Dispatch deviations caused by environmental uncertainties: Photovoltaic output is significantly affected by natural factors such as light intensity and ambient temperature, and load demand also fluctuates suddenly (such as industrial users starting and stopping, and sudden changes in peak electricity consumption in residential areas). Traditional dispatching methods based on deterministic models (such as static programming and PID control) cannot quantify such uncertainties in real time, which can easily lead to photovoltaic output prediction deviations, load matching imbalances, and in turn cause problems such as voltage exceeding limits and line overload. 2. The contradiction between decision precision and continuity: Photovoltaic inverters support continuous power regulation at the 0.1kW level, requiring fine-grained power allocation commands to match their regulation capabilities; however, the decisions output by traditional discrete optimization algorithms (such as dynamic programming and pure DQN) are discrete values, which are difficult to meet the requirements of continuous regulation, while pure continuous strategy algorithms (such as deep deterministic strategy gradient) are prone to getting trapped in local optima under multiple constraints (voltage, line power, photovoltaic output limits) and cannot take into account the global optimization objective. 3. Insufficient multi-objective collaborative optimization capability: Power allocation in distribution areas needs to simultaneously achieve three major objectives: minimizing grid losses, maximizing photovoltaic (PV) absorption, and mitigating load fluctuations. These objectives are coupled and subject to trade-offs (e.g., increasing PV absorption may increase grid losses). Existing single-objective algorithms (such as those that only optimize PV absorption) or simple weighted multi-objective algorithms cannot dynamically balance the priorities of each objective, resulting in low overall operational efficiency. 4. Lack of robustness of traditional algorithms: Existing technologies rely heavily on historical data to train models, which are poorly adapted to extreme scenarios that are not covered (such as sudden rain causing a sharp drop in photovoltaic output or a short-term doubling of load). They are prone to decision failures, requiring manual intervention and adjustments, which increases operation and maintenance costs. Summary of the Invention

[0003] This invention provides a power optimization allocation method for multiple photovoltaic (PV) distribution areas based on FL-DQN-AC fusion reinforcement learning. It constructs a complete technical system for power optimization allocation of multiple PV distribution areas, including "data acquisition, preprocessing, model building, training and verification, and real-time execution". By integrating algorithms, it breaks through the limitations of traditional technologies and realizes intelligent, precise, and safe allocation of power in multiple PV distribution areas, providing technical support for the high penetration rate of distributed energy in the distribution network.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning includes the following steps: S1. Data collection for multiple photovoltaic access areas; S2. Data preprocessing: The collected data is cleaned, normalized, and feature constructed to obtain preprocessed data including state vectors and uncertainty features; S3. Based on the preprocessed data, construct a mathematical model for power allocation; S4. Based on the power allocation mathematical model, a fusion reinforcement learning model of fuzzy logic (FL) + deep Q network (DQN) + Actor-Critic (AC) is constructed; the fuzzy logic module takes uncertainty features as input and outputs uncertainty coefficients; the deep Q network module takes state vectors as input and outputs candidate actions and the corresponding action value; the Actor-Critic module takes state vectors, uncertainty coefficients and action value as input and outputs the final power allocation action. S5. Training of the fusion reinforcement learning model: Using the preprocessed data, the data is divided into training set, validation set and test set according to the proportion. First, the fuzzy logic module is pre-trained, and then the deep Q network module and the Actor-Critic module are jointly trained to obtain the trained fusion reinforcement learning model. S6. Validation of the fusion reinforcement learning model: The trained fusion reinforcement learning model is validated using the validation set and the test set to obtain a valid fusion reinforcement learning model. S7. Real-time power allocation execution: Realizes the data acquisition and preprocessing process to obtain real-time state vectors and real-time uncertainty features, inputs them into a validated fusion reinforcement learning model, and outputs real-time power allocation actions.

[0005] In this specification, the specific process of constructing the fuzzy logic module in S4 includes: defining light intensity fluctuation, temperature fluctuation, and load prediction error as fuzzy input variables, and uncertainty coefficient as fuzzy output variable; dividing fuzzy subsets and representing them using Gaussian membership functions; establishing a fuzzy rule base covering all input combinations based on expert experience; calculating the rule trigger intensity through the max-min inference method, and then obtaining the quantitative uncertainty coefficient by defuzzifying using the centroid method.

[0006] In this specification, the specific process of constructing the deep Q-network module in S4 includes: the module contains a policy network and a target network with consistent structure; the policy network outputs candidate actions and action values, and the target network provides stable target action values; the parameters of the two networks are synchronized every 100 training rounds.

[0007] In this specification, the specific process of constructing the Actor-Critic module in S4 includes: the module contains an Actor network and a Critic network; the Actor takes a state vector as input and outputs an action probability distribution; the Critic takes a state vector as input and outputs a state value; the Actor calculates an advantage function based on the action value of the deep Q-network and the state value of the Critic, and updates the parameters by maximizing the advantage function objective function; the Critic updates the parameters by minimizing the mean square error between the actual and target state values.

[0008] In this manual, the data collected in S1 includes photovoltaic power plant data, distribution network data, load data, and constraint parameter data.

[0009] In this specification, the data preprocessing in S2 specifically includes: cleaning, which includes handling missing values ​​and outliers; normalization, which maps all data to the [0,1] interval; and feature construction, which generates three types of uncertainty features: light intensity fluctuation, temperature fluctuation, and load prediction error, to obtain the preprocessed data.

[0010] In this specification, the specific process of training the fusion reinforcement learning model in S5 includes: a pre-training stage for the fuzzy logic module, using the uncertainty features generated in S2 as input and the normalized power deviation as a label, minimizing the mean square error of the uncertainty coefficient and the normalized power deviation using gradient descent, iterating until the mean square error is less than 0.005; and a joint training stage for the deep Q-network and Actor-Critic, employing an experience replay mechanism to store the state-action-reward-new state samples generated by the agent's interaction with the environment into an experience replay pool. When the number of samples in the experience replay pool is not less than 64, batch sampling updates the parameters of the deep Q-network and Actor-Critic, testing on the validation set every 500 rounds until the loss function of the deep Q-network is less than 0.01 and the loss function of the Critic network in Actor-Critic is less than 0.008, at which point training stops.

[0011] In this specification, when dividing the training set, validation set, and test set in S5, the ratio is 7:2:1, and the test set includes 30% high-uncertainty scene samples, which are light intensity fluctuations greater than 500W / m². 2 Samples with temperature fluctuations greater than 10℃ or load prediction errors greater than 150kW.

[0012] In this specification, the power allocation mathematical model constructed in S3 is as follows: based on the data preprocessed in S2, an objective function is constructed with the goal of minimizing the total grid loss in the distribution area, maximizing the photovoltaic absorption rate, and smoothing the total load fluctuation in the distribution area. At the same time, photovoltaic output constraints, power balance constraints, voltage constraints, and line transmission constraints are set as constraints.

[0013] In this manual, the qualification criteria for model verification in S6 are: grid loss reduction rate not less than 15%, photovoltaic absorption rate increase not less than 0.08, load fluctuation smoothing rate not less than 20%, constraint satisfaction rate not less than 99%, and decision deviation not greater than 0.5kW. When all verification indicators meet the above criteria, the model verification is deemed qualified.

[0014] In summary, the present invention has at least the following beneficial effects: 1. Significantly improves the ability to adapt to uncertainty: The FL module can convert qualitative uncertainties such as light, temperature and load fluctuations into quantitative coefficients, dynamically correct the reward function, enable the algorithm to perceive environmental changes in real time, avoid scheduling deviations caused by uncertainties in advance, and reduce the occurrence of abnormal situations such as voltage overruns and line overloads. 2. Balancing decision-making accuracy and continuity: DQN provides a value benchmark for discrete actions, ensuring that decisions conform to the global optimization objective; AC outputs a continuous strategy based on this value benchmark, matching the fine adjustment capability of the photovoltaic inverter, solving the problems of insufficient accuracy of traditional discrete algorithms and local optima of pure continuous algorithms, making power allocation commands more in line with the operating characteristics of the equipment; 3. Achieve multi-objective collaborative optimization: The reward function of the fusion model is directly related to multiple objectives such as "grid loss, photovoltaic absorption, and load fluctuation". Through the collaboration of DQN and AC, the priority of each objective is dynamically balanced. While improving the photovoltaic absorption level, the grid loss in the distribution area is reduced, load fluctuation is smoothed, and the overall energy utilization efficiency is improved. 4. Enhance algorithm robustness and power grid security: FL's ability to quantify the uncertainty of extreme scenarios, combined with DQN's experience replay mechanism (covering historical data of multiple operating conditions), enables the model to adaptively adjust to sudden scenarios that have not been trained, reducing the probability of decision failure, reducing the need for manual intervention, and ensuring the long-term stable operation of the power grid in the distribution area. 5. Real-time performance and engineering feasibility: The model is deployed on the edge computing nodes of the distribution area. The entire process of data acquisition, preprocessing, decision-making, and command issuance can be completed in minutes, matching the real-time requirements of power dispatching in the distribution area. At the same time, the algorithm is compatible with existing distribution automation systems (such as the IEC61850 protocol), requiring no large-scale hardware modification and resulting in low engineering implementation costs. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1This is a flowchart illustrating the power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning involved in this invention.

[0017] Figure 2 This is a schematic diagram of the data acquisition and preprocessing process involved in this invention.

[0018] Figure 3 This is a schematic diagram illustrating the process of constructing the fusion model involved in this invention.

[0019] Figure 4 This is a schematic diagram of the real-time power allocation execution process involved in this invention. Detailed Implementation

[0020] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0021] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0022] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] like Figure 1 As shown, this embodiment provides a power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning, which includes the following steps: S1. Data collection for multiple photovoltaic access areas; S2. Data preprocessing: The collected data is cleaned, normalized, and feature constructed to obtain preprocessed data including state vectors and uncertainty features; S3. Based on the preprocessed data, construct a mathematical model for power allocation; S4. Based on the power allocation mathematical model, a fuzzy logic + deep Q network + Actor-Critic fusion reinforcement learning model is constructed; the fuzzy logic module takes uncertainty features as input and outputs uncertainty coefficients; the deep Q network module takes state vectors as input and outputs candidate actions and the corresponding action value; the Actor-Critic module takes state vectors, uncertainty coefficients and action value as input and outputs the final power allocation action. S5. Training of the fusion reinforcement learning model: Using the preprocessed data, the data is divided into training set, validation set and test set according to the proportion. First, the fuzzy logic module is pre-trained, and then the deep Q network module and the Actor-Critic module are jointly trained to obtain the trained fusion reinforcement learning model. S6. Validation of the fusion reinforcement learning model: The trained fusion reinforcement learning model is validated using the validation set and the test set to obtain a valid fusion reinforcement learning model. S7. Real-time power allocation execution: Realizes the data acquisition and preprocessing process to obtain real-time state vectors and real-time uncertainty features, inputs them into a validated fusion reinforcement learning model, and outputs real-time power allocation actions.

[0024] In some embodiments, the specific process of constructing the fuzzy logic module in S4 includes: defining light intensity fluctuation, temperature fluctuation, and load prediction error as fuzzy input variables, and uncertainty coefficient as fuzzy output variable; dividing fuzzy subsets and representing them using Gaussian membership functions; establishing a fuzzy rule base covering all input combinations based on expert experience; calculating the rule trigger intensity using the max-min inference method, and then obtaining the quantitative uncertainty coefficient by defuzzifying using the centroid method.

[0025] In some embodiments, the specific process of constructing a deep Q-network module in S4 includes: the module contains a policy network and a target network with consistent structure; the policy network outputs candidate actions and action values, the target network provides stable target action values, and the parameters of the two networks are synchronized every 100 rounds of training.

[0026] In some embodiments, the specific process of constructing the Actor-Critic module in S4 includes: the module contains an Actor network and a Critic network; the Actor inputs a state vector and outputs an action probability distribution; the Critic inputs a state vector and outputs a state value; the Actor calculates an advantage function based on the action value of the deep Q-network and the state value of the Critic, and updates the parameters by maximizing the advantage function objective function; the Critic updates the parameters by minimizing the mean square error between the actual and target state values.

[0027] In some embodiments, the data collected in S1 includes photovoltaic power plant data, distribution network data, load data, and constraint parameter data.

[0028] In some embodiments, the data preprocessing in S2 specifically includes: cleaning including missing value processing and outlier processing; normalization processing to map all data to the [0,1] interval; and feature construction to generate three types of uncertainty features: light intensity fluctuation, temperature fluctuation, and load prediction error, to obtain the preprocessed data.

[0029] In some embodiments, the specific process of training the fused reinforcement learning model in S5 includes: a pre-training stage of the fuzzy logic module, using the uncertainty features generated in S2 as input and the normalized power deviation as label, minimizing the mean square error of the uncertainty coefficient and the normalized power deviation through gradient descent, and iterating until the mean square error is less than 0.005; a joint training stage of the deep Q network and Actor-Critic, using an experience replay mechanism, storing the state-action-reward-new state samples generated by the interaction between the agent and the environment into the experience replay pool, when the number of samples in the experience replay pool is not less than 64, batch sampling samples are used to update the parameters of the deep Q network and Actor-Critic, and testing is performed on the validation set every 500 rounds until the loss function of the deep Q network is less than 0.01 and the loss function of the Critic network in Actor-Critic is less than 0.008, and training stops.

[0030] In some embodiments, when dividing the training set, validation set, and test set in S5, the ratio is 7:2:1, and the test set contains 30% high-uncertainty scene samples, which are light intensity fluctuations greater than 500W / m². 2 Samples with temperature fluctuations greater than 10℃ or load prediction errors greater than 150kW.

[0031] In some embodiments, the construction of the power allocation mathematical model in S3 specifically involves: based on the data preprocessed in S2, constructing an objective function with the goals of minimizing the total grid loss in the distribution area, maximizing the photovoltaic absorption rate, and smoothing the total load fluctuation in the distribution area, while setting photovoltaic output constraints, power balance constraints, voltage constraints, and line transmission constraints as constraints.

[0032] In some embodiments, the qualification criteria for model verification in S6 are: grid loss reduction rate not less than 15%, photovoltaic absorption rate increase not less than 0.08, load fluctuation smoothing rate not less than 20%, constraint satisfaction rate not less than 99%, and decision deviation not greater than 0.5kW. When all verification indicators meet the above criteria, the model verification is deemed qualified.

[0033] The technical concept of this invention is as follows: This solution proposes a fusion reinforcement learning model combining fuzzy logic (FL), deep Q-network (DQN), and actor-critic (AC). FL handles uncertainty, DQN provides discrete value benchmarks, and AC optimizes continuous strategies, achieving high-precision and robust power allocation decisions. The technical roadmap is as follows: data collection → data preprocessing → mathematical model construction → fusion algorithm design → model training → performance verification → real-time execution. These steps are closely linked, forming a closed loop from data to decision. Details are as follows: S1: Basic Data Collection for Multi-PV Access Areas To support the training and application of the S4 fusion algorithm, four types of data need to be collected (covering the full operational status and environmental characteristics of the entire distribution area). The data collection and preprocessing process is as follows: Figure 2 As shown, the specific content is as follows: 1. Photovoltaic power plant data (Source: Photovoltaic inverter monitoring terminal): Real-time output of a single inverter (kW, sampling frequency 1min); Real-time light intensity of photovoltaic array (W / m) 2 (sampling frequency 1 min) Real-time ambient temperature of photovoltaic array (℃, sampling frequency 1min); Historical average sunshine duration (same season and time period over the past 3 years) With temperature average (For calculating fluctuations, see S4).

[0034] 2. Distribution network data (Source: Distribution Automation System): Real-time voltage at each node (pu, sampling frequency 1min); Real-time transmission power of each line (kW, sampling frequency 1min); Total network loss in the area (kW, calculated by the SCADA system, sampling frequency 5min).

[0035] 3. Load data (Source: Smart meters and load forecasting system): Real-time load of each node (kW, sampling frequency 1min); Forecast load output by the load forecasting system (kW, prediction particle size 15min, used to calculate prediction error).

[0036] 4. Constraint parameter data (Source: Power grid planning documents): Maximum output limit of photovoltaic power station (kW, determined by inverter capacity); Node voltage upper and lower limits , ; Line transmission power limit (kW, determined by the conductor's current carrying capacity).

[0037] S2: Data Preprocessing for Multiple Photovoltaic Access Areas The raw data needs to undergo three steps: cleaning, normalization, and feature construction, to provide high-quality input for the S4 algorithm. The specific steps are as follows: 1. Data cleaning: Missing value handling: For missing data at the 1-minute level (accounting for <5%), linear interpolation is used to fill in the missing data; for data with consecutive missing values ​​>10 minutes, the historical average value is used as the replacement. Outlier handling: Outliers (such as voltage >1.1pu or <0.9pu) are identified using the 3σ criterion and replaced with the average value of the time before and after that moment.

[0038] 2. Normalization process: Map all data to the [0,1] interval (to eliminate the influence of dimensions and improve the convergence speed of the algorithm), the formula is: ; in , For the historical minimum and maximum values ​​of the collected data (e.g.) of , ).

[0039] 3. Feature Construction: The uncertainty features required to generate S4 based on the original data (directly used as FL input): Light intensity fluctuation (W / m) 2 ); Temperature fluctuations (°C); Load forecasting error (kW).

[0040] S3: Construction of Power Allocation Mathematical Model 3.1 Objective Function (related to the S4 reward function) With "minimum grid loss, highest photovoltaic absorption rate, and smoothest load fluctuation" as the multi-objective optimization goals, the mathematical expression is: ; in: Total network loss (kW) in the distribution area is calculated as the sum of the losses of all lines. ( For line current, (Line resistance); Photovoltaic grid integration rate (unitless), defined as the ratio of actual photovoltaic power absorbed to total power output: ; Total load fluctuation in the distribution area (kW) is defined as the absolute value of the difference between the current load and the load at the previous moment. ; Weighting coefficients: (Network loss has the highest priority) (Consumption rate is the second highest) (Finally, load fluctuations) were assessed and determined by five power grid dispatching experts using the Analytic Hierarchy Process (AHP).

[0041] 3.2 Constraints (related to the S4 action space) Power allocation decisions must satisfy four types of physical constraints (S4's action space must strictly adhere to these constraints): 1. Photovoltaic power output constraint: The allocated power of each photovoltaic power station shall not exceed its maximum output. (i=1,2,...,n, covering all n photovoltaic power stations within the distribution area); This represents the actual power allocation value for the i-th photovoltaic power station.

[0042] 2. Power balance constraint: The total power supplied by the distribution area (grid + photovoltaic) must equal the total load. ;in The input power to the grid (kW, which can be positive or negative; a negative value indicates the power fed back to the grid from the distribution area). It is the real-time power consumption of the j-th load.

[0043] 3. Voltage Constraints: The voltage at each node must be within the allowable range. (j=1,2,...,m); Obtained through power flow calculations, it is directly related to photovoltaic power distribution: when photovoltaic output increases, the voltage at the connection point rises, and m is the total number of all nodes in the distribution area.

[0044] 4. Line transmission constraints: The transmission power of each line must not exceed the thermal stability limit. (k=1,2,...,l); Let be the actual transmission power of the k-th line, and there are l transmission lines in the entire distribution area.

[0045] S4: Construct a fusion reinforcement learning model of "DQN+AC+FL" In multi-PV grid interconnection areas, power allocation decisions face three major challenges: first, environmental uncertainties caused by random fluctuations in sunlight, temperature, and load (such as a sudden drop in PV output due to rain); second, the matching problem between the adjustment accuracy of PV inverters and discrete decision-making (balancing the feasibility of the strategy with the optimization accuracy); and third, the synergistic satisfaction of multi-objective optimization (grid loss, absorption rate, load fluctuation) and grid constraints (voltage, line power). To address these challenges, this step integrates three algorithms—Deep Q-Network (DQN), Actor-Critic (AC), and Fuzzy Logic (FL)—on the original reinforcement learning framework to construct a fusion model: "FL handles uncertainty → DQN estimates action value → AC optimizes strategy update." This model achieves high-precision and robust power allocation decisions through close interaction between algorithms (such as FL output correcting DQN rewards and DQN value guiding AC strategy). The fusion model construction process is as follows: Figure 3 As shown.

[0046] 4.1 Basic Definition of Core Elements The subscript "_D" indicates that it is related to DQN, the subscript "_A / _C" indicates that it is related to AC, and the subscript "_F" indicates that it is related to FL.

[0047] 4.1.1 State Space (State, s) The state space is the "input window" for the algorithm to perceive the operating status of the transformer substation. It must completely contain all key variables that affect power allocation decisions and is defined as a 1×(4n+m+l+3) vector (where n is the number of photovoltaic power stations, m is the number of nodes in the transformer substation, and l is the number of lines). Its specific structure is as follows: 1. Photovoltaic power plant parameters (n variables): Real-time maximum output of each photovoltaic power plant (i=1,2,...,n, unit: kW), taken from preprocessed real-time monitoring data, representing the maximum power that the photovoltaic power station can currently provide; 2. Distribution area load parameters (m variables): Real-time power load of each node in the distribution area. (j=1,2,...,m, unit: kW), taken from pre-processed load monitoring terminal data, reflecting users' real-time electricity demand; 3. Node voltage parameters (m variables): Real-time voltage of each node in the distribution area. (j=1,2,...,m, unit: pu), taken from preprocessed voltage sensor data, and must meet the voltage constraint of S3 (0.95-1.05pu). 4. Line transmission parameters (1 variable): Real-time transmission power of each line in the transformer area. (k=1,2,...,l, unit: kW), taken from preprocessed line monitoring data, and must meet the line thermal stability limit constraint of S3; 5. Environmental uncertainty parameters (3 variables, inputs to the FL algorithm): Light intensity fluctuation (Unit: W / m) 2 Real-time light intensity Compared with the historical average sunshine during the same period The absolute value of the difference, i.e. , Taken from historical data statistics; Temperature fluctuations (Unit: °C): Real-time ambient temperature Compared with the historical average temperature for the same period The absolute value of the difference, i.e. , Taken from historical data statistics; Load forecasting error (Unit: kW): Real-time load Compared with the predicted load The absolute value of the difference, i.e. , Provided by the transformer area load forecasting system.

[0048] 4.1.2 Action Space (Action, a) The action space is the power allocation decision output by the algorithm, defined as a 1×n vector, representing the actual power allocation for each photovoltaic power station. (i=1,2,...,n, unit: kW). This decision must strictly satisfy the photovoltaic output constraint of S3: (Ensure that the real-time maximum output of the photovoltaic power station is not exceeded). To match the adjustment accuracy of the photovoltaic inverter (typically supporting 0.1kW-level fine-tuning), the discretization granularity of the action space is set to 0.1kW (i.e., each...). The value interval is 0.1kW).

[0049] 4.1.3 Environment (E) The environment is a "virtual stage" simulating the operation of the transformer substation, responsible for receiving actions output by the algorithm and calculating new states and rewards. This solution's environment comprises two core modules: Physical rules module: Based on the S3 power allocation mathematical model, after inputting action 'a' (each photovoltaic power), it calculates the grid power supply through power balance constraints. Update node voltages through power flow calculations With line transmission power This eventually generates a new state s'; FL Uncertainty Processing Module: Receives three uncertain variables from state s ( Uncertainty coefficients are calculated using fuzzy logic. This provides a basis for correcting the reward function.

[0050] 4.2 Construction of the fusion algorithm (DQN+AC+FL) The three algorithms do not operate independently, but are deeply integrated through a "data input-result transmission-decision correction" link: FL transforms uncertainty into quantitative coefficients to correct the rewards of DQN and AC; DQN outputs the value benchmark of actions to guide the strategy optimization of AC; AC outputs more refined continuous decisions based on the value benchmark, ultimately forming a power allocation scheme that balances robustness and accuracy.

[0051] 4.2.1 Algorithm 1: Fuzzy Logic (FL) – Uncertainty Quantification Module Core Function: During the operation of a photovoltaic (PV) distribution area, uncertainties such as sudden changes in sunlight (e.g., cloud cover), rapid temperature increases (affecting PV conversion efficiency), and sudden load surges (e.g., industrial user startup) can cause PV output or load to deviate from expectations, thereby rendering power allocation decisions ineffective. The role of the FL (Power Filter) is to transform these qualitative uncertainties into quantitative uncertainties. By modifying the reward function, DQN and AC can still make robust decisions when the environment fluctuates.

[0052] 4.2.1.1 FL Model Construction Process The construction of the FL model involves four steps: "defining fuzzy variables → designing membership functions → establishing a fuzzy rule base → inference and defuzzification". Each step is designed based on the actual operating characteristics of the transformer substation. 1. Definition of fuzzy variables Fuzzy variables include inputs (sources of uncertainty) and outputs (uncertainty coefficients). Their universe of discourse (range of values) needs to be determined based on S2 historical data statistics to ensure coverage of more than 99% of real-world scenarios. Input variable 1: Light intensity fluctuation Domain of discourse [0, 800] W / m 2 (In the historical data of S2, the maximum light fluctuation is 780W / m) 2 (taking 800 as the upper limit), it is divided into 3 fuzzy subsets: small (S, 0-300W / m 2 ), medium (M, 200~600W / m) 2 Large (L, 500~800W / m) 2 (The overlapping portion of the subset is used for smooth transitions); Input variable 2: Temperature fluctuation The domain of discourse is [0,15]℃ (in the historical data of S2, the maximum temperature fluctuation is 14.2℃, and 15 is taken as the upper limit), which is divided into 3 fuzzy subsets: small (S, 0-5℃), medium (M, 3-10℃), and large (L, 8-15℃). Input variable 3: Load forecasting error The domain of discourse is [0,200]kW (in the historical data of S2, the maximum load error is 195kW, and 200 is taken as the upper limit), which is divided into 3 fuzzy subsets: small (S, 0-60kW), medium (M, 40-140kW), and large (L, 120-200kW). Output variable: Uncertainty coefficient The domain of discourse [0,1] (0 represents no uncertainty, 1 represents extreme uncertainty) is divided into 3 fuzzy subsets: low (Lw, 0-0.4), medium (Md, 0.3-0.7), and high (Hh, 0.6-1.0).

[0053] 2. Membership Function Design Membership functions are used to quantify the degree to which a variable belongs to a certain fuzzy subset (values ​​range from 0 to 1). Considering the continuous distribution characteristics of environmental uncertainties (such as light fluctuations within 300 W / m²), 2 When the variable belongs to both "small" and partially to "medium", the Gaussian membership function (which has good smoothness and is suitable for continuous variables) is used, and the formula is: (4-1) in: : The membership degree of variable x to a certain fuzzy subset (e.g. W / m 2 Membership degree belonging to "small (S)"); The center value of the fuzzy subset (the typical value of the subset, determined by S2 historical data clustering); : Standard deviation of fuzzy subset (controls the width of the function; the larger the value, the wider the coverage of the subset).

[0054] The membership function parameters for each variable are as follows (based on S2 data statistics and expert experience calibration): : Small (S): ; Middle (M): ; Large (L): ; : Small (S): ; Middle (M): ; Large (L): ; : Small (S): ; Middle (M): ; Large (L): ; : Low (Lw): ; Middle (Md): ; High (Hh): .

[0055] 3. Establishment of a fuzzy rule base (27 complete rules) Fuzzy rules are the mapping relationship between inputs and outputs, formulated based on the experience of five power system experts, covering all input combinations (3×3×3=27 rules) to ensure the completeness of the rules. Each rule is formatted as "If...". A and For B and If it is C, then "D" is the correct answer, as follows:

[0056] 4. Fuzzy Reasoning and Defuzzification Reasoning is the process of calculating output based on rules, while defuzzification is the process of converting fuzzy output into quantitative values. Reasoning (Maximum-Minimum Method): For each rule, calculate the trigger strength. (k=1~27), ​​which is the minimum membership degree of all input variables in the rule (the rule is only triggered when all input conditions are partially satisfied): (4-2) For example, if the current (The membership degree of "S" is 0.3, and the membership degree of "M" is 0.6.) (The membership degree of "S" is 0.1, and the membership degree of "M" is 0.8.) (The membership degree of "S" is 0.9, and the membership degree of "M" is 0.2), then rule 2 (S,S,M) .

[0057] Defuzzification (centroid method): A weighted sum of the trigger strengths of all rules and the center values ​​of the output subsets is obtained to obtain a quantitative result. : (4-3) in Output the k-th rule The center value of the corresponding subset (such as in rule 1) Rule 27 ).

[0058] 4.2.1.2 FL Model Training Process The training objective of the FL model is to optimize the membership function parameters. ), making the output It can accurately reflect the impact of actual uncertainties on power allocation. The training data and process are as follows: Training dataset: 10,000 historical samples were selected from S2, each sample containing (Input), and the corresponding actual power deviation. (output labels, To contribute to actual photovoltaic power, (For the expected output based on the S3 model). Objective function: Minimize With normalization Mean square error (MSE): (4-4) in kW (S2 historical maximum deviation) is used to... Normalized to [0,1] (and) Consistency of domain of discourse); Training method: Gradient descent is used for updating. Learning rate Iterate 5000 times until... (After testing, at this time) The correlation with the actual deviation reached 0.92.

[0059] 4.2.1.3 FL Model Application Process In real-time power allocation, the FL model operates according to the following steps: Extract from the current state s W / m 2 , ℃ kW; Calculate the membership degree of each input variable (e.g.) Belonging to "S" "M" ); Calculate the trigger strength of the 27 rules using formula (4-2) (e.g., rule 13). ); 4. Solve the fuzziness using formula (4-3) to obtain... ; 5. The output is fed into the reward function (see 4.3.2) to correct the decisions of DQN and AC.

[0060] 4.2.2 Algorithm 2: Deep Q-Network (DQN) – Action Value Estimation Module Core Function: In complex power allocation scenarios, it is necessary to determine the long-term benefits (such as reducing grid losses and improving grid integration) of choosing a particular photovoltaic power allocation scheme. The role of DQN is to estimate the value (Q-value) of "state-action" pairs through a deep neural network, providing a benchmark for the AC algorithm to determine whether the action is worthwhile. The -greedy strategy balances "exploring new solutions" with "utilizing known optimal solutions" to avoid getting trapped in local optima.

[0061] 4.2.2.1 DQN Model Construction Process The core of the DQN model is a dual-network structure consisting of a "policy network" and a "target network," which improves stability by separating value estimation and policy updates. Policy Network ( ): Function: Input state s, output the Q value of all possible actions a ( The larger the value, the higher the long-term benefit of action a in state s. Structure: Input layer (number of nodes = state space dimension) → Hidden layer 1 (64 nodes, ReLU activation) → Hidden layer 2 (64 nodes, ReLU activation) → Output layer (number of nodes = action space dimension, linear activation). parameter: The weights and biases of the network (optimized through training).

[0062] Target network ( ): Function: Provides a stable target Q value (avoiding value fluctuations during policy network updates); Structure: Completely identical to the policy network, but with different parameters. Not updated in real time, each Round training synchronizes from the policy network ( ).

[0063] 4.2.2.2 DQN Model Training Process DQN training is based on an "experience replay" mechanism, which optimizes network parameters using historical interaction data. The specific process is as follows: 1. Experience replay pool initialization: Create capacity The replay pool D is used to store samples of "state-action-reward-new state". ( (For the rewards after merging FL, see 4.3.2). 2. Iterative training (10,000 rounds): Step 1: Status Input and Action Selection Initial state sampled from S5 training set ,based on -greedy strategy selects action (Balancing exploration and utilization): (4-5) in The initial value is 0.9 (90% probability of exploring new actions), which decreases by 5% every 500 rounds, down to a minimum of 0.1 (10% probability of exploration, ensuring stable use of the optimal strategy in the later stages).

[0064] Step 2: Environmental Interaction and Sample Storage Will Input environment, calculate new state using S3 model Combined with FL output Calculate rewards (Formula 4-13), will Store in D.

[0065] Step 3: Batch Sampling and Target Q-Value Calculation when At that time, 64 samples were randomly selected, and the target Q value was calculated. (Current rewards + discounts on future rewards): (4-6) in (Discount factor, determined by expert evaluation: the reward in the next 15 minutes has a 90% impact on the current decision).

[0066] Step 4: Update policy network parameters Minimize the mean squared error between the current Q value and the target Q value (loss function) Update via Adam optimizer : (4-7) Learning rate (The value decreases by 10% every 1000 rounds, with a minimum of 0.001, to avoid fluctuations in the later stages).

[0067] Step 5: Target network synchronization Every 100 training rounds, the policy network parameters are... Copy to target network ( ).

[0068] 2. Convergence criterion: When there are 100 consecutive rounds of convergence... Training is stopped and data is saved when the average Q-value fluctuation on the S5 validation set is less than 1%. .

[0069] 4.2.2.3 Application Process of DQN Model In real-time decision-making, DQN provides a value benchmark for AC by following these steps: 1. Enter the current status (Includes real-time data such as photovoltaic output, load, and voltage); 2. Calculate the Q-values ​​of all possible actions (satisfying the S3 constraints) using the policy network: ; 3. Output the action with the largest Q value. and the corresponding Q value ; 4. It is passed to the AC algorithm to calculate the advantage function (see 4.2.3.3).

[0070] 4.2.3 Algorithm 3: Actor-Critic (AC) – Policy Optimization and Update Module Core Function: The actions output by the DQN are discrete (0.1kW granularity), which may lead to insufficient adjustment precision (e.g., the optimal power should be 125.3kW, but the DQN can only output 125.2kW or 125.3kW). The AC function constructs an "advantage function" (quantifying the relative value of actions) based on the Q value of the DQN, and optimizes the output of a more refined power allocation scheme through continuous strategy, while ensuring that the decision conforms to the multi-objective optimization objective of S3.

[0071] 4.2.3.1 AC Model Construction Process AC consists of "Actor (policy network)" and "Critic (value network)," which work together: Actor Network ( ): Function: Outputs the probability distribution of choosing action a in state s (continuous policy). The higher the probability, the more likely the action is to be the optimal solution. Structure: Consistent with the DQN input and hidden layers, the output layer uses the Softmax activation function (ensuring that the sum of the probabilities of all actions is 1); parameter: Network weights and biases (independent of DQN) ).

[0072] Critic Network ( ): Function: Estimate the fundamental value of state s (independent of specific actions) to provide a benchmark for the advantage function; Structure: Input layer (state space dimension) → Hidden layer 1 (64 nodes, ReLU) → Hidden layer 2 (64 nodes, ReLU) → Output layer (1 node, linear activation, outputs state value) ); parameter: For network weights and biases (independent) ).

[0073] 4.2.3.2 AC Model Training Process AC training is performed concurrently with DQN, sharing the experience replay pool D, and the value output of DQN is correlated through the advantage function: 1. Initialization: Initialization It retains the D playback pool D of DQN; 2. Iterative training (10,000 rounds): Step 1: Sample collection 64 samples were drawn from D. Get the corresponding FL .

[0074] Step 2: Critic Network Training (Update) ) Calculate the target state value (Current reward + discount on future state value): (4-8) in (with DQN) Maintain consistency and ensure uniform time discounts.

[0075] Minimize Critic loss (MSE of current state value versus target value): (4-9) Learning rate (Below DQN to ensure stable value estimation).

[0076] Step 3: Actor Network Training (Update) ) Calculate the advantage function (Core interaction steps: Subtract the Critic's state value from the DQN action value to quantify the relative advantage of the action): (4-10) The action value output by DQN is directly used as the input to AC, realizing a close connection between the two algorithms.

[0077] Maximize the Actor objective function (Increase the probability of actions with a large advantage function): (4-11) Update using gradient ascent method Learning rate .

[0078] 3. Convergence criterion: When there are 100 consecutive rounds of convergence... ,and If the growth rate is less than 1%, stop training and save. .

[0079] 4.2.3.3 Application Process of AC Model In real-time decision-making, the AC outputs the final power allocation scheme based on the Q value of DQN: 1. Enter the current status Calculate state value using Critic network ; 2. Obtain the Q-values ​​of all actions output by DQN. Calculate the advantage function for each action: (4-12) 3. Output action probabilities through the Actor network. ; 4. Select the action that maximizes the product of the advantage function and the probability as the final decision. : (4-13) This action combines high value and high probability, balancing optimization and stability.

[0080] 4.3 Core of Algorithm Integration: Collaborative Decision-Making and Reward Function 4.3.1 Three-Algorithm Collaborative Decision-Making Process (Data Interaction Link) The three algorithms achieve deep integration through a closed loop of "input-processing-output-feedback", with the specific interaction chain as follows: 1. Environment → FL: The environment will introduce the uncertainties in state s ( Input FL, FL output ; 2.FL→DQN / AC: Input the reward function to adjust the Q value and the advantage function of AC in DQN (the higher the uncertainty, the stronger the reward adjustment). 3. Environment → DQN: The environment inputs the complete state s into DQN, and DQN outputs the action. and its Q value ; 4. DQN→AC: As the benchmark for calculating the AC dominance function (Formula 4-10), it guides AC to identify "relatively better" actions; 5. AC → Environment: AC outputs the final action The environment is calculated based on the S3 model to determine the new state s' and reward R; 6. Environment → Replay Pool: "The samples are stored in the replay pool D to provide data for the next round of training of DQN and AC, forming a closed loop."

[0081] 4.3.2 Fusion Reward Function (R) – The Link for Algorithmic Collaboration The reward function is the core of the three-algorithm synergy, and it must simultaneously reflect the multi-objective optimization of S3, the uncertainty correction of FL, and the constraint penalty of S3. The formula is as follows: (4-14) The calculation process and physical meaning of each parameter are as follows: 1.F: S3's multi-objective optimization function (completely identical to S3): (4-15) in The power loss (kW) is the network loss in the distribution area. Photovoltaic grid integration rate (unitless). For load fluctuations (kW). (Determined by AHP method, see S3). The smaller F is, the better the solution is, therefore -F is taken in the reward function (so that the better solution gets a higher reward).

[0082] 2. Uncertainty correction term (the correlation between FL and reward): As a reward for uncertainty (to compensate for deviations caused by environmental fluctuations): (4-16) in (kW, This represents the expected power of the S3 model. (This refers to the actual power). The larger the value (the more severe the deviation). The smaller (penalty bias); The larger it is (the higher the uncertainty). right The higher the sensitivity (enhanced correction).

[0083] 3. Constraint penalty terms (corresponding to S3 constraints): If the action violates any constraint (such as...) ),but (A fixed value, determined through S5 simulation testing: this value is sufficient to prevent the algorithm from violating constraints); If all constraints are satisfied, then .

[0084] 4.4 Definition of Intelligent Agent The intelligent agent is the physical carrier of the fusion algorithm, integrating the computational logic of FL, DQN, and AC, with the following specific functions: 1. Data reception: Receive the current state s (including photovoltaic, load, voltage, uncertainties, etc.) from the environment; 2. Uncertainty handling: Call the FL module and input... Output ; 3. Value estimation: Call the DQN module, input s and Output action and its Q value ; 4. Strategy Optimization: Call the AC module and input... Output the final action ; 5. Decision Implementation: Output to the environment to complete power allocation decisions; 6. Experience storage: [This refers to storing] " "The samples are stored in the replay pool D for algorithm iteration and updates."

[0085] This step addresses the limitations of single algorithms in uncertainty handling, decision accuracy, and multi-objective optimization by constructing a fusion model of "FL+DQN+AC". Specifically, FL... Modifying the reward function and using the Q-value of DQN to guide the calculation of the advantage function of AC are the core interaction points for algorithm collaboration, ensuring that the three algorithms form an organic whole rather than independent modules.

[0086] S5: Fusion Model Training Process 5.1 Training Dataset Partitioning The data preprocessed using S2 (approximately 500,000 samples over 12 months) was divided into training, validation, and test sets in a 7:2:1 ratio. Training set (350,000 records): used to update algorithm parameters (FL membership function, DQN and AC network weights); Validation set (100,000 records): Used to adjust hyperparameters (such as learning rate and discount factor) to avoid overfitting; Test set (50,000 records): Simulates real-world scenarios to evaluate the final performance of the model.

[0087] 5.2 Phased Training Strategy Perform the training in the order of "FL pre-training → DQN and AC joint training" to ensure that each module converges before merging them. 5.2.1 FL Module Pre-training Objective: Optimize the membership function parameters ,make It accurately reflects actual uncertainties.

[0088] Input: from the training set ; Tags: Normalized power deviation (See S4.2.1.2); Training process: Minimize using gradient descent. (Formula 4-4), iterate 5000 times, verify once every 100 iterations, when the verification set... Stop when the time is right and save the optimal parameters.

[0089] 5.2.2 Joint Training of DQN and AC Objective: To enable DQN to accurately estimate action value with its Q-value, and for AC's policy output to conform to the multi-objective optimization objective.

[0090] Initialization: DQN policy network With the target network Random initialization; AC Random initialization; experience replay pool D is cleared; Training process: 1. Randomly select states from the training set DQN-based -greedy strategy selects action (Formula 4-5); 2. Call the pre-trained FL module to calculate Based on the objective function and constraints of S3, the reward is calculated according to formula (4-14). ; 3. Store in D, when At that time, batch sampling updates DQN and AC: DQN: Calculate the target Q value using formula (4-6) and minimize it. (Formula 4-7) Update Synchronization every 100 rounds ; AC: Calculate the target state value according to formula (4-8) and minimize it. (Formula 4-9) Update Then, based on the Q-value of DQN, calculate the dominance function (Equation 4-10) to maximize it. (Formula 4-11) Update ; 4. Test on the validation set every 500 rounds. If DQN's... And AC's Stop training and save the model parameters.

[0091] 5.3 Training Hyperparameter Settings All hyperparameters were determined through a grid search on the validation set to ensure a balance between model convergence and performance.

[0092] S6: Model Performance Verification 6.1 Design of Verification Indicators We designed metrics based on both optimization effectiveness and robustness to compare the performance of the fusion model and single algorithms (pure DQN, pure AC): 1. Optimize performance metrics: Network loss reduction rate: ; Increase in photovoltaic grid integration rate: ; Load fluctuation mitigation rate: .

[0093] 2. Robustness indicators (assessing the ability to adapt to uncertainty): Constraint satisfaction rate: ; Decision bias: ( (This is the theoretically optimal value calculated offline).

[0094] 6.2 Verification Results and Analysis In the test set (including scenarios with 30% high uncertainty, such as a sudden drop in illumination of 200W / m), 2 The verification results on the device are as follows:

[0095] S7: Real-time power allocation decision execution (the real-time power allocation execution process is as follows) Figure 4 (As shown) 7.1 Hardware Deployment Architecture The fusion model is deployed on edge computing nodes in the distribution area (industrial-grade server, CPU: Intel Xeon E5-2690, memory: 32GB), and achieves real-time decision-making through the following links: 1. Data Acquisition Layer: Photovoltaic inverters, smart meters, and voltage sensors upload real-time data (1-minute granularity) to edge nodes via 4G / fiber optics; 2. Preprocessing layer: Edge nodes run the S2 preprocessing algorithm to generate state vector s and uncertainty features ( ); 3. Decision Layer: Loads the pre-trained model from S5 and outputs power allocation decisions. ; 4. Execution Layer: Edge nodes will use the IEC61850 protocol to... The data is sent to the photovoltaic inverter to control its output.

[0096] 7.2 Real-time decision-making process (corresponding to S4 collaborative logic) 1. Data input (time t): Edge nodes receive real-time data, preprocess it, and then generate a status. .

[0097] 2. Quantification of Uncertainty: Call the FL module and input... Calculate according to formula (4-1)-(4-3) .

[0098] 3. Value estimation and strategy optimization: DQN module input Calculate the Q-value for all actions and output it. and ; AC module input Calculate the advantage function (Formula 4-12) and action probability, and output the final decision according to Formula (4-13). .

[0099] 4. Constraint Validation and Execution: Edge node verification If the constraints of S3 (formulas 3-2 to 3-5) are met, the information is sent to the inverter; if not, an emergency adjustment is triggered (calling the suboptimal action of DQN).

[0100] 5. Feedback Update: At time t+1, collect the new state. Compared with actual rewards The data is stored in the local playback pool of the edge node (uploaded to the cloud every 24 hours for regular model retraining).

[0101] 7.3 Operational Support Mechanism 1. Model retraining: The model is fine-tuned weekly using newly collected data (approximately 10,000 data points) (learning rate reduced to 0.0001) to adapt to seasonal changes (such as higher light intensity in summer). 2. Fault redundancy: If an edge node fails, it automatically switches to the traditional PID control strategy to ensure basic power distribution function; 3. Communication encryption: Data transmission uses AES-256 encryption to prevent decision-making instructions from being tampered with.

[0102] Solution Summary This solution systematically addresses the power allocation challenge across multiple photovoltaic substations by employing a fusion reinforcement learning model combining FL, DQN, and AC. 1. Innovation: Combining fuzzy logic with deep reinforcement learning, it achieves for the first time the synergy between uncertainty quantification and continuous policy optimization; 2. Practicality: The model is deployed on edge nodes and completes a decision within 1 minute, meeting real-time requirements; 3. Effectiveness: Test results show that compared with the single algorithm, the fusion model improves the network loss reduction rate by more than 30% and the constraint satisfaction rate reaches 99.7%.

[0103] In some embodiments, the core challenges of existing multi-PV power allocation, besides environmental uncertainty, conflicting decision-making accuracy, and multi-objective optimization, also include the challenge of finding the global optimum under complex constraints: traditional reinforcement learning algorithms are prone to getting trapped in local optima in high-dimensional action spaces (e.g., 10 PV power plants, each with a 0.1kW granularity corresponding to 1000 combinations), leading to long-term high grid losses or insufficient PV absorption. Introducing a quantum genetic algorithm (QGA), through qubit encoding and quantum rotation gate operations, provides powerful global search capabilities, which can compensate for the local optimization deficiencies of reinforcement learning.

[0104] After introducing the quantum genetic algorithm (QGA), it is upgraded to a four-algorithm fusion model of "quantum genetic algorithm (QGA) + fuzzy logic (FL) + deep Q network (DQN) + Actor-Critic (AC)". Through the collaborative logic of "QGA global optimization → FL uncertainty quantification → DQN value estimation → AC fine optimization", the model achieves the unity of global optimization and local fine decision-making.

[0105] Constructing a fusion reinforcement learning model of "QGA+FL+DQN+AC" The subscript "_Q" indicates that it is related to QGA; Quantum bit encoding: ( For complex amplitudes, satisfying , representing the quantum state of the i-th gene); Population size: (The number of candidate solutions to be optimized simultaneously in QGA); Quantum rotation angle: (Used to update the rotation gate parameters of the j-th qubit of the i-th individual); Fitness function: (Indicators for evaluating the quality of QGA candidate solutions, related to the S3 objective function).

[0106] Algorithm 4: Quantum Genetic Algorithm (QGA) – Global Optimal Search Module Core function: Rapidly search for globally optimal candidate solutions in a high-dimensional action space, providing a high-quality initial action set for DQN and AC, and preventing reinforcement learning from getting trapped in local optima. QGA, through its quantum parallel computing characteristics, explores multiple potential optimal solutions simultaneously, making it particularly suitable for multi-peak optimization problems such as photovoltaic power allocation (where different photovoltaic combinations may correspond to similar short-term rewards, but with significant differences in long-term returns).

[0107] QGA model construction process 1. Quantum Coding Design The power of each photovoltaic power station is encoded using qubits to define the action space. Corresponding to a quantum chromosome, the length is (Determined by the discretization granularity, such as 0-100kW or 0.1kW granularity requiring 1000 states,) The bits are sufficient to cover, because ).

[0108] Individual coding example: For two photovoltaic power plants, the quantum chromosome is... The first 10 bits correspond to The last 10 bits correspond to .

[0109] 2. Definition of fitness function Using the S3 objective function F as the core, and combining it with constraint penalties, a fitness function is defined (the larger the value, the better the solution): ; Where x represents the candidate action (power allocation scheme) generated by QGA. The constraint penalty term is set to 1000 if the constraint is violated, otherwise it is set to 0 to ensure that suboptimal solutions are eliminated.

[0110] 3. Quantum Revolving Door Update Population evolution is achieved by adjusting the amplitude of qubits through rotation angle. Determined by fitness difference: ; in (Step size coefficient) For two individuals, It is a symbolic function (ensuring evolution towards higher fitness).

[0111] 4. Measurement and Variation Operations Measurement: For each quantum individual, according to The probability collapses into a binary solution (e.g.) There is an 80% probability that the value will be "1", which is then decoded into the actual power value. ; Mutation: Perform NOT gate operation on the qubit with a 5% probability ( ), to maintain population diversity.

[0112] QGA model training process 1. Initialization: Randomly generated Each quantum individual, each qubit (Equal probability superposition state); 2. Iterative optimization (100 generations): Step 1: Measure all individuals and decode into candidate actions. (k=1...50); Step 2: Calculate each fitness ; Step 3: Select the top 20% of individuals by fitness as parents, and update the remaining individuals through the quantum rotation gate; Step 4: Perform mutation operations to avoid premature convergence; 3. Convergence criterion: If the optimal fitness fluctuation is less than 0.1% for 10 consecutive generations, output the optimal candidate action set. (Top 10 optimal solutions).

[0113] QGA Model Application Process In real-time decision-making, QGA acts as a "pre-filter" to output high-quality candidate actions: Enter the current status In (Determine the upper limit of power); Run QGA for 10 generations and generate 10 candidate actions that satisfy the constraints. ; 3. This is passed to DQN as a subset of its action space (reducing the search range of DQN and improving efficiency).

[0114] Four-algorithm fusion logic (adding interaction between QGA and other algorithms) Data interaction link QGA→DQN: QGA outputs a set of candidate actions. DQN only needs to compute the Q-value within this subset (reducing the action space dimension from...). Reduced to 10, significantly improving efficiency); FL→DQN / AC / QGA: Output of FL At the same time, the evaluation indicators for the three were revised: Modify the fitness function of QGA: (The higher the uncertainty, the more aggressive the solution is penalized.) Modify the reward function of DQN / AC; DQN→AC: Based on The calculated Q-value guides the AC dominance function; AC → Final Action: AC in Continuous optimization is performed within the neighborhood (e.g., QGA output is 125.2kW, AC is fine-tuned to 125.3kW).

[0115] Collaborative decision-making example (specific scenario) With 3 photovoltaic power stations ( For example, (kW): QGA generates candidate action sets (10 solutions); 2. FL Calculation (Medium uncertainty), adjust QGA fitness and input the reward function; 3. DQN in Select the action with the highest Q value. ; 4. AC based on Calculate the advantage function and fine-tune it to obtain... (Meets 0.1kW accuracy).

[0116] Fusion model training process Phased training strategy 1. QGA pre-training: Input: 1000 typical states from S2 historical data (covering peak / flat / valley load scenarios); Objective: Train a QGA for each state so that it converges to the vicinity of the global optimum within 10 generations; Output: Saves the rotation angle parameters of the QGA and the population initialization strategy.

[0117] 2. FL pre-training: Outputs the optimized membership function.

[0118] 3. Joint training of four algorithms: Step 1: QGA is generated for the current state. ; Step 2: DQN in Select an action, FL calculation ; Step 3: The environment returns a reward R, and AC optimizes the action based on the Q value; Step 4: Store the sample in the playback pool, update the DQN and AC parameters synchronously, and fine-tune the rotation angle step k of QGA every 500 rounds.

[0119] Model Performance Validation

[0120] Real-time decision-making process QGA candidate solution generation: Receive state Then, QGA generates 10 candidate actions within 2 seconds (through GPU-accelerated quantum computing). 2. Subsequent process: FL quantifies uncertainty → DQN evaluates Q value → AC fine optimization → constraint verification → execution decision (total time <10 seconds, meeting the 1-minute real-time requirement).

[0121] Solution Summary (QGA's Core Contributions) The introduction of quantum genetic algorithms solves the global search problem in high-dimensional space in traditional reinforcement learning, through four-layer collaboration with FL, DQN, and AC: Efficiency Improvement: From the perspective of motion space dimension Reducing it to 10 decreases decision-making time by 60%; 2. Enhanced robustness: Under extreme scenarios such as sudden drops in light intensity, the constraint satisfaction rate increases from 99.7% to 99.9%; 3. Optimization results: The annual grid loss was further reduced by 5.2%, and the photovoltaic absorption rate increased to 98.6%.

Claims

1. A power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fused reinforcement learning, characterized in that, Includes the following steps: S1. Data collection for multiple photovoltaic access areas; S2. Data preprocessing: The collected data is cleaned, normalized, and feature constructed to obtain preprocessed data including state vectors and uncertainty features; S3. Based on the preprocessed data, construct a mathematical model for power allocation; S4. Based on the power allocation mathematical model, a fuzzy logic + deep Q network + Actor-Critic fusion reinforcement learning model is constructed; the fuzzy logic module takes uncertainty features as input and outputs uncertainty coefficients; the deep Q network module takes state vectors as input and outputs candidate actions and the corresponding action value; the Actor-Critic module takes state vectors, uncertainty coefficients and action value as input and outputs the final power allocation action. S5. Training of the fusion reinforcement learning model: Using the preprocessed data, the data is divided into training set, validation set and test set according to the proportion. First, the fuzzy logic module is pre-trained, and then the deep Q network module and the Actor-Critic module are jointly trained to obtain the trained fusion reinforcement learning model. S6. Validation of the fusion reinforcement learning model: The trained fusion reinforcement learning model is validated using the validation set and the test set to obtain a valid fusion reinforcement learning model. S7. Real-time power allocation execution: Realizes the data acquisition and preprocessing process to obtain real-time state vectors and real-time uncertainty features, inputs them into a validated fusion reinforcement learning model, and outputs real-time power allocation actions.

2. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, The specific process of constructing the fuzzy logic module in S4 includes: defining light intensity fluctuation, temperature fluctuation, and load prediction error as fuzzy input variables, and uncertainty coefficient as fuzzy output variable; dividing fuzzy subsets and representing them using Gaussian membership functions; establishing a fuzzy rule base covering all input combinations based on expert experience; calculating the rule trigger intensity through the max-min inference method, and then obtaining the quantitative uncertainty coefficient by defuzzifying using the centroid method.

3. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, The specific process of constructing a deep Q-network module in S4 includes: the module contains a policy network and a target network with consistent structure; the policy network outputs candidate actions and action values, and the target network provides stable target action values; the parameters of the two networks are synchronized every 100 training rounds.

4. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, The specific process of constructing the Actor-Critic module in S4 includes: the module contains an Actor network and a Critic network; the Actor takes a state vector as input and outputs an action probability distribution; the Critic takes a state vector as input and outputs a state value; the Actor calculates the advantage function based on the action value of the deep Q-network and the state value of the Critic, and updates the parameters by maximizing the advantage function objective function; the Critic updates the parameters by minimizing the mean square error between the actual and target state values.

5. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, The data collected in S1 includes photovoltaic power plant data, distribution network data, load data, and constraint parameter data.

6. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, The data preprocessing in S2 specifically includes: cleaning, which includes handling missing values ​​and outliers; normalization, which maps all data to the [0,1] interval; and feature construction, which generates three types of uncertainty features: light intensity fluctuation, temperature fluctuation, and load prediction error, to obtain the preprocessed data.

7. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, The specific process of training the reinforcement learning model in S5 includes: a pre-training stage for the fuzzy logic module, using the uncertainty features generated in S2 as input and the normalized power deviation as a label, minimizing the mean square error of the uncertainty coefficient and the normalized power deviation using gradient descent, iterating until the mean square error is less than 0.005; and a joint training stage for the deep Q-network and Actor-Critic, employing an experience replay mechanism to store the state-action-reward-new state samples generated by the agent's interaction with the environment into the experience replay pool. When the number of samples in the experience replay pool is not less than 64, batch sampling is used to update the parameters of the deep Q-network and Actor-Critic, testing is performed on the validation set every 500 rounds until the loss function of the deep Q-network is less than 0.01 and the loss function of the Critic network in Actor-Critic is less than 0.008, at which point training stops.

8. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, In S5, the training, validation, and test sets are divided in a 7:2:1 ratio, with the test set containing 30% high-uncertainty scene samples. High-uncertainty scene samples are those with illumination intensity fluctuations greater than 500 W / m². 2 Samples with temperature fluctuations greater than 10℃ or load prediction errors greater than 150kW.

9. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, The power allocation mathematical model constructed in S3 is as follows: Based on the data preprocessed in S2, an objective function is constructed with the goal of minimizing the total grid loss in the distribution area, maximizing the photovoltaic absorption rate, and smoothing the total load fluctuation in the distribution area. At the same time, photovoltaic output constraints, power balance constraints, voltage constraints, and line transmission constraints are set as constraints.

10. The power optimization allocation method for multiple photovoltaic access areas based on FL-DQN-AC fusion reinforcement learning according to claim 1, characterized in that, The qualification criteria for model verification in S6 are as follows: the grid loss reduction rate is not less than 15%, the photovoltaic absorption rate increase is not less than 0.08, the load fluctuation smoothing rate is not less than 20%, the constraint satisfaction rate is not less than 99%, and the decision deviation is not greater than 0.5kW. When all verification indicators meet the above criteria, the model verification is deemed qualified.

Images