Microgrid energy management system based on large language model and deep reinforcement learning

Abstract

This application relates to the field of microgrid energy management technology, specifically disclosing a microgrid energy control system based on large language models and deep reinforcement learning. The system constructs a standardized dataset and simulation environment through a preprocessing module; utilizes an instruction generation and preference adaptation module, and obtains a preference quantization adapter through a proxy closed-loop system and supervised fine-tuning; a day-ahead optimization module combines user instructions with linear programming to generate a day-ahead scheduling plan; a policy training module trains real-time policies based on the SAC algorithm; and finally, an online control module drives equipment response. This application achieves accurate quantification of human preferences towards optimization objectives, significantly improving the economy, robustness, and decision interpretability of microgrids in complex and multi-scenario environments.

Description

Technical Field

[0001] This application relates to the field of microgrid energy management technology, and more specifically, to a microgrid energy management system based on large language models and deep reinforcement learning. Background Technology

[0002] Microgrids, as independent and controllable units integrating distributed energy resources, energy storage systems, diesel generators, and controllable loads, are an important means to achieve efficient energy utilization and improve power supply reliability. Their efficient operation highly depends on advanced energy management strategies to optimize the scheduling of internal resources, reduce operating costs, and ensure system safety. However, with the increasing penetration rate of distributed energy resources, microgrid energy management faces severe challenges. Their operating environment is highly complex and uncertain; photovoltaic output fluctuates dramatically due to weather conditions; load demand is random; and electricity prices change dynamically over time.

[0003] While traditional optimization methods such as mixed-integer programming can provide hourly day-ahead scheduling plans, they struggle to effectively incorporate the qualitative preferences of human operators and often lack sufficient flexibility and adaptability when facing minute-level real-time fluctuations and sudden grid-connected / islanded mode switching. In recent years, deep reinforcement learning methods have been applied to learn adaptive control strategies, but their reward function design is extremely complex, and it is difficult to accurately translate the high-level intentions of human experts into quantitative optimization objectives. Furthermore, although large language models perform well in natural language understanding and logical reasoning, how to leverage their powerful semantic understanding capabilities to automatically and reliably transform fuzzy human preferences into specific mathematical objectives that can be embedded in optimization models, and to achieve effective collaboration with the real-time control layer of deep reinforcement learning, remains a key technical challenge that urgently needs to be addressed in the field of smart microgrid energy management. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a microgrid energy management system based on a large language model and deep reinforcement learning.

[0005] According to one aspect of this application, a microgrid energy management system based on a large language model and deep reinforcement learning is provided, comprising: The data preprocessing module is used to perform time alignment, outlier interpolation and imputation, and flexible load baseline separation on the collected historical operating data of the microgrid to obtain a standardized dataset and a microgrid simulation environment with integrated physical constraints; The instruction generation module is used to iteratively generate and audit standardized datasets and expert preference texts based on a closed-loop system consisting of generation agents and evaluation agents to obtain instruction fine-tuning datasets containing high-quality dialogue samples. The preference adaptation module is used to supervise the fine-tuning of a pre-trained large language model using an instruction fine-tuning dataset to obtain a preference quantization adapter that can parse user intent. The day-ahead optimization module is used to input the received user control instructions into the preference quantization adapter to infer the objective function weights, and combine them with future prediction data to perform global optimization through mixed integer programming to obtain a day-ahead scheduling plan that includes real-time suggested state of charge and power reference values. The strategy training module is used to introduce the day-ahead scheduling plan as a guiding signal into the microgrid simulation environment and perform deep reinforcement learning training based on the SAC algorithm to obtain the trained SAC strategy. The online control module is used to input real-time collected operating status data into the trained SAC strategy for online inference to obtain energy control commands, which are used to drive the power response of microgrid energy storage and power generation equipment.

[0006] Compared with existing technologies, this application provides a microgrid energy management system based on a large language model and deep reinforcement learning. Firstly, by automatically generating supervised fine-tuning and instruction fine-tuning datasets for the large language model, it effectively overcomes the technical obstacles of traditional optimization methods, such as low utilization of human expert experience and extremely complex and difficult-to-quantify reward function design. This significantly improves the interpretability and human-computer interaction capabilities of the microgrid energy management system in the decision-making process. Secondly, the integration of day-ahead scheduling and reinforcement learning strategies enables the system to not only provide hourly globally optimal paths but also maintain extremely high adaptability and robustness under minute-level real-time fluctuations and sudden grid-connected / islanded mode switching scenarios, significantly reducing system operating costs, curtailment rate, and power fluctuation risks. Furthermore, the flexible load baseline separation and integrated physical constraint simulation environment in data preprocessing ensure that the trained SAC strategy can operate safely under complex physical constraints. Attached Figure Description

[0007] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0008] Figure 1 This is a schematic block diagram of a microgrid energy management system based on a large language model and deep reinforcement learning, according to an embodiment of this application.

[0009] Figure 2This is a schematic diagram illustrating the workflow and interaction of an LLM agent closed-loop system according to an embodiment of this application, showing the iterative logic of agent generation and agent evaluation in the instruction generation module; Figure 3 This is a comparison diagram showing the effect of the upper-layer day-ahead scheduling optimizer and the lower-layer SAC controller working together according to an embodiment of this application; Figure 4 This is a diagram illustrating the training effect verification of the introduction of random power grid mode switching and random day-ahead scheduling guidance according to embodiments of this application. Figure 5 This is a diagram illustrating the effect of fine-tuning a large language model based on LoRA according to an embodiment of this application. Detailed Implementation

[0010] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0011] Figure 1 This is a schematic block diagram of a microgrid energy management system based on a large language model and deep reinforcement learning, according to an embodiment of this application. Figure 1 As shown, the microgrid energy management system 100 based on large language models and deep reinforcement learning according to an embodiment of this application includes: The data preprocessing module 110 is used to perform time alignment, outlier interpolation and imputation, and flexible load baseline separation on the collected historical operating data of the microgrid to obtain a standardized dataset and a microgrid simulation environment with integrated physical constraints. It should be understood that the raw operating data of a microgrid often suffers from problems such as asynchronous timestamps, missing data, and abnormal noise caused by sensor malfunctions. Directly using this low-quality data for model training can lead to difficulties in policy convergence or incorrect decision-making logic. Simultaneously, in order to train the reinforcement learning agent without damaging the actual equipment, a simulation environment with integrated strict physical constraints must be constructed to ensure the safety of the exploration process.

[0012] In a specific example of this application, the data preprocessing module includes: a multi-source alignment unit for performing multi-source data time alignment and resampling on historical microgrid operation data to obtain aligned time-series data; a time-series cleaning unit for performing anomaly detection and missing value cleaning on the aligned time-series data to obtain cleaned time-series data; a load separation unit for separating a rigid load baseline from the total load of the cleaned time-series data based on a preset flexible scaling factor and splicing it to the feature dimension to obtain a standardized dataset; and a physical simulation unit for defining the physical equations and boundary conditions for microgrid operation to obtain a microgrid simulation environment with integrated physical constraints.

[0013] Specifically, the multi-source alignment unit performs time alignment and resampling of multi-source data. According to the specification's description of multi-scenario dataset acquisition and preprocessing, the system is designed with... Heaven is in The original photovoltaic, load, and electricity price at each time step are respectively , , Then, resampling is performed at a uniform sampling interval to ensure that all physical quantities are strictly aligned on a uniform time axis.

[0014] Next, the time-series cleaning unit performs anomaly detection and missing value cleaning on the aligned time-series data. For missing value points, the system uses weighted interpolation to repair them using data from adjacent valid time steps. The interpolation formula is as follows: In the formula, This represents any quantity of photovoltaic power, load, or electricity price after completion. and These are the original values ​​of the effective time steps before and after the missing point, respectively. The interpolation weight coefficients are determined by the time distance between the missing time step t and the preceding and following valid time steps. For outlier identification, the system uses the Z-score criterion for discrimination and calculates statistical features for any given value: In the formula, This represents the physical quantity at time t. and These are the mean and standard deviation of the physical quantity over historical data. This represents the statistical characteristics of the physical quantity at time t. If the calculation result satisfies the criterion... If it is, then it is determined to be an outlier. The preset abnormal threshold. The value is usually set to 3. According to the 3σ principle of normal distribution, data points exceeding this threshold are considered to be noise or sensor malfunctions with extremely low probability and need to be removed. Points determined to be abnormal will be replaced by the above interpolation formula to obtain cleaned data.

[0015] Furthermore, the load separation unit separates a flexible load baseline from the total load of the cleaned time-series data based on a preset flexibility scaling factor. To ensure the requirements of both rigid and flexible loads are met, the flexible load baseline is defined as follows: In the formula, Indicates time Flexible and adjustable load capacity This is a flexible scaling factor, representing the proportion of adjustable load in the total load. Its value is determined based on the proportion of the installed capacity of controllable devices (such as air conditioners, electric water heaters, electric vehicle charging stations, etc.) in the total load in the actual microgrid scenario. The total load requirement is then determined. Subsequently, the system standardizes the data, calculates the global mean and standard deviation for each dimension, and performs dimensionless Z-Score normalization on all features to obtain a standardized dataset.

[0016] Finally, the physical simulation unit is used to define the physical equations and boundary conditions for microgrid operation to obtain a microgrid simulation environment with integrated physical constraints. This unit builds the environment based on the Gymnasium framework, integrating physical constraints, day-ahead dispatch guidance, and grid modes (grid-connected / islanded). This environment integrates the physical evolution laws of microgrid operation, especially the state-of-charge constraints for battery energy storage systems, which satisfy the following physical equations: In the formula, For a moment The state of charge, and These represent the battery charging and discharging power, and For charging and discharging efficiency, For the battery's rated capacity, and These are the minimum and maximum state-of-charge limits, respectively. This is the simulated step size.

[0017] The instruction generation module 120 is used to iteratively generate and audit standardized datasets and expert preference texts based on a closed-loop system consisting of a generation agent and an evaluation agent, to obtain an instruction fine-tuning dataset containing high-quality dialogue samples. It should be understood that traditional microgrid energy management often relies on complex mixed-integer programming or a single reinforcement learning reward function. These methods struggle to directly understand and incorporate the qualitative preferences of human operators, such as vague instructions like "moderately reduce diesel engine usage." Through the instruction generation module, the system can leverage the powerful semantic understanding and logical reasoning capabilities of large language models to automatically transform vague human preferences into quantitative microgrid parameter adjustment models. Figure 2 As shown, the instruction generation module constructs a closed-loop system consisting of a generator and an evaluator. The generator proposes objective functions based on scenario features and expert preference texts in a standardized dataset, while the evaluator performs multi-dimensional verification of the proposals based on audit rules. Through iterative interaction of "generation-audit-feedback-regeneration," the two parties solve technical challenges that may arise during preference quantification, such as systematic evaluation deficiencies, local optima, or hidden damages. This ensures that the generated scheduling strategy satisfies human preferences without drastically worsening other non-target physical states within the system, thereby guaranteeing the economy, robustness, and interpretability of microgrid operation.

[0018] In a specific example of this application, the instruction generation module includes: a scenario extraction unit, used to extract corresponding operational scenario data from a standardized dataset based on the time index of the benchmark key performance indicators, and concatenate the statistical features of the operational scenario data with expert preference text and benchmark key performance indicators to obtain the generator prompt context. The statistical features of the operational scenario data specifically include scenario type, correlation coefficient between photovoltaic and load, load peak information, initial state, and electricity price curve features to ensure that the generation agent can fully understand the current microgrid operating conditions; a preference generation unit, used to input the generator prompt context into the generation agent for quantitative reasoning and parsing to obtain a target function proposal containing preference item identifiers and corresponding weight values; a scheduling reconstruction unit, used to reconstruct the day-ahead scheduling model with the target function proposal as an additional penalty term and solve it to obtain the corresponding optimized scheduling trajectory and new key performance indicators; a preference auditing unit, used to perform multi-dimensional preference validity audits on the new key performance indicators and benchmark key performance indicators to obtain audit feedback results; and a data synthesis unit, used to combine the operational scenario data and expert preference text into prompts and use the target function proposal as a response if the audit feedback result is successful to obtain an instruction fine-tuning dataset.

[0019] Specifically, firstly, the scenario extraction unit performs scenario instance extraction and context assembly. Based on the time index of the benchmark key performance indicators (KPIs), it extracts the corresponding operational scenario data from the standardized dataset. Then, it concatenates the statistical features of the operational scenario data, expert preference text, and the benchmark KPIs according to a preset template to obtain the generator prompt context. The benchmark KPIs refer to the indicator data generated by scheduling optimization before the benchmark date under the objective of minimizing pure economic cost. Next, the preference generation unit performs quantitative proposal reasoning based on a generation agent. The generator prompt context is input into the generation agent for quantitative reasoning and parsing. The generator agent is strictly guided, following the rule that negative preference weights are negative, positive preference weights are positive, and achieved goal weights are zero. It outputs a target function proposal containing preference item identifiers and corresponding weight values.

[0020] Subsequently, the scheduling reconfiguration unit performs deterministic simulation verification and calculates new indicators. It reconfigures the day-ahead scheduling model using the proposed objective function as an additional penalty term and calls the solver to obtain the corresponding optimized scheduling trajectory and new key performance indicators. In this process, the overall objective function from the preference mapping weights is calculated as follows: In the formula, The corrected overall objective function; The scheduling period; , , They are time points The basic economic costs of grid interaction, diesel engine and battery operation; For a set of preferences; The first one extracted from the objective function proposal Item preference weights; For the first The physical quantity mapping function corresponding to the item preference, such as the diesel engine output value; For a moment The system state variables.

[0021] Furthermore, the preference audit unit performs multi-dimensional preference validity audits. It should be understood that a microgrid is a strongly nonlinearly coupled physical system. Single-dimensional verification cannot detect short-sighted decisions that involve "robbing Peter to pay Paul." For example, to achieve a preference of "reducing diesel engine usage," energy storage batteries may undergo extremely high-rate charge-discharge cycles, thereby overdrawing battery life or increasing the risk of curtailment. By calculating the global coupling interference, the audit mechanism can quantify the disturbance of the target preference to other physical subsystems (such as voltage stability and state of equilibrium), eliminating samples that, while achieving the preference, cause severe system oscillations or hidden damage.

[0022] In a basic implementation, the first step is a target-preference alignment check, verifying whether the optimization tool chosen by the generator (such as diesel_generation) is logically consistent with the user's qualitative preferences; the second step is a weight-rule compliance audit, checking whether the weight symbols conform to preset rules; and the third step is a direction correctness verification, calculating the changes in key indicators. Confirm whether the target metric is moving in the direction users expect (e.g., if the preference is "reduce diesel consumption"), then verification is required. If the target indicator is already zero in the baseline, it automatically passes; finally, a reasonableness assessment is conducted, calculating the additional economic cost of achieving the preference. We determine whether the cost increase is within the allowable fixed tolerance range (e.g., the cost increase does not exceed 5%) in order to eliminate solutions with excessive costs.

[0023] In the process of constructing the instruction fine-tuning dataset for a microgrid energy management system, the multidimensional preference validity auditing step in the basic embodiment has significant logical limitations and functional blind spots. Specifically, this mechanism only performs single-dimensional direction verification (verifying whether the sign of the indicator change meets expectations) and static cost threshold verification (verifying whether the increased cost is within the fixed tolerance). However, a microgrid system is essentially a multivariable physical system with strong nonlinear coupling characteristics. The various key performance indicators do not evolve independently, but rather, a change in one affects the whole system.

[0024] First, the basic implementation lacks consideration of implicit costs. While achieving user preferences (such as reducing diesel engine reliance) may not reach the explicit direct operating cost floor, it often comes at the cost of drastically deteriorating other non-target physical states within the system. For example, to forcibly reduce diesel engine output, the scheduling strategy may cause the energy storage battery to undergo extremely high-rate charge-discharge cycles in a short period. Although this is not immediately reflected in the current operating cost, it severely depletes the battery life (SOH) and increases the risk of curtailment. Because the basic implementation ignores the mutual interference between these physical states, the resulting scheduling strategy may be a short-sighted decision.

[0025] Secondly, the basic implementation lacks sensitivity to the law of diminishing marginal utility. In the Pareto front region of multi-objective optimization, the optimization effect exhibits non-linear characteristics: in the large deviation range, a small investment can yield huge benefits; however, in the approaching extreme value range, the same marginal cost investment may only yield negligible preference satisfaction. The basic implementation uses a fixed cost tolerance range for a one-size-fits-all judgment, failing to distinguish between high-efficiency optimization and low-efficiency fine-tuning. This may result in a large number of samples with extremely low cost-effectiveness being mixed into the dataset, thus misleading the value judgment logic of subsequent large models, causing them to tend to pay unreasonable economic costs even when faced with small benefits.

[0026] To address the aforementioned shortcomings, a preference effectiveness evaluation method based on multidimensional coupling elasticity auditing is proposed. This method constructs a dynamic dual auditing mechanism that balances system global stability and economic efficiency by introducing global coupling mutual disturbance degree and economic trade-off elasticity index. In another preferred embodiment, the preference auditing unit is further configured to: identify target preference items from the objective function proposal and define non-target indicators in the new key performance indicators and benchmark key performance indicators as a set of related indicators; calculate the ratio of the relative change of the set of related indicators to the relative change of the target preference item using the weighted Euclidean distance formula to obtain the global coupling mutual disturbance degree characterizing the system sacrifice rate; perform preference-cost dynamic thermodynamic elasticity estimation on the new key performance indicators and benchmark key performance indicators to obtain the economic trade-off elasticity index; determine the nonlinear decision boundary based on the sigmoid activation function and the disturbance penalty threshold; and, based on the nonlinear decision boundary, perform weighted scoring on the economic trade-off elasticity index and global coupling mutual disturbance degree, and combine it with semantic direction consistency verification of expert preference text to obtain audit feedback results.

[0027] First, the global coupling perturbation matrix is ​​constructed and calculated to quantify the degree of disturbance caused to other physical subsystems of the microgrid by the achievement of the target preference. The system receives not only new and old key performance indicators (KPIs) and Furthermore, it is necessary to parse the current optimization objective preference term identifier from the objective function proposal. Subsequently, all technical indicators other than the objective term and cost term (such as voltage stability, SOC variance, and curtailment rate) are defined as the association set. By calculating the weighted geometric relationship between the relative change rate of the target indicator and the relative change rates of each indicator in the associated set, the global coupling perturbation degree can be solved. .

[0028] This step can be represented as: in, The global coupling mutual interference degree is used to characterize the state sacrifice rate or mutual interference degree of the system; For an associative set; and They represent the associated sets respectively. The values ​​of the i-th physical index before and after preference adjustment and under the baseline state; and This corresponds to the value of the target preference index; These are preset physical importance weighting coefficients used to distinguish the degree of impact of different physical states on system safety (e.g., for critical energy storage lifetime, SOH). It can be set to 0.8, while for minor voltage fluctuations, (Can be set to 0.2). To prevent small regularization constants with a denominator of zero.

[0029] It is understandable that there are strong couplings between the various physical quantities of a microgrid, and the optimization of a single objective should not come at the expense of the overall robustness of the system. This step can accurately identify low-quality decisions that, while achieving user preferences, have caused severe system-level oscillations (the butterfly effect), thereby eliminating samples at the source of data generation that are not only unrobust but may also cause hidden damage to physical equipment, ensuring that the fine-tuned model has a global perspective.

[0030] Secondly, the system performs preference-cost dynamic thermodynamic elasticity estimation on the new key performance indicators (KPIs) and benchmark KPIs to obtain an economic trade-off elasticity index. This aims to introduce a microeconomic perspective to evaluate the cost-effectiveness of optimization decisions. The system is based on cost data under both new and old conditions. , ) and target value data ( , The economic trade elasticity index is calculated as the percentage of preference satisfaction gained by increasing costs by a unit percentage, i.e., the economic trade elasticity index. , means as follows: in, In exchange for an economy-wide resilience index, used to measure the optimized energy efficiency ratio; It is the natural logarithm function, used to smooth numerical fluctuations and simulate the perceived characteristics of diminishing marginal utility; This is an adjustment factor used to maintain numerical stability when cost changes are extremely small (e.g., taking...). ).

[0031] This is understandable, because a simple cost difference cannot reflect the efficiency of optimization, especially when different units of measurement (such as dollars and kilowatt-hours) are mixed. When When the growth rate of preference gains outpaces the growth rate of costs, it indicates a highly efficient optimization. This mechanism solves the problem of direct comparison of heterogeneous dimensions in multi-objective optimization, and can keenly identify and eliminate inefficient samples in the diminishing marginal utility region, preventing the model from learning erroneous logic that disregards costs.

[0032] Finally, a comprehensive decision and feedback generation based on the convex hull boundary is performed. This step no longer relies on a single static threshold, but instead constructs a non-linear decision boundary to comprehensively consider the samples from multiple dimensions. The system will then calculate the mutual perturbation degree. With elasticity index Input the decision model and calculate the overall effectiveness score. Based on this, an audit feedback result containing Boolean status and metadata is generated. The judgment logic is as follows: In the formula, This serves as the final validity score; and The weighting coefficients for elasticity gain and mutual interference penalty are respectively (e.g. ); The function is used to map the elasticity exponent to a normalized interval and introduce a soft threshold. (For example ); The function is used to penalize disturbances exceeding a threshold. (For example A linear penalty is applied to the samples. This is a directional consistency verification function based on [the function name].

[0033] Using the Sigmoid function to trade economics for an elasticity index Mapped to The range is defined, and a flexible soft threshold is introduced. As a center point, this allows for a smooth assessment of elasticity gains, where small changes near the threshold can cause significant changes in the score, while saturation occurs far from the threshold. Utilizing... Functions exceeding the safe interference threshold Global coupling mutual interference A linear penalty is applied. This means that points are deducted only when the system disturbance exceeds a certain limit, and the greater the disturbance, the heavier the penalty. The above two parts are combined and weighted by preset coefficients. (Elastic return weight) and (Interference penalty weight) Calculate the final effectiveness score.

[0034] After obtaining the comprehensive score, the system combines it with the basic directional consistency verification function. To generate the final audit conclusion. Only when the comprehensive score is... A sample is considered valid only if the return is greater than 0 (meaning the benefit outweighs the risk / cost) and the directional consistency check passes; otherwise, it is invalid. The logic of the basic directional consistency check function is as follows: Expert preference text (e.g., "reducing diesel engine use"), new key performance indicators and benchmark key performance indicators Analysis of preference direction: from The expected direction of change in the indicator is analyzed. For example, "decrease" corresponds to a negative change, and "increase" corresponds to a positive change. The actual change in the target indicator is then calculated. If the preference is "to reduce a certain indicator", then it is required that... (or If the preference is "to increase a certain indicator", then it requires... (or If the goal is to "reduce" an indicator that is already zero in the benchmark (i.e.) and If the condition is met, it is considered to pass automatically (returns True).

[0035] This comprehensive decision-making mechanism ensures that all samples ultimately included in the database fall within the high-resilience, low-disturbance quality region. Only samples that are economically viable (high resilience) and have a controllable negative impact on the system (low disturbance) are deemed valid. This not only significantly improves the quality of the fine-tuning dataset but also endows the subsequently trained model with both obedient and intelligent decision-making capabilities—that is, while satisfying human preferences, it can autonomously avoid strategies that could potentially harm the stability and long-term economics of the microgrid system.

[0036] This preferred embodiment aims to address the technical problem in existing microgrid energy management where the preference quantification process lacks systematic evaluation, easily leading to local optima or hidden damage. By introducing a multi-dimensional coupled elastic audit mechanism, it quantifies the impact of preference realization on the global stability of the system using global coupled mutual interference metric, effectively identifying and eliminating short-sighted strategies that sacrifice battery life or voltage stability. Simultaneously, it establishes a unified energy efficiency evaluation standard using an economic trade-off elasticity index, accurately filtering out invalid fine-tuning samples with extremely low marginal utility. Ultimately, this preferred embodiment achieves a leap from single-dimensional verification to multi-dimensional dynamic auditing, ensuring that the generated instruction fine-tuning dataset possesses high robustness and high energy efficiency ratio. This allows for the training of an intelligent scheduling model that can accurately respond to human intentions while maintaining the long-term healthy operation of the system, significantly improving the security and economy of microgrid energy management.

[0037] Finally, the data synthesis unit performs fine-tuning dataset formatting and storage. If the audit feedback is successful, it combines the runtime scenario data with expert preference text to form prompts and provides the objective function proposal as a response, resulting in an instruction fine-tuning dataset containing high-quality dialogue samples. The structure of a single instruction fine-tuning sample is as follows: In the formula, This is a sample for fine-tuning a single instruction; Text formatting functions; For running scenario data; Texts preferred by experts; To verify the proposed weight parameters of the objective function, if the audit fails, the system will store the feedback in the history and guide the generator to iterate.

[0038] It is worth mentioning that, in order to prevent the closed-loop system from becoming inefficient in data generation due to overly strict audit standards, this system also includes a "final review and standard relaxation" mechanism. If the objective function proposal put forward by the generator still fails to pass the evaluator's regular audit after reaching the preset maximum number of attempts (e.g., 5 times), the system will initiate the final review process.

[0039] At this point, the evaluator agent receives a special system instruction instructing it to moderately relax audit criteria (e.g., temporarily increase the penalty threshold for global coupling interference) and re-examine all failed proposals in the history. The evaluator's goal is to sift through these failed attempts to find the "suboptimal solution" that is "directionally correct" and has the "least costly" option. If an acceptable suboptimal solution is found, it is marked as successful and included in the fine-tuning dataset; if all attempts fail to meet the minimum requirements, the combination of the scenario and preference is marked as infeasible and ultimately abandoned.

[0040] The preference adaptation module 130 is used to supervise the fine-tuning of a pre-trained large language model using an instruction fine-tuning dataset to obtain a preference quantification adapter capable of parsing user intent. It should be understood that while the pre-trained large language model possesses general semantic understanding capabilities, it lacks specialized knowledge of the specific scheduling logic and numerical weight distribution of microgrids. By utilizing a high-quality instruction fine-tuning dataset containing combinations of "preference-scenario-optimal JSON weights," the model can be guided to learn to generate the desired machine-readable response in a specific context, thereby achieving a precise mapping from qualitative intent to quantitative objectives. In this process, the low-rank adaptation technique effectively reduces computational overhead. By freezing the main parameters of the base model and training only a small-scale adapter layer, the general reasoning capabilities of the base model are preserved, while significantly improving the model's specialized adaptability in the field of microgrid energy management. This solves the technical problems of excessive resource consumption and catastrophic forgetting inherent in traditional full-parameter fine-tuning.

[0041] In a specific example of this application, the preference adaptation module includes: a data formatting unit, used to format the instruction fine-tuning dataset using a standard dialogue template and perform word segmentation encoding to obtain a serialized training batch containing the input index sequence and only retaining the labels of the response portion; a LoRA injection unit, used to freeze the original weight parameters of the pre-trained large language model and inject a low-rank adapter containing a low-rank matrix and scaling coefficients into the attention layer bypass to obtain a LoRA-enhanced model architecture; and an adapter training unit, used to input the serialized training batch into the LoRA-enhanced model architecture for forward inference, calculate the cross-entropy loss between the predicted distribution and the true labels, and train the preference quantization adapter by updating only the parameters of the low-rank adapter.

[0042] Specifically, firstly, the data formatting unit is responsible for instruction data serialization and tensor encoding. It loads and parses the JSON dataset file generated in the previous steps. Each successful record in the file is constructed as a training sample in a standard dialogue format, using specific role labels (such as system, user, assistant) to clearly define the dialogue structure, guiding the model to learn to generate the expected JSON response in a specific context. Next, a tokenizer is used to convert the formatted text into a sequence of integer indices to generate the input index sequence, and a corresponding attention mask is constructed. During this process, the system creates a label sequence of the same length as the input, ignoring the labels of the prompt part and retaining only the TokenID of the completion part for subsequent loss calculation.

[0043] Subsequently, the LoRA injection unit performs low-rank adapter injection and architecture refactoring. This unit first freezes most of the parameters of the base model. Small, trainable low-rank adapter layers are injected only around the critical parts of the model (attention layers). These adapters consist of two low-rank matrices. and The product is used to approximate the weight update amount during full parameter fine-tuning. The calculation during forward propagation is modified as follows: In the formula, The output activation value of the current layer; This is the input vector for the current layer; The pre-trained weight matrix is ​​frozen; A low-rank matrix initialized to zero; A low-rank matrix initialized by Gauss.

[0044] Next, the adapter training unit performs supervised optimization and parameter updates. This unit takes sequential training batches as input to the LoRA augmentation model architecture for forward inference; the training process is a supervised learning task. The model receives formatted cue portions as input and is prompted to generate corresponding JSON output. The goal of the training algorithm is to minimize the difference between the predicted output and the true label by adjusting the parameters of the LoRA adapters (A and B), typically using the cross-entropy loss function. The optimization objective is: In the formula, For the entire training set, This is a prompt, or alert. This is the completion generated. Gradient updates are applied only to parameters A and B of the LoRA adapter, and not to the base model parameters. It remains unchanged. By iterating repeatedly across the entire training dataset, the LoRA adapter gradually learns to adapt to specific preferences and operating conditions.

[0045] In a specific implementation example of this application, the system uses a reserved validation set to evaluate the performance of the fine-tuned model. Evaluation criteria include: whether it can generate syntactically correct JSON for previously unseen preference-condition combinations, and whether the internal logic of the JSON is reasonable. Once the model performance meets expectations, the trained LoRA adapter weights can be merged with the weights of the original base model to generate a new, independent model file. This merged model can then be deployed and used for inference. Experimental analysis shows that LoRA-based LLM fine-tuning not only significantly improves the model's understanding of human qualitative preferences but also ensures that the model can output logically rigorous and physically constrained quantitative weight parameters when dealing with complex microgrid scenarios. Figure 5 The analysis of the fine-tuning effect shown indicates that, compared with the untuned base model, the model fine-tuned by LoRA has significantly improved accuracy and logical consistency when parsing preference instructions and generating corresponding JSON weight parameters, verifying the effectiveness of low-rank adaptation technology in injecting specialized knowledge in the microgrid field.

[0046] The day-ahead optimization module 140 is used to input received user control commands into a preference quantization adapter to infer the objective function weights, and then perform global optimization through mixed-integer programming in conjunction with future forecast data to obtain a day-ahead scheduling plan that includes real-time suggested state of charge and power reference values. It is understandable that microgrid systems not only need to cope with minute-level real-time fluctuations during operation, but also need to optimize resource allocation throughout the day to reduce operating costs. However, traditional optimization methods struggle to effectively incorporate the qualitative preferences of human operators. This module embeds the quantitative weights parsed from a large language model into the optimization model, enabling precise responses to complex user control intentions while ensuring physical constraints and global economic efficiency. This compensates for the shortcomings of relying solely on algorithm self-iteration in terms of decision interpretability and human-machine collaboration.

[0047] In a specific example of this application, the day-ahead optimization module includes: a preference parsing unit, used to assemble user control instructions and future prediction data into a prompt word input preference quantization adapter for inference and JSON parsing to obtain objective function weights containing preference types and weight coefficients; an optimization modeling unit, used to define a sequence of decision variables and a constraint system containing power balance and equipment mutual exclusion logic based on future prediction data, and to construct a composite objective function that minimizes the sum of total cost and preference penalty terms in combination with the objective function weights to obtain an instance of the optimization problem to be solved; and a scheduling solution unit, used to call the optimization solver to perform branch and bound solution on the instance of the optimization problem to be solved, and if an optimal solution exists, to extract the key state sequence as macro-guidance to obtain a day-ahead scheduling plan containing immediate suggested state of charge and power reference values.

[0048] Specifically, firstly, the preference parsing unit is responsible for inputting the received user control commands into the preference quantization adapter for intent parsing. During this process, the system assembles the user control commands and future forecast data (including photovoltaic output, load demand, and electricity price forecasts) into prompt words. These prompt words are then used by the preference quantization adapter, which has been loaded with trained low-rank adapter (LoRA) weights, for inference and JSON parsing. The output is structured data containing the target preference type and its corresponding numerical weights. The weight set is represented as follows: In the formula, This is the set of weights for the extracted objective function; For the first Types of preferences; This refers to the weighting coefficient corresponding to this preference; This is the currently active set of preference indices.

[0049] Secondly, the optimization modeling unit is responsible for defining the sequence of decision variables and the constraint system based on future forecast data, and constructing a composite objective function. The objective function is to minimize the total cost of the microgrid, and its formula is: Among them, basic operating costs It covers Grid cost at any time Diesel engine cost Battery degradation costs and load reduction costs Preference penalty item In the formula For preference toolbox, These are the weights generated by LLM. Is this preference in The physical quantity corresponding to a given moment.

[0050] The specific calculations for each cost item are as follows: Grid costs: ,in For electricity purchase price, and For buying and selling electricity, This is the electricity sales discount factor.

[0051] Diesel engine cost: ,in , , This is the fuel consumption coefficient. To provide power to the diesel engine, For fuel prices, For start-up costs, It is in the startup state. This indicates the cost of fuel.

[0052] Battery cost: ,in This is the battery loss coefficient. and This refers to the charging and discharging power.

[0053] Load reduction costs: ,in The load reduction penalty factor, To reduce power.

[0054] The first constraint is power balance: .

[0055] For mutual exclusion constraints, the Big-M constraint method is used to avoid the phenomenon of simultaneous charging and discharging of energy storage batteries and simultaneous buying and selling of electricity by the grid: Secondly, there are device-level scheduling constraints: representing the battery's state of charge at time t, which must satisfy conditions including dynamic equations, etc. In the formula, For charging power, For discharge power, For battery capacity, Improve battery charging efficiency. For discharge efficiency, and These represent the minimum and maximum battery charge, respectively.

[0056] For diesel engines, the equations they satisfy include output constraints and ramping constraints: In the formula, This refers to the output power of the diesel engine. For the diesel engine's maximum output, This is the minimum output of the diesel engine. This is a status indicator for the diesel engine. This represents the slope constraint range.

[0057] For flexible loads, the following must be satisfied: In the formula, Power reduction for load, Let be the total amount of reducible flexible load at time t, where This refers to the flexible load baseline obtained during the data preprocessing stage. .

[0058] Subsequently, the scheduling unit invokes the optimization solver to perform branch-bound global optimization on the instance of the optimization problem to be solved. The system checks the states returned by the solver; if an optimal solution exists, it extracts the key state sequence as macroscopic guidance. The extracted reference trajectory vector is: In the formula, For guiding trajectory set; The optimized real-time suggested state of charge; The recommended diesel engine power; The system then proposes power reductions for the load. Through these steps, the system ultimately generates a day-ahead dispatch plan that includes real-time proposed state of charge and power reference values.

[0059] The strategy training module 150 is used to introduce the day-ahead scheduling plan as a guiding signal into the microgrid simulation environment, and to perform deep reinforcement learning training based on the SAC algorithm to obtain a trained SAC strategy. It is understandable that the microgrid operating environment is highly complex and uncertain; photovoltaic output fluctuates drastically due to weather conditions, and load demand exhibits randomness. Although day-ahead scheduling can provide hourly global economic solutions, it is difficult to cope with minute-level real-time fluctuations and sudden grid mode switching (such as dynamic switching between grid-connected and islanded modes). By introducing the day-ahead scheduling plan as a guiding signal, the reinforcement learning agent can be guided to track long-term economic plans in minute-level control, ensuring system operational safety and robustness while also considering long-term economic efficiency, thus solving the problem that single-layer optimization or rule-based control cannot simultaneously address multi-scale objectives. Furthermore, the SAC algorithm (Soft Actor-Critic algorithm), by introducing an entropy regularization term, can improve the agent's exploration ability and avoid prematurely falling into local optima, making the trained strategy more generalizable in complex multi-scenario situations.

[0060] In a specific example of this application, the policy training module includes: a state observation unit, used to read real-time physical quantities and mode flags from the microgrid simulation environment, and concatenate them with guidance reference values ​​extracted synchronously from the day-ahead scheduling plan to obtain a high-dimensional integrated state observation vector; a reward calculation unit, used to calculate a single-step reward value consisting of negative operating cost, physical constraint violation penalty, and mean square tracking error penalty for the guidance reference value after the agent outputs an action and receives feedback, and store it in the experience replay buffer; and a policy optimization unit, used to sample transfer samples from the experience replay buffer, update the critic network parameters by minimizing the Bellman residual, and update the policy network parameters by maximizing the weighted sum of action value and policy entropy to iteratively optimize and obtain the trained SAC policy.

[0061] Specifically, firstly, the state observation unit reads real-time physical quantities and mode flags from the microgrid simulation environment and concatenates them with guidance reference values ​​synchronously extracted from the day-ahead dispatch plan to obtain a high-dimensional comprehensive state observation vector. This high-dimensional observation vector includes real-time states (such as photovoltaic output, load, and SOC), short-term forecasts (net load for the next 3 hours), and constraint states (SOC safety margin, etc.). In particular, in the feature construction of the constraint states, to enable the neural network to effectively capture the time-periodic characteristics of microgrid operation (such as diurnal variations in photovoltaic power and peak-valley patterns in load), the state observation unit performs sine and cosine encoding on the current time step index, mapping it to continuous periodic numerical features, as well as key day-ahead dispatch guidance signals: the day-ahead dispatch plan's SOC, battery power, diesel engine power, flexible load reduction, and grid-sold power. Furthermore, it includes day-ahead dispatch status flags to indicate the effectiveness of day-ahead dispatch in grid-connected mode.

[0062] Secondly, after the agent outputs an action and receives feedback, the reward calculation unit calculates the single-step reward value and stores it in the experience replay buffer. The reward function is designed as follows: In the formula, To enhance the total reward value of the learning controller, Total operating cost, These are the basic constraint penalties (including power imbalance, light curtailment, etc.). Penalty for RL decisions deviating from the day-ahead scheduling guidance of the upper-level optimizer. To meet the requirements of decision smoothness, To encourage the low-charge-high-discharge excitation term for the battery, the specific calculation formula is as follows: In the formula, This refers to the grid interaction cost (electricity purchase cost minus electricity sales revenue). This refers to the operating cost of the diesel generator (including fuel and start-up costs). For the cost of battery degradation, To reduce costs by reducing load, , , , , These represent the power imbalance penalty, the curtailment penalty, the target SOC penalty, the SOC violation penalty, and the ramp rate violation penalty, respectively. Indicates the power imbalance. Indicates the amount of light discarded. This indicates the current state of charge. Indicates the target state of charge. This indicates the degree to which the SOC violates the constraints. This indicates the amount of violation of the ramp-up rate; , , , , This indicates deviations from the day-ahead dispatch guidance in terms of SOC, battery power, diesel engine power, grid interaction, and flexible load reduction instructions, which will incur penalties. , , , , These are the guidance values ​​for SOC, battery power, diesel engine power, grid interaction power, and flexible load reduction power given in the recent dispatch plan. , , , These are the maximum power limits for batteries, diesel engines, grid interconnection, and flexible load shedding, respectively. , This indicates constraints on energy storage and diesel engine smoothing terms. and These represent the battery power and diesel engine power at the previous moment, respectively. The percentile of the current electricity price in history is used to measure the arbitrage reward of energy storage on the grid. This represents the current battery power (positive for charging, negative for discharging). For arbitrage incentive weighting coefficients, The decision-making time step.

[0063] Finally, the policy optimization unit samples transfer samples from the experience replay buffer, updates the critic network parameters by minimizing the Bellman residual, and updates the policy network parameters by maximizing the weighted sum of action value and policy entropy. SAC is chosen as the reinforcement learning model algorithm, whose optimization objective is to maximize the expected reward with entropy. In the formula, It's a temperature coefficient used to balance rewards and exploration. Representation strategy The objective function value, i.e., the expected return with entropy, It is the mathematical expectation. State-action pairs represent policies Sampling from the generated trajectory distribution, In the state Execute action The rewards received For strategy In state Entropy under the condition is defined as .

[0064] In addition, SAC introduces two independent Q-networks (commentator networks) to alleviate the overestimation problem of the value function: In the formula, and These represent two independent Critic networks (Q networks), with parameters respectively. , . The target for updating the Q value is the target value. for: in, This is a discount factor, typically set to 0.99, used to measure long-term returns. This represents the state at the next moment. The state value function is typically calculated using the TargetQ network, i.e. .

[0065] In actual calculations, the smaller value of the two Q-networks is chosen as the target in order to suppress the accumulation of positive bias.

[0066] Subsequently, policy optimization is performed, and the policy network can obtain the parameter update direction by minimizing the KL divergence: In the formula, Kullback-Leibler divergence is used to measure the difference between two probability distributions. It is the normalization factor, and the Actor loss function can be derived from it: in, This is the loss function for the policy network (Actor). This serves as a buffer for experience replay. The smaller Q value of the two Critic networks is chosen to mitigate the overestimation bias.

[0067] Temperature parameters can be set manually or automatically. The solution for automatic adjustment is as follows: In the formula, This is the target entropy value, usually set as a negative number for the action space dimension. This mechanism allows the algorithm to dynamically balance exploration and utilization based on the training progress.

[0068] Finally, the SAC model parameters are designed. First, the network structure: both the policy network and the value network use a three-layer fully connected neural network, with 256 neurons in each layer. The Tanh activation function is used to handle non-linear features, as shown in the following equation: In the formula, and These are the outputs of the hidden layer, respectively. It is the hyperbolic tangent activation function. and These are different weight matrices, and These are different bias vectors. The input vector.

[0069] Furthermore, there is the experience replay mechanism: the experience replay buffer is designed with a capacity of 150,000 transition samples. Each training session randomly samples 256 samples (batch_size=256) from the buffer for gradient updates. Network training is started after collecting 5,000 samples. Optimization objective: A discount factor γ = 0.99 is used to measure long-term returns. The soft update coefficient controls the synchronization rate of the target network parameters. The update rule is: In the formula, Refers to the parameters of the Critic network. Let the learning rate be set to... , Set the batch size to 256. The soft update factor is set to 0.005. These are the parameters of the current network. Parameters of the target network Training scheduling: The total number of training steps is set to 1,500,000. Gradient updates are performed once for each environment interaction (train_freq=1, gradient_steps=1) to ensure real-time policy optimization.

[0070] Finally, set the initial temperature value to Automatic adjustment via gradient descent: In the formula, is the learning rate for the temperature parameter. The target entropy. This parameter combination, by balancing learning stability and policy performance, enables the model to quickly converge to the optimal policy in microgrid multi-mode control tasks.

[0071] In the specific implementation of the experience replay mechanism, to ensure sample diversity and training stability, the capacity of the experience replay buffer (ReplayBuffer) is designed to be 150,000 transition samples. A "warm start" strategy is adopted during the training startup phase, meaning that after collecting 5,000 samples through interaction with the environment under a random strategy, gradient updates of the network parameters are only initiated.

[0072] Furthermore, to ensure the agent can fully explore and converge to the optimal policy, the total number of training steps is set to 1,500,000. A high-frequency update mode is adopted for training frequency, meaning that a gradient update is performed once for each environmental interaction step (Gradientsteps=1), thus ensuring that the policy network can adapt to dynamic changes in the environment in real time.

[0073] To further improve the generalization ability and robustness of the SAC policy model under different operating conditions, this embodiment introduces a strict random probability mechanism during the initialization phase of the training environment. Specifically, at the beginning of a training episode, the environment randomly selects a point in the historical data as the starting time point to avoid overfitting the model to a specific time series.

[0074] Simultaneously, the probability distribution of the initial operating modes of the microgrid is set: the probability of occurrence of grid-connected mode and islanded mode is each set to 50% (i.e., [0.5, 0.5]). Furthermore, to simulate extreme cases of communication failure or upper-level optimization failure, the probabilities of scenarios with and without day-ahead scheduling guidance are also set to [0.5, 0.5]. Only in the grid-connected mode with effective guidance will the agent's observation vector contain non-zero day-ahead scheduling reference trajectories; otherwise, the relevant dimensions are set to zero. This hybrid training mechanism forces the agent to both learn to track upper-level guidance and independently maintain the safe operation of the system when guidance is lost.

[0075] The online control module 160 is used to input real-time collected operating status data into a trained SAC strategy for online inference to obtain energy control commands. These commands drive the power response of microgrid energy storage and power generation equipment. It is understood that the microgrid operating environment is highly complex and uncertain. Photovoltaic output fluctuates drastically due to weather conditions, and load demand is random. Traditional rule-based control or hourly optimization is insufficient to handle minute-level real-time power fluctuations and sudden grid-connected / islanded mode switching. Through online inference, the system can utilize the adaptive capabilities of deep reinforcement learning strategies to quickly provide optimal action responses based on real-time collected status data. While ensuring system operational safety and robustness, it also considers long-term economic efficiency by tracking day-ahead guidance signals, thereby achieving efficient and intelligent resource allocation.

[0076] In a specific example of this application, the online control module includes: a state standardization unit, used to perform Z-Score standardization on real-time operating state data to obtain a standard observation vector matching the input of the policy network; a policy inference unit, used to input the standard observation vector into the policy network of the trained SAC policy for deterministic inference to obtain a normalized action value; and an instruction generation unit, used to linearly map the normalized action value back to the physical power range, and execute islanded forced zeroing logic and safety boundary clamping according to the grid connection mode in the real-time operating state data to obtain energy control instructions.

[0077] Specifically, firstly, the state standardization unit performs normalization on the real-time operational state data. The system collects real-time operational state data from sensors and uses pre-saved training environment statistics (i.e., the mean vector and standard deviation vector of the training set data) to perform Z-score standardization on the real-time physical quantities to eliminate dimensional differences and match the input distribution of the neural network. The standardization calculation formula is as follows: In the formula, Represents the standardized first Each state component Represents the original physical state value. and These represent the mean and standard deviation of the training set for the corresponding dimensions, respectively. To prevent numerical stability constants with a denominator of zero, the processed components are assembled into tensors in a preset order to form a standard observation vector.

[0078] Secondly, the policy inference unit performs policy network inference based on standard observation vectors. The system inputs the standard observation vectors into the actor network within the trained SAC policy. During inference, to ensure the determinism and stability of control commands, the system directly extracts the mean of the policy distribution as the action output, rather than performing random sampling. The inference calculation formula is as follows: In the formula, The output is a normalized action vector. The mean function of the Gaussian distribution output by the actor network. Represents the standard observation vector. This is the activation function, used to constrain the output within the valid range.

[0079] Finally, the instruction generation unit performs action denormalization, mode adaptation logic, and safety checks to generate energy management instructions. The system linearly maps the normalized action values ​​back to the physical power range of the corresponding device; the denormalization formula is as follows: In the formula, For the first Power command value for each device For the network output components of the corresponding dimension, and These are the preset upper and lower limits of the physical power for the device. After the initial mapping is completed, the system performs logical corrections based on the grid connection mode in the real-time operating status data: if the current mode is isolated, the system forcibly sets the grid interaction power command to zero and rebalances the power gap by adjusting the energy storage or generator output; at the same time, the system performs safety clamp checks on key boundaries such as battery state of charge to ensure that the generated energy management commands do not cause the device to operate beyond its limits, and finally the drive module drives the microgrid energy storage and power generation equipment to perform power response.

[0080] Furthermore, to achieve efficient integration and flexible deployment of the control system in real-world microgrid scenarios, the system also includes a deployment interface unit. This unit is designed with a high-performance asynchronous communication interface based on the FastAPI framework to integrate the functions of the aforementioned modules. Through this interface, the system supports continuous multi-day operation and constructs a collaborative architecture of "cloud-based decision-making and edge execution": the heavy workload of LLM inference and day-ahead optimization calculations are executed on the cloud server, while real-time SAC policy inference and command issuance are deployed on edge computing nodes, thereby outputting real-time status and rapidly responding to the millisecond-level control needs of the microgrid.

[0081] In one embodiment of this application, the energy management method of this application is verified using a standardized microgrid dataset containing a variety of typical scenarios (normal, extreme weather, high load).

[0082] Experimental setup: A microgrid environment was constructed based on the Gymnasium framework, and the dataset was divided into training and test sets in an 8:2 ratio. The lower-level controller adopted the SAC algorithm (three-layer MLP, 256 neurons, 10-minute decision granularity), the upper-level optimizer implemented an hourly day-ahead scheduling MIQP model based on CVXPY, and the LLM agent system was used to transform the qualitative preferences of human operators into quantitative objective function weights.

[0083] The comparison methods include: Baseline 1 (MIQP optimization only + rule control), Baseline 2 (SAC lower-level scheduling only, no upper-level guidance), and Baseline 3 (economic MIQP + SAC tracking, no preference). Tests cover normal, extreme weather, and high-load scenarios. A "reduce diesel engine usage" preference is introduced for specific scenarios, and grid mode (grid-connected to islanded) switching events are simulated. Evaluation metrics include total operating cost, preference achievement rate, total cost over the entire cycle after switching, and average power deviation; combined with... Figure 3 As can be seen from the data in Table 1, the two-layer collaboration is effective: the cost of Baseline 3 is significantly lower than that of Baseline 2 (Table 1), which verifies the value of MIQP's economic optimization guidance SAC. Figure 3 This intuitively demonstrates that, under the two-layer collaborative architecture, the lower-layer SAC controller can closely track the globally economically optimal trajectory given by the upper-layer day-ahead scheduler, avoiding the short-sightedness problem of pure reinforcement learning in long-term planning.

[0084] Table 1. Comparison of average total operating costs of different methods in various scenarios. Table 2 Comparison of the achievement rate and key indicators of the method in this application under the preference of "reducing diesel engine use". Table 3. Comparison of whole-cycle performance after grid mode switching (from grid connection to islanding) The examples demonstrate that: 1) Two-layer collaboration is effective: Baseline 3 has a significantly lower cost than Baseline 2 (Table 1), verifying the value of MIQP's economic optimization guidance SAC; 2) Preference embedding is successful and economical: The method in this application (Table 2) achieves significant preference (12.7% reduction in diesel engine usage) while the total cost is not significantly higher than the unbiased Baseline 3, proving that after LLM quantification of preferences and embedding them into MIQP, the solver can find a new optimal solution that balances preference and economy; 3) Excellent dynamic adaptability and robustness: such as Figure 4As shown, after introducing stochastic grid mode switching and stochastic day-ahead scheduling guidance during training, the model's reward curve exhibits a steady upward trend and convergence. Combined with Table 3, it can be seen that the proposed method demonstrates the lowest total cycle cost and the smallest post-switching power deviation after sudden grid mode switching, significantly outperforming pure optimization methods, thus verifying the effectiveness of the state-space design and robust training mechanism.

[0085] In summary, the microgrid energy management system based on large language models and deep reinforcement learning according to the embodiments of this application is explained. It utilizes a closed-loop system consisting of a generation agent and an evaluation agent to transform the fuzzy qualitative preferences of human operators into precise quantitative objective function weights through iterative generation and auditing mechanisms. This is then used to train a preference quantization adapter to achieve real-time parsing of user control intentions. At the execution level, this application uses a day-ahead optimization module to ensure the global economy of the system within the future prediction window through mixed-integer programming. Simultaneously, the generated scheduling plan is introduced as a macro-level guidance signal into the microgrid simulation environment to drive the learning of the lower-level SAC strategy, thereby achieving deep synergy between long-term economic scheduling and short-term millisecond-level response across multiple time scales.

[0086] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted; furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted; furthermore, the steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above-described embodiments are merely illustrative of several implementation methods of this disclosure, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent for the embodiments of this disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this disclosure, and these all fall within the protection scope of the embodiments of this disclosure. Therefore, the protection scope of the embodiments of this disclosure should be determined by the appended claims. As described above, although this application has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the application itself. Various changes in form and detail can be made without departing from the spirit and scope of this application as defined by the appended claims.

[0087] The present application and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present application. The actual structure is not limited to this. In conclusion, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the present application, such design should fall within the protection scope of the present application.

Claims

1. A microgrid energy management system based on a large language model and deep reinforcement learning, characterized in that, include: The data preprocessing module is used to perform time alignment, outlier interpolation and imputation, and flexible load baseline separation on the collected historical operating data of the microgrid to obtain a standardized dataset and a microgrid simulation environment with integrated physical constraints; The instruction generation module is used to iteratively generate and audit standardized datasets and expert preference texts based on a closed-loop system consisting of generation agents and evaluation agents to obtain instruction fine-tuning datasets containing high-quality dialogue samples. The preference adaptation module is used to supervise the fine-tuning of a pre-trained large language model using an instruction fine-tuning dataset to obtain a preference quantization adapter that can parse user intent. The day-ahead optimization module is used to input received user control commands into a preference quantization adapter to infer the objective function weights, and combine future forecast data to perform global optimization through mixed integer programming to obtain a day-ahead scheduling plan that includes immediate suggested state of charge and power reference values. It includes: a preference parsing unit, which assembles user control commands and future forecast data into prompt words and inputs them into the preference quantization adapter for inference and JSON parsing to obtain the objective function weights containing preference types and weight coefficients; an optimization modeling unit, which defines a sequence of decision variables and a constraint system including power balance and equipment mutual exclusion logic based on future forecast data, and constructs a composite objective function that minimizes the sum of total cost and preference penalty terms to obtain an instance of the optimization problem to be solved; and a scheduling solution unit, which calls the optimization solver to perform branch and bound solutions on the instance of the optimization problem to be solved. If an optimal solution exists, it extracts the key state sequence as macro-level guidance to obtain a day-ahead scheduling plan that includes immediate suggested state of charge and power reference values. The strategy training module is used to introduce the day-ahead scheduling plan as a guiding signal into the microgrid simulation environment and perform deep reinforcement learning training based on the SAC algorithm to obtain the trained SAC strategy. The online control module is used to input real-time collected operating status data into the trained SAC strategy for online inference to obtain energy control commands, which are used to drive the power response of microgrid energy storage and power generation equipment.

2. The microgrid energy management system based on large language model and deep reinforcement learning according to claim 1, characterized in that, The data preprocessing module includes: The multi-source alignment unit is used to perform multi-source data time alignment and resampling on the historical operation data of the microgrid to obtain aligned time-series data. The time series cleaning unit is used to perform anomaly detection and missing value cleaning on the aligned time series data to obtain cleaned time series data. The load separation unit is used to separate the rigid load baseline from the total load of the cleaned time series data based on a preset flexible scaling factor and stitch it to the feature dimension to obtain a standardized dataset. The physical simulation unit is used to define the physical equations and boundary conditions for microgrid operation to obtain a microgrid simulation environment with integrated physical constraints.

3. The microgrid energy management system based on large language model and deep reinforcement learning according to claim 1, characterized in that, The instruction generation module includes: The scene extraction unit is used to extract the corresponding running scene data from the standardized dataset based on the time index of the benchmark key performance indicators, and to concatenate the statistical features of the running scene data with the expert preference text and the benchmark key performance indicators to obtain the context of the generator prompt words. The preference generation unit is used to input the generator prompt context into the generation agent for quantitative reasoning and parsing to obtain a target function proposal containing preference item identifiers and corresponding weight values. The scheduling reconfiguration unit is used to reconfigure the day-ahead scheduling model by taking the objective function proposal as an additional penalty term and solving it to obtain the corresponding optimized scheduling trajectory and new key performance indicators. The preference audit unit is used to conduct multi-dimensional preference validity audits on new key performance indicators and benchmark key performance indicators to obtain audit feedback results. The data synthesis unit is used to combine the runtime scenario data and expert preference text into prompt words and the objective function proposal as a response to obtain the instruction fine-tuning dataset if the audit feedback result is successful.

4. The microgrid energy management system based on large language model and deep reinforcement learning according to claim 1, characterized in that, The preference adaptation module includes: The data formatting unit is used to format the instruction fine-tuning dataset using a standard dialogue template and perform word segmentation encoding to obtain a serialized training batch containing the input index sequence and only retaining the labels of the response portion; LoRA injection units are used to freeze the original weight parameters of a pre-trained large language model and inject a low-rank adapter containing a low-rank matrix and scaling factor into the attention layer side path to obtain the LoRA-enhanced model architecture. The adapter training unit is used to input the serialized training batches into the LoRA augmentation model architecture for forward inference, calculate the cross-entropy loss between the predicted distribution and the true label, and train the preference quantization adapter by updating the parameters of the low-rank adapter only.

5. The microgrid energy management system based on large language model and deep reinforcement learning according to claim 1, characterized in that, The policy training module includes: The state observation unit is used to read real-time physical quantities and mode flags from the microgrid simulation environment and splice them with the guidance reference values ​​extracted synchronously from the day-ahead scheduling plan to obtain a high-dimensional integrated state observation vector. The reward calculation unit is used to calculate a single-step reward value consisting of negative operating cost, physical constraint violation penalty and mean square tracking error penalty to the guidance reference value after the agent outputs an action and receives feedback, and store it in the experience playback buffer. The policy optimization unit is used to sample transfer samples from the experience replay buffer, update the commentator network parameters by minimizing the Bellman residual, and update the policy network parameters by maximizing the weighted sum of action value and policy entropy to iteratively optimize and obtain the trained SAC policy.

6. The microgrid energy management system based on large language model and deep reinforcement learning according to claim 1, characterized in that, The online control module includes: The state normalization unit is used to perform Z-Score normalization on real-time running state data to obtain the standard observation vector for the input of the matching strategy network. The policy inference unit is used to perform deterministic inference on the policy network of the trained SAC policy by inputting the standard observation vector into it in order to obtain the normalized action value. The instruction generation unit is used to linearly map the normalized action value back to the physical power range, and execute the islanded grid forced zeroing logic and safety boundary clamping according to the grid connection mode in the real-time operating status data to obtain energy management instructions.

7. The microgrid energy management system based on large language model and deep reinforcement learning according to claim 3, characterized in that, The preference auditing unit is further used for: Identify target preference terms from the objective function proposal and define the non-target indicators in the new key performance indicators and the benchmark key performance indicators as the set of related indicators. Use the weighted Euclidean distance formula to calculate the ratio of the relative change of the set of related indicators to the relative change of the target preference terms to obtain the global coupling perturbation degree characterizing the system sacrifice rate. The new key performance indicators and the benchmark key performance indicators are compared by a preference-cost dynamic thermodynamic elasticity estimate to obtain the economic trade-off elasticity index. The nonlinear decision boundary is determined based on the sigmoid activation function and the perturbation penalty threshold; Based on the nonlinear decision boundary, a weighted score is applied to the economic exchange elasticity index and the global coupling mutual interference degree, and the semantic direction consistency of the expert preference text is combined to obtain the audit feedback results.

Images