Virtual power plant scheduling method based on large language model and deep reinforcement learning

Abstract

The invention discloses a virtual power plant scheduling method based on a large language model and deep reinforcement learning, and belongs to the technical field of virtual power plant scheduling. Comprising the following steps: constructing a virtual power plant multi-agent cloud edge collaborative scheduling framework based on large language model driving; predicting wind power, photovoltaic power and load power based on a large language model; constructing a mathematical model of virtual power plant optimization scheduling; converting the virtual power plant optimization scheduling model into a Markov game process in combination with a large language model; performing initialization training on the strategy network of the edge layer intelligent agent by adopting imitation learning to obtain a pre-trained edge layer intelligent agent strategy network; and based on the pre-training strategy network of the boundary layer intelligent agent, combining with a large language model and adopting an improved multi-agent near-end strategy optimization algorithm to solve a scheduling strategy.

Description

Technical Field

[0001] The present invention belongs to the field of virtual power plant scheduling, and specifically relates to a virtual power plant scheduling method based on a large language model and deep reinforcement learning. Background Art

[0002] With the continuous optimization of the global energy structure and the introduction of carbon reduction targets, distributed resources represented by wind energy, solar energy, distributed gas turbines, and energy storage equipment have gradually been widely used in power systems. Distributed resources are clean, efficient, and flexible, and can effectively promote energy transformation and sustainable development. However, the widespread access to distributed resources also brings problems of intermittency, randomness, and uncertainty, which poses huge challenges to the stable operation and optimized scheduling of traditional power grids, resulting in reduced energy utilization efficiency, increased equipment operating costs, and even resource waste such as wind and solar power abandonment.

[0003] To address these challenges in operating distributed resources, virtual power plant (VPP) technology has emerged. Leveraging advanced communication and control technologies, VPPs integrate a large number of distributed resources and loads into a controllable whole, forming a unified and coordinated power supply unit. This coordinated control approach can effectively improve the utilization efficiency of distributed resources, enhance the stability and economic efficiency of power system operations, and thus reduce system operating costs and environmental impact.

[0004] Traditional virtual power plant scheduling strategies typically employ mathematical optimization methods, such as mixed-integer linear programming, nonlinear programming, and dynamic programming. While these approaches can solve optimization problems in specific scenarios, they suffer from complex modeling, low computational efficiency, and difficulty in making real-time decisions when dealing with the highly random and uncertain nature of distributed resources and loads. Furthermore, traditional methods require high model accuracy. Once the actual operating environment changes, the model's effectiveness and robustness will significantly decline, severely restricting the practical application of virtual power plants.

[0005] To address the shortcomings of traditional optimization methods, the recently emerging deep reinforcement learning (DRL) technique offers an effective solution. Deep reinforcement learning automatically learns decision-making strategies through the interaction between agents and their environment, avoiding complex mathematical modeling and possessing strong adaptability and self-learning capabilities. However, most current deep reinforcement learning methods for virtual power plant scheduling primarily employ a single-agent reinforcement learning architecture. This approach exhibits significant limitations when faced with the complex interactions of multiple heterogeneous distributed resources and loads, making it difficult to balance the local decisions of each unit with the global objective, thus limiting optimization effectiveness. The emergence of multi-agent deep reinforcement learning (MADRL) algorithms offers a more advantageous solution for virtual power plant scheduling, as they can more effectively coordinate the interactions between various types of agents, significantly improving overall system performance. However, limited research in this area has been conducted in the field of virtual power plant scheduling.

[0006] 1. After searching, a Chinese invention patent with publication number CN117112202A proposes a distributed resource scheduling method for a virtual power plant based on deep reinforcement learning, belonging to the field of power technology. This invention first proposes a virtual power plant scheduling model that aggregates distributed resources, HVAC systems, and other loads using a comprehensive objective function to balance the economic cost of load regulation with the comfort cost of air conditioning. It then describes a constrained Markov decision process for microgrid energy management and proposes a scheduling problem-solving method based on a deep deterministic policy gradient algorithm (Deep Reinforcement Learning). The intelligent agent uses historical and current load demand, photovoltaic output, and temperature data to determine the air conditioning operating power, load regulation, and charge and discharge power. Ultimately, distributed photovoltaic resources are effectively utilized to implement a demand response strategy for load regulation. Intelligent control of the air conditioner is achieved through smart home devices, and the resulting comprehensive cost is only 2.03% lower than the theoretical optimal value.

[0007] The technical comparison between this application and the above-mentioned reference documents is as follows:

[0008] 1. The aforementioned reference document proposes a distributed resource scheduling method for virtual power plants based on the Deep Deterministic Policy Gradient (DDPG) algorithm. This method optimizes the sum of economic cost and air conditioning temperature penalty, constructs a scheduling model that accounts for residents' home status and indoor temperature fluctuations, and uses a single agent to control air conditioning operating power, load reduction, and energy storage charging and discharging. This method generates a virtual user group based on questionnaire data for scheduling simulation, and is primarily applicable to residential energy management.

[0009] This patent constructs a multi-agent virtual power plant cloud-edge collaborative scheduling system consisting of a cloud layer, an edge layer, and an end layer. The scheduling targets include five types of distributed resources: wind power, photovoltaics, gas turbines, energy storage, and adjustable loads. In terms of scheduling algorithms, imitation learning pre-training is combined with an improved multi-agent proximal policy optimization (MAPPO) algorithm for collaborative learning. The training process introduces a large language model to perform semantically driven adjustments to key parameters such as the reward function weight and the policy entropy coefficient, thereby improving the system's stability, explainability, and generalization capabilities. The scheduling objectives comprehensively consider economic, environmental, user satisfaction, and system stability, and are suitable for multi-energy integrated virtual power plants oriented to market-based transactions.

[0010] There are essential differences between the two in terms of system structure and scheduling target setting.

[0011] 2. The aforementioned comparative documents focus on smart home load optimization, emphasizing the balance between air conditioning comfort and economic costs. They are suitable for small-scale, single-scenario residential demand response management. Their optimization objectives are relatively simple, and their intelligent agent structure is monolithic, without addressing edge deployment and resource collaborative control.

[0012] This patent emphasizes source-load-storage synergy and device diversity, proposes multi-agent modeling and a hierarchical collaboration mechanism for distributed resource scheduling, and builds a unified reward representation, state observation, and action modeling framework suitable for industrial, park, and electricity spot market scenarios. The system supports the joint training and flexible deployment of different resource strategies.

[0013] There are essential differences between the two in terms of intelligent body construction and scope of application.

[0014] Second, a search revealed that Chinese invention patent publication number CN117314108A proposes a virtual power plant scheduling method based on reinforcement learning. Specifically, the method comprises: establishing a virtual power plant economic scheduling framework; establishing a virtual power plant economic scheduling model; setting an optimization objective function; establishing a reinforcement learning model based on the proximal policy optimization (PPO) algorithm; and solving the virtual power plant (VPP) economic scheduling based on the PPO algorithm. This invention improves the fully connected network in the original PPO algorithm to a GRU network that considers temporal characteristics, achieving a better optimal economic scheduling strategy for the virtual power plant.

[0015] The technical comparison between this application and the above-mentioned reference documents is as follows:

[0016] 1. The aforementioned comparative document proposes a virtual power plant economic dispatch method based on the Proximal Policy Optimization (PPO) algorithm. By applying deep reinforcement learning to dispatch model training, it enhances the unified optimization capabilities of power generation units, energy storage units, and adjustable loads. This approach belongs to the traditional centralized dispatch structure. Its core lies in the introduction of the GRU network to enhance time series modeling capabilities and achieving optimal dispatch decisions by constructing a multi-constrained objective function. In terms of algorithmic structure, this solution still falls within the scope of centralized reinforcement learning. Its data-driven approach relies primarily on offline historical sampling and is unable to dynamically adapt to local changes in the state of heterogeneous resources in a distributed environment.

[0017] This patent builds a cloud-edge collaborative architecture that combines a large language model with multi-agent deep reinforcement learning, and constructs a multi-layer centralized-distributed scheduling structure for a virtual power plant consisting of a cloud layer, an edge layer, and an end layer, supporting the heterogeneous characteristics of distributed resources and local optimal response. The core includes: (1) using a large language model to fuse structured time series data and unstructured perturbation text information to achieve wind power, photovoltaic and load power prediction; (2) based on Markov game modeling, various distributed resources are constructed as agents, and the policy network is initialized through imitation learning; (3) using an improved multi-agent proximal policy optimization (MAPPO) algorithm to dynamically adjust the policy entropy adjustment factor to enhance the exploration ability and stability of the agent; (4) the cloud-based agent generates optimization suggestions based on the large language model to achieve semantic intervention and human-computer collaborative optimization of the edge-layer agent training process.

[0018] There are essential differences between the two in terms of system architecture, intelligent body construction and technical path.

[0019] 3. The Chinese invention patent with publication number CN118195389A proposes a method and system for evaluating the real-time response capability of a virtual power plant based on cloud-edge collaboration. This invention constructs an overall architecture of a virtual power plant cloud-edge collaborative scheduling system based on the characteristics of the virtual power plant and the data processing requirements. At the edge, corresponding response capability analysis models are established based on the characteristics of various aggregated resources. These models are calculated by the edge cloud and the results are transmitted to the central cloud; in the cloud, a virtual power plant aggregate response capability analysis model is constructed. The central cloud calculates the overall aggregate response capability of the virtual power plant based on the received information on various distributed resources and the established target response capacity.

[0020] The technical comparison between this application and the above-mentioned reference documents is as follows:

[0021] 1. The aforementioned comparative document achieves real-time assessment of the aggregated response capability of a virtual power plant through cloud-edge collaboration. The key technical approach involves collecting the operating status of distributed resources (such as energy storage, electric vehicles, air conditioners, and gas turbines) at the edge, establishing response capability assessment models for various types of equipment, and aggregating the assessment results from each edge in the central cloud to construct an aggregated response capability model for the virtual power plant. This patent focuses on constructing physical-level performance models for metrics such as regulation amplitude, response time, and ramp rate. It emphasizes the ability to estimate and aggregate different types of resources during short-term regulation, serving to assess the ability of virtual power plants to participate in frequency regulation or emergency response tasks.

[0022] This patent builds a three-layer cloud-edge-end architecture, leveraging a large language model to predict wind and solar load power. By modeling the scheduling problem as a Markov game process, it proposes a joint imitation learning-improved MAPPO algorithm. This algorithm centrally trains intelligent agents at the cloud layer and deploys trained edge agents at the edge layer, enabling online policy updates and real-time scheduling responses in a dynamic operating environment. The core innovation lies in the use of a large language model for prediction and natural language-guided reinforcement learning entropy factor regulation, enabling active learning and dynamic optimization of multi-source heterogeneous resource scheduling behaviors.

[0023] There are essential differences between the two in terms of technical solutions and implementation paths.

[0024] 2. The above-mentioned comparative document uses physical modeling and mathematical aggregation to conduct a static evaluation of the response capability of virtual power plants. It is suitable for assisting power grid dispatching agencies in conducting estimation and capability verification, and emphasizes the accuracy of physical indicators of response performance and the integrated calculation of multi-source resources.

[0025] This invention focuses on autonomous scheduling optimization with deep reinforcement learning as the core, emphasizing the adaptive control and intelligent strategy evolution capabilities of the virtual power plant system in complex environments. It not only has response evaluation capabilities, but can also complete scheduling control and continuous optimization, and adapt to energy management scenarios with higher complexity and real-time requirements.

[0026] There are essential differences between the two in their application scenarios. Summary of the Invention

[0027] In view of this, the object of the present invention is to provide a virtual power plant scheduling method based on a large language model and deep reinforcement learning.

[0028] To achieve the above object, the technical solution adopted by the present invention is:

[0029] A virtual power plant scheduling method based on a large language model and deep reinforcement learning includes the following steps:

[0030] S1: Build a multi-agent cloud-edge collaborative scheduling framework for virtual power plants driven by a large language model;

[0031] S2: Based on the large language model, predict wind power, photovoltaic power and load power;

[0032] S3: Construct a mathematical model for optimal scheduling of virtual power plants, including objective functions and constraints;

[0033] S4: Combined with the large language model, the virtual power plant optimization scheduling model is converted into a Markov game process;

[0034] S5: Use imitation learning to initialize the policy network of the edge agent to obtain a pre-trained edge agent policy network;

[0035] S6: Based on the pre-trained policy network of the edge agent, combined with the large language model, an improved multi-agent proximal policy optimization algorithm is used to solve the scheduling strategy.

[0036] As a further improvement of the present invention, in S1, the established virtual power plant multi-agent cloud-edge collaborative scheduling framework driven by a large language model consists of a cloud layer, an edge layer, and an end layer. The cloud layer corresponds to the virtual power plant dispatching center platform, deploys cloud-layer agents, and is responsible for collecting historical data to predict wind and solar power generation and load based on the large language model and guide edge-layer agents to perform centralized training and learning based on the large language model.

[0037] Edge layer: Responsible for pre-processing data collected by the end layer and uploading it to the cloud layer, receiving scheduling signals from the cloud layer, and deploying trained edge layer agents to make scheduling decisions for the controlled distributed resource devices based on local observations;

[0038] The edge layer corresponds to the collection of distributed resource equipment in the virtual power plant system, including wind power, photovoltaics, gas turbines, energy storage, and loads. Through intelligent sensors and monitoring equipment, it collects and uploads the operating status and environmental data of distributed resource equipment, and provides feedback and responses based on the control instructions issued by the edge layer.

[0039] As a further improvement of the present invention, step S2 is specifically as follows:

[0040] S21: Collect structured time series data and unstructured text information for wind power, photovoltaic power, and load forecasting. The structured time series data includes feature data and historical wind power, photovoltaic power, and load power. The unstructured text information includes disturbance information within the forecast area and forecast period.

[0041] S22: The feature data in the structured time series data is processed through data alignment and feature engineering to extract local trend features and statistical description information, and construct time series data prompt words. Unstructured text information is filtered according to time window, geographic location and keyword rules to form a candidate set. The pre-trained large language model is used for semantic analysis and causal reasoning to identify key events that affect wind and solar output and load levels, and construct event prompt words for semantic enhancement.

[0042] S23: Combine the time series data prompt words and the event prompt words and input them into the pre-trained large language model to output the prediction results of wind power, photovoltaic power and load power;

[0043] S24: Processing the prediction results, including denormalization, linearization, and time scale unification of the output results, and outputting prediction curves of wind power, photovoltaic power, and load power within the future target period;

[0044] S25: Perform error evaluation on the prediction curve, using mean absolute error, root mean square error, and determination coefficient to evaluate the prediction performance. If the prediction error exceeds the preset threshold, adjust the data processing parameters, and at the same time, backtrack the text information of the corresponding time period, update the semantic prompt words and reasoning logic, and re-predict until the prediction accuracy requirements are met.

[0045] As a further improvement of the present invention, step S3 is specifically as follows:

[0046] S31: Constructing the cost function of the virtual power plant dispatch process;

[0047] Cost function of a gas turbine By fuel cost and pollutant emission costs The expression is as follows:

[0048] (1)

[0049] Where, is the power generation of the gas turbine at time t, a, b, c are the fuel cost coefficients of the gas turbine, and M is the type of pollutants emitted, including CO2, SO2, and NO x , and are the unit emission cost and emission factor of pollutant m respectively.

[0050] User load is divided into curtailable load and non-adjustable load, and the cost function Compensation of costs by load reduction and user satisfaction costs Composition, among which, building a segmented compensation mechanism can reduce load compensation costs , the expression is as follows:

[0051] (2)

[0052] Constructing user comfort evaluation indicators Calculating the cost of user satisfaction , the expression is as follows:

[0053] (3)

[0054] Where, Compensation price reduction for the user's unit, The time-of-use electricity purchase price is is the load reduction amount that the user can reduce at time t, The load reduction compensation coefficient is the load reduction range, which is divided into two parts: shallow reduction area and deep reduction area. In the shallow reduction area, the reduction compensation coefficient is a fixed value. , in the deep curtailment zone, the curtailment compensation coefficient is a quadratic function related to the curtailment ratio. is the maximum amount of load that can be reduced at time t, is the satisfaction cost coefficient of the load;

[0055] The cost function of energy storage is composed of maintenance costs, which can be expressed as follows:

[0056] (4)

[0057] Where, is the charging and discharging cost coefficient of energy storage, is the charging and discharging power of the energy storage at time t;

[0058] The cost function of wind power generation is composed of the maintenance cost of wind power generation, which is expressed as follows:

[0059] (5)

[0060] Where, is the power generated by the wind turbine at time t, is the wind power generation cost coefficient;

[0061] The cost function of photovoltaic power generation is composed of the maintenance cost of photovoltaic power generation, and the expression is as follows:

[0062] (6)

[0063] Where, is the power generated by photovoltaic power generation at time t, is the photovoltaic power generation cost coefficient;

[0064] The cost of interacting with the market is expressed as follows:

[0065] (7)

[0066] Where, and are the electricity selling price and the electricity purchasing price, Indicates selling electricity to the market. Indicates purchasing electricity from the market;

[0067] S32: Construct the virtual power plant dispatch objective function with the goal of minimizing the total cost:

[0068] (8)

[0069] S33: Constraints for building a virtual power plant dispatch model

[0070] Load balancing constraints:

[0071] (9)

[0072] Gas turbine constraints:

[0073] (10)

[0074] Where, and are the maximum and minimum values ​​of the gas turbine power generation respectively;

[0075] Energy storage constraints:

[0076] (11)

[0077] Where, is the maximum value of energy storage charging and discharging power, and are the maximum and minimum values ​​of energy storage SOC respectively, is the state of charge of the energy storage at time t, is the state of charge of the energy storage at time t-1, is the charging and discharging efficiency of energy storage, is the capacity of energy storage;

[0078] Wind power generation and photovoltaic constraints:

[0079] (12)

[0080] Where, is the maximum power generation of wind power at time t, is the maximum power generation of photovoltaic power at time t;

[0081] Adjustable load restraint:

[0082] (13).

[0083] As a further improvement of the present invention, in step S4, the virtual power plant as a whole is regarded as a multi-agent system with cloud-edge collaboration. The virtual power plant scheduling model established in S3 is converted into a Markov game process by combining the large language model. The state space, action space, and reward function of each edge-layer agent are designed as follows:

[0084] S41. Design of state space and action space of each edge layer agent;

[0085] The state space and action space of the gas turbine agent are defined as follows:

[0086] (14)

[0087] The state space and action space of the energy storage agent are defined as follows:

[0088] (15)

[0089] The state space and action space of the load-reducing agent are defined as follows:

[0090] (16)

[0091] The state space and action space of the wind power generation agent are defined as follows:

[0092] (17)

[0093] The state space and action space of the photovoltaic power generation agent are defined as follows:

[0094] (18)

[0095] S42. Design of the reward function of the edge agent. The reward function of the edge agent includes the negative value of the objective function in the optimization scheduling model. and restraint punishment The two parts are expressed as follows:

[0096] (19)

[0097] (20)

[0098] Where, are the weights of economic item, environmental item, system stability item and user satisfaction item respectively, is the power imbalance penalty coefficient;

[0099] (twenty one)

[0100] Where, They are gas turbine output over-limit penalty, energy storage charge and discharge over-limit penalty, energy storage SOC over-limit penalty, curtailable load reduction over-limit penalty, wind power output over-limit penalty and photovoltaic output over-limit penalty;

[0101] (twenty two)

[0102] Where, The penalty coefficient for gas turbine output exceeding the limit;

[0103] (twenty three)

[0104] Where, The penalty coefficient for energy storage charging and discharging exceeding the limit;

[0105] (twenty four)

[0106] Where, The energy storage SOC over-limit penalty coefficient;

[0107] (25)

[0108] Where, is the penalty coefficient for over-limit reduction of the reducible load;

[0109] (26)

[0110] Where, Penalty coefficient for wind power output exceeding the limit;

[0111] (27)

[0112] Where, The penalty coefficient for photovoltaic output exceeding the limit;

[0113] As a further improvement of the present invention, the cloud-side agent guides the training of the side-layer agent through multiple rounds of dialogue based on the large language model, including:

[0114] S411: Collect the state actions of each edge layer agent during the scheduling training process and the weights of each indicator in the current reward function to construct a semantic feature set that can describe the system operation status;

[0115] S412: Input the above semantic feature set into a natural language prompt template function to generate a natural language prompt text that can be understood by the large language model. The natural language prompt template function is constructed as follows:

[0116] (28)

[0117] Where b is the preset background and q is the question.

[0118] S413: The natural language prompt text generated in S412 is input as semantics into the pre-trained large language model, and the large language model outputs the weights between the reward items in the reward function and corresponding explanations.

[0119] As a further improvement of the present invention, in S5, the cloud agent pre-trains the policy network of the edge agent using imitation learning based on the expert strategy, specifically as follows:

[0120] S51: Constructing expert demonstration dataset based on expert strategy ,The expert strategy includes the dispatcher's historical operation trajectory and the optimal or suboptimal scheduling results obtained by solving the model built by S3 in typical operation scenarios;

[0121] S52: Building a behavioral cloning network model and edge-layer agent strategy network model , for behavioral cloning network parameters Perform supervised learning pre-training to minimize the difference between its output action and the expert action The difference between , where the loss function is constructed as follows:

[0122] (29)

[0123] S53: Update the behavior cloning network parameters using the Adam optimizer To minimize the imitation loss, save the behavior cloning network parameters after pre-training ;

[0124] S54: Behavior cloning network parameters after imitation learning As the initial policy network parameters of the reinforcement learning main training stage, it enters the reinforcement learning main training process.

[0125] As a further improvement of the present invention, in S6, based on the edge agent policy network pre-trained in S5, the cloud agent combines the large language model with the improved MAPPO algorithm to train the edge agent. The specific training process is as follows:

[0126] S61: Build the value network of the edge-layer agent, determine the total scheduling period T and the maximum number of training rounds max_episode, initialize the experience pool and the neural network parameters, policy network parameters, and value network parameters of each edge-layer agent, read data from the training dataset in each episode, and initialize the basic system parameters;

[0127] S62: At each time step t, the edge agent observes the current state Input the policy network to get the action distribution at this time step ;

[0128] S63: Each edge layer agent takes the sampled action Execute scheduling operations to interact with the environment and obtain the state of the next time step , calculate the immediate reward ;

[0129] S64: Change the current state ,action , reward value and the next state Composition of empirical data samples Stored in the experience replay buffer and used for model training, at each time step, the state vectors of all agents Splice into a joint state vector , and input the centralized value network to calculate the current system state value estimate , and calculate the value of the next state , using the generalized dominance estimation method to calculate the dominance value of each agent ;

[0130] S65: Use the improved MAPPO algorithm to update the policy network of each agent, improve the original PPO objective function by introducing a dynamically adjustable policy entropy incentive term, and construct the following policy loss function:

[0131] (30)

[0132] As a further improvement of the present invention, in step S65,

[0133] The policy entropy incentive term has an entropy adjustment factor , entropy regulation factor The cloud-based intelligent agent dynamically adjusts based on the large language model through the following specific steps:

[0134] S651: Every fixed training step Or when it is detected that the strategy of the edge agent is unstable, the strategy entropy value of the current training step of each edge agent is collected , policy entropy change rate and recently The average reward within training steps ;

[0135] S652: Use the collected data to construct a semantic feature set, and input it into a natural language prompt template function to generate a natural language prompt text that can be understood by the large language model. The natural language prompt template function is constructed as follows:

[0136] (31)

[0137] Where b is the preset background and q is the question.

[0138] S653: The natural language prompt text generated by S652 is input as semantics into the pre-trained large language model, and the large language model outputs the policy entropy adjustment factor under the current training step and corresponding explanations;

[0139] S66: Use Adam optimizer to optimize the policy network and value network;

[0140] S67: Update the time step, repeat the above process, and determine whether the current number of training rounds n reaches the maximum number of training rounds max_episode or whether it converges. If it reaches the maximum number of training rounds max_episode or converges, save the network model parameters and end the training, and send the saved network model parameters to the edge layer. Otherwise, start a new round of training from step S62.

[0141] Compared with the prior art, the present invention has the following beneficial effects:

[0142] 1. By building a cloud-edge-end collaborative scheduling architecture, various distributed resource intelligent entities can coordinate responses based on local perception, enabling the virtual power plant to effectively balance distributed control and global optimization, reduce communication costs, and improve the overall scheduling efficiency and stability of the virtual power plant.

[0143] 2. The dispatching model comprehensively considers factors such as operating costs, carbon emissions, system stability and user satisfaction, effectively reducing system operating costs while taking into account user response, power supply quality and environmental protection.

[0144] 3. Use the imitation learning-improved MAPPO joint algorithm to train intelligent agents. During the training process, use the large language model to dynamically adjust the reward function weight and policy entropy coefficient to achieve semantically driven regulation of multi-agent reinforcement learning strategies, thereby improving the convergence speed of the training process, the exploreability of the intelligent agents, and the interpretability of the reinforcement learning model.

[0145] 4. It has good scalability and applicability, and is suitable for various scenarios such as industrial parks, distributed energy stations, and electricity spot markets. It can be flexibly expanded to virtual power plant systems of different sizes and types. BRIEF DESCRIPTION OF THE DRAWINGS

[0146] Figure 1It is the overall flow chart of the present invention;

[0147] Figure 2 This is a diagram of the cloud-edge collaborative scheduling framework of a virtual power plant based on multi-agent deep reinforcement learning;

[0148] Figure 3 This is the training flow chart of the imitation learning-improved MAPPO joint algorithm;

[0149] Figure 4 This is a comparison chart of training results of different algorithms. DETAILED DESCRIPTION

[0150] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0151] like Figure 1 As shown, the present invention proposes a virtual power plant scheduling method based on a large language model and deep reinforcement learning, which includes the following steps:

[0152] S1: Build a multi-agent cloud-edge collaborative scheduling framework for virtual power plants driven by a large language model;

[0153] like Figure 2 As shown in Figure 2, the multi-agent cloud-edge collaborative scheduling framework for virtual power plants driven by a large language model consists of a cloud layer, an edge layer, and a terminal layer.

[0154] Cloud layer: corresponds to the virtual power plant dispatch center platform, deploying cloud-based intelligent agents. The cloud-based intelligent agents are responsible for collecting historical data, predicting wind and solar power generation and load based on the large language model, and guiding the edge-layer intelligent agents to conduct centralized training and learning based on the large language model.

[0155] Edge layer: Responsible for pre-processing data collected by the end layer and uploading it to the cloud layer, receiving scheduling signals from the cloud layer, and deploying trained edge layer agents to make scheduling decisions for the controlled distributed resource devices based on local observations;

[0156] The edge layer corresponds to the collection of distributed resource equipment in the virtual power plant system, including wind power, photovoltaics, gas turbines, energy storage, and loads. Through intelligent sensors and monitoring equipment, it collects and uploads the operating status and environmental data of distributed resource equipment, and provides feedback and responses based on the control instructions issued by the edge layer.

[0157] S2: Based on the large language model, predict wind power, photovoltaic power and load power;

[0158] Step S2 is specifically as follows:

[0159] S21: Collect structured time series data and unstructured text information for wind power, photovoltaic power, and load forecasting. The structured time series data includes feature data and historical wind power, photovoltaic power, and load power. The unstructured text information includes disturbance information within the forecast area and forecast period.

[0160] S22: Structured time series feature data is processed through data alignment and feature engineering to extract local trend features and statistical description information, and time series data prompt words are constructed. Unstructured text information is filtered according to time windows, geographic locations, and keyword rules to form a candidate set. A pre-trained large language model is used to perform semantic analysis and causal reasoning, identify key events that affect wind and solar output and load levels, and construct event prompt words for semantic enhancement.

[0161] S23: Combine the time series data prompt words and the event prompt words to construct a comprehensive prompt word and input it into the pre-trained large language model. The large language model outputs the prediction results of wind power, photovoltaic power and load power.

[0162] S24: Processing the prediction results, including denormalization, linearization and time scale unification of the output data, and outputting prediction curves of wind power, photovoltaic power and load power in the future target period;

[0163] S25: Evaluate the error of the prediction curve and use indicators such as mean absolute error, root mean square error, and determination coefficient to evaluate the prediction performance. If the prediction error exceeds the preset threshold, adjust the data processing parameters, and at the same time, backtrack the text information of the corresponding time period, update the semantic prompt words and reasoning logic, and re-predict until the prediction accuracy requirements are met.

[0164] S3: Based on S1, a mathematical model for optimal scheduling of virtual power plants is constructed;

[0165] Step S3 is as follows:

[0166] S31: Constructing the cost function in the virtual power plant dispatch process;

[0167] (1)

[0168] Where, is the power generation of the gas turbine at time t, a, b, c are the fuel cost coefficients of the gas turbine, and M is the type of pollutants emitted, including CO2, SO2, and NO x , and are the unit emission cost and emission factor of pollutant m respectively.

[0169] User load is divided into curtailable load and non-adjustable load, and the cost function Compensation of costs by load reduction and user satisfaction costs Composition, among which, building a segmented compensation mechanism can reduce load compensation costs , the expression is as follows:

[0170] (2)

[0171] Constructing user comfort evaluation indicators Calculating the cost of user satisfaction , the expression is as follows:

[0172] (3)

[0173] Where, Compensation price reduction for the user's unit, The time-of-use electricity purchase price is is the load reduction amount that the user can reduce at time t, The load reduction compensation coefficient is the load reduction range. The load reduction range is divided into two parts: shallow reduction area and deep reduction area. In the shallow reduction area, the reduction compensation coefficient is a fixed value. , in the deep curtailment zone, the curtailment compensation coefficient is a quadratic function related to the curtailment ratio. is the maximum amount of load that can be reduced at time t, is the satisfaction cost coefficient of the load;

[0174] The cost function of energy storage is composed of maintenance costs, which can be expressed as follows:

[0175] (4)

[0176] Where, is the charging and discharging cost coefficient of energy storage, is the charging and discharging power of the energy storage at time t;

[0177] The cost function of wind power generation is composed of the maintenance cost of wind power generation, which is expressed as follows:

[0178] (5)

[0179] Where, is the wind power generation cost coefficient;

[0180] The cost function of photovoltaic power generation is composed of the maintenance cost of photovoltaic power generation, and the expression is as follows:

[0181] (6)

[0182] Where, is the photovoltaic power generation cost coefficient;

[0183] The cost of interacting with the market is expressed as follows:

[0184] (7)

[0185] Where, and are the electricity selling price and the electricity purchasing price, Indicates selling electricity to the market. Indicates purchasing electricity from the market;

[0186] S32: Construct the virtual power plant dispatch objective function with the goal of minimizing the total cost;

[0187] (8)

[0188] S33: Constraints for constructing a virtual power plant dispatch model;

[0189] Load balancing constraints:

[0190] (9)

[0191] Gas turbine constraints:

[0192] (10)

[0193] Where, and are the maximum and minimum values ​​of the gas turbine power generation respectively;

[0194] Energy storage constraints:

[0195] (11)

[0196] Where, is the maximum value of energy storage charging and discharging power, and are the maximum and minimum values ​​of energy storage SOC respectively, is the state of charge of the energy storage at time t, is the state of charge of the energy storage at time t-1, is the charging and discharging efficiency of energy storage, is the capacity of energy storage;

[0197] Wind power generation and photovoltaic constraints:

[0198] (12)

[0199] Where, is the maximum power generation of wind power at time t, is the maximum power generation of photovoltaic power at time t;

[0200] Adjustable load restraint:

[0201] (13)

[0202] S4: Based on S3, the virtual power plant optimization scheduling model is converted into a Markov game process by combining the large language model;

[0203] In step S4, the virtual power plant as a whole is regarded as a cloud-edge collaborative multi-agent system, with each controllable distributed resource within it as an agent. The virtual power plant scheduling model established in S3 is converted into a Markov game process by combining the large language model. The state space, action space, and reward function of each edge-layer agent are designed as follows:

[0204] S41. Design of state space and action space of each edge layer agent

[0205] The state space, action space and reward function of the gas turbine agent are defined as follows:

[0206] (14)

[0207] The state space, action space and reward function of the energy storage agent are defined as follows:

[0208] (15)

[0209] The state space, action space, and reward function of the load-reducing agent are defined as follows:

[0210] (16)

[0211] The state space, action space, and reward function of the wind power generation agent are defined as follows:

[0212] (17)

[0213] The state space, action space, and reward function of the photovoltaic power generation agent are defined as follows:

[0214] (18)

[0215] S42. Design of the reward function of the edge layer agent. The reward function of the edge layer agent includes the negative value of the objective function. and restraint punishment The two parts are expressed as follows:

[0216] (19)

[0217] Furthermore,

[0218] (20)

[0219] Where, are the weights of economic item, environmental item, system stability item and user satisfaction item respectively, is the power imbalance penalty coefficient;

[0220] (twenty one)

[0221] Where, They are gas turbine output over-limit penalty, energy storage charge and discharge over-limit penalty, energy storage SOC over-limit penalty, curtailable load reduction over-limit penalty, wind power output over-limit penalty and photovoltaic output over-limit penalty;

[0222] (twenty two)

[0223] Where, The penalty coefficient for gas turbine output exceeding the limit;

[0224] (twenty three)

[0225] Where, The penalty coefficient for energy storage charging and discharging exceeding the limit;

[0226] (twenty four)

[0227] Where, The energy storage SOC over-limit penalty coefficient;

[0228] (25)

[0229] Where, is the penalty coefficient for over-limit reduction of the reducible load;

[0230] (26)

[0231] Where, Penalty coefficient for wind power output exceeding the limit;

[0232] (27)

[0233] Where, The penalty coefficient for photovoltaic output exceeding the limit;

[0234] Furthermore, the cloud-based agent guides the edge agent training based on the large language model, including:

[0235] S411: Collect the state actions of each edge layer agent during the scheduling training process and the weights of each indicator in the current reward function to construct a semantic feature set that can describe the system operation status;

[0236] S412: Input the above semantic feature set into a natural language prompt template function to generate a natural language prompt text that can be understood by the large language model. The natural language prompt template function is constructed as follows:

[0237] (28)

[0238] Where b is the preset background and q is the question.

[0239] S413: The natural language prompt text generated in S412 is input as semantics into the pre-trained large language model, and the large language model outputs the weights between the reward items in the reward function and corresponding explanations.

[0240] S5. Initialize and train the policy network of the edge agent using imitation learning to obtain a pre-trained policy network of the edge agent.

[0241] Specifically, in S5, the cloud agent uses imitation learning based on the expert strategy to pre-train the policy network of the edge agent, including:

[0242] S51: Constructing expert demonstration dataset based on expert strategy ,The expert strategy includes the dispatcher’s historical operation trajectory and the optimal or suboptimal scheduling results obtained by solving the model constructed by S3 in typical operation scenarios;

[0243] S52: Building a behavioral cloning network model and edge-layer agent strategy network model , for behavioral cloning network parameters Perform supervised learning pre-training to minimize the difference between its output action and the expert action, where the loss function is constructed as follows:

[0244] (29)

[0245] S53: Update the behavior cloning network parameters using the Adam optimizer To minimize the imitation loss, save the behavior cloning network parameters after pre-training ;

[0246] S54: Behavior cloning network parameters after imitation learning As the initial policy network parameters of the reinforcement learning main training stage, it enters the reinforcement learning main training process.

[0247] S6, based on the pre-trained policy network of the edge agent, combined with the large language model, uses an improved multi-agent proximal policy optimization algorithm to solve the scheduling strategy;

[0248] Specifically, based on the S5 pre-trained edge agent policy network, the cloud agent combines the large language model with the improved MAPPO algorithm to train the edge agent. The specific training process is as follows:

[0249] like Figure 3 As shown in the figure, the cloud-side agent uses the improved MAPPO algorithm based on the large language model to train the edge-layer agent. The specific training process is as follows:

[0250] S61: Build the value network of the edge-layer agent, determine the total scheduling period T and the maximum number of training rounds max_episode, initialize the experience pool and the neural network parameters, policy network parameters, and value network parameters of each edge-layer agent, read data from the training dataset in each episode, and initialize the basic system parameters;

[0251] S62: At each time step t, the edge agent observes the current state Input the policy network to get the action distribution at this time step ;

[0252] S63: Each edge layer agent takes the sampled action Execute scheduling operations to interact with the environment and obtain the state of the next time step , calculate the immediate reward ;

[0253] S64: Change the current state ,action , reward value and the next state Composition of empirical data samples Stored in the experience replay buffer and used for model training, at each time step, the state vectors of all agents Splice into a joint state vector , and input the centralized value network to calculate the current system state value estimate , and calculate the value of the next state , using the generalized dominance estimation method to calculate the dominance value of each agent ;

[0254] S65: Use the improved MAPPO algorithm to update the policy network of each agent. The improvement introduces a policy entropy incentive term into the original PPO objective function and constructs the following policy loss function:

[0255] (30)

[0256] In step S65, the policy entropy incentive term has an entropy adjustment factor , entropy regulation factor The cloud-based intelligent agent dynamically adjusts based on the large language model through the following specific steps:

[0257] S651: Every fixed training step Or when it is detected that the strategy of the edge agent is unstable, the strategy entropy value of the current training step of each edge agent is collected , policy entropy change rate and recently The average reward within training steps ;

[0258] S652: Use the collected data to construct a semantic feature set, and input it into a natural language prompt template function to generate a natural language prompt text that can be understood by the large language model. The natural language prompt template function is constructed as follows:

[0259] (31)

[0260] Where b is the preset background and q is the question.

[0261] S653: The natural language prompt text generated by S652 is input as semantics into the pre-trained large language model, and the large language model outputs the policy entropy adjustment factor under the current training step and corresponding explanations;

[0262] S66: Use Adam optimizer to optimize the policy network and value network;

[0263] S67: Update the time step and repeat the above process to determine whether the current number of training rounds n reaches the maximum number of training rounds max_episode or whether it converges. If it reaches the maximum number of training rounds max_episode or converges, save the network model parameters and end the training. Send the saved network model parameters to the edge layer. Otherwise, start a new round of training from step S62.

[0264] The edge layer agents are trained by the proposed imitation learning-improved MAPPO joint method, MAPPO method, MADDPG method and QMIX method, as shown in the attached figure. Figure 4 The training curve is shown in the attached Figure 4 It can be seen that the proposed imitation learning-improved MAPPO joint method is significantly better than other methods in terms of convergence speed.

[0265] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent variation based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A virtual power plant scheduling method based on a large language model and deep reinforcement learning, characterized in that: The steps include: S1: Build a multi-agent cloud-edge collaborative scheduling framework for virtual power plants driven by a large language model; S2: Based on the large language model, predict wind power, photovoltaic power and load power; S3: Construct a mathematical model for optimal scheduling of virtual power plants, including objective functions and constraints; S4: Combined with the large language model, the virtual power plant optimization scheduling model is converted into a Markov game process; S5: Initialize and train the policy network of the edge agent using imitation learning to obtain a pre-trained policy network of the edge agent. S6: Based on the pre-trained policy network of the edge agent, combined with the large language model, an improved multi-agent proximal policy optimization algorithm is used to solve the scheduling strategy.

2. The virtual power plant scheduling method based on large language model and deep reinforcement learning according to claim 1 is characterized in that: In S1, the established virtual power plant multi-agent cloud-edge collaborative scheduling framework based on large language model driving consists of cloud layer, edge layer and terminal layer, among which, Cloud layer: corresponds to the virtual power plant dispatch center platform, deploying cloud-based intelligent agents. The cloud-based intelligent agents are responsible for collecting historical data, predicting wind and solar power generation and load based on the large language model, and guiding the edge-layer intelligent agents to conduct centralized training and learning based on the large language model. Edge layer: Responsible for pre-processing data collected by the end layer and uploading it to the cloud layer, receiving scheduling signals from the cloud layer, and deploying trained edge layer agents to make scheduling decisions for the controlled distributed resource devices based on local observations; The edge layer corresponds to the collection of distributed resource equipment in the virtual power plant system, including wind power, photovoltaics, gas turbines, energy storage, and loads. Through intelligent sensors and monitoring equipment, it collects and uploads the operating status and environmental data of distributed resource equipment, and provides feedback and responses based on the control instructions issued by the edge layer.

3. The virtual power plant scheduling method based on large language model and deep reinforcement learning according to claim 1 is characterized in that: Step S2 is specifically as follows: S21: Collect structured time series data and unstructured text information for wind power, photovoltaic power, and load forecasting. The structured time series data includes feature data and historical wind power, photovoltaic power, and load power. The unstructured text information includes disturbance information within the forecast area and forecast period. S22: The feature data in the structured time series data is processed through data alignment and feature engineering to extract local trend features and statistical description information, and construct time series data prompt words. Unstructured text information is filtered according to time window, geographic location and keyword rules to form a candidate set. The pre-trained large language model is used for semantic analysis and causal reasoning to identify key events that affect wind and solar output and load levels, and construct event prompt words for semantic enhancement. S23: Combine the time series data prompt words and the event prompt words and input them into the pre-trained large language model to output the prediction results of wind power, photovoltaic power and load power; S24: Processing the prediction results, including denormalization, linearization, and time scale unification of the output results, and outputting prediction curves of wind power, photovoltaic power, and load power within the future target period; S25: Perform error evaluation on the prediction curve, using mean absolute error, root mean square error, and determination coefficient to evaluate the prediction performance. If the prediction error exceeds the preset threshold, adjust the data processing parameters, and at the same time, backtrack the text information of the corresponding time period, update the semantic prompt words and reasoning logic, and re-predict until the prediction accuracy requirements are met.

4. The virtual power plant scheduling method based on large language model and deep reinforcement learning according to claim 1 is characterized in that: Step S3 is as follows: S31: Constructing the cost function of the virtual power plant dispatch process Cost function of a gas turbine By fuel cost and pollutant emission costs The expression is as follows: （1） Where, is the power generation of the gas turbine at time t, a, b, c are the fuel cost coefficients of the gas turbine, and M is the type of pollutants emitted, including CO2, SO2, and NO x , and are the unit emission cost and emission factor of pollutant m respectively; User load is divided into curtailable load and non-adjustable load, and the cost function Compensation of costs by load reduction and user satisfaction costs Composition, among which, building a segmented compensation mechanism can reduce load compensation costs , the expression is as follows: （2） Constructing user comfort evaluation indicators Calculating the cost of user satisfaction , the expression is as follows: （3） Where, Compensation price reduction for the user's unit, The time-of-use electricity purchase price is is the load reduction amount that the user can reduce at time t, The load reduction compensation coefficient is the load reduction range, which is divided into two parts: shallow reduction area and deep reduction area. In the shallow reduction area, the reduction compensation coefficient is a fixed value. , in the deep curtailment zone, the curtailment compensation coefficient is a quadratic function related to the curtailment ratio. is the maximum amount of load that can be reduced at time t, is the satisfaction cost coefficient of the load; The cost function of energy storage is composed of maintenance costs, which can be expressed as follows: （4） Where, is the charging and discharging cost coefficient of energy storage, is the charging and discharging power of the energy storage at time t; The cost function of wind power generation is composed of the maintenance cost of wind power generation, which is expressed as follows: （5） Where, is the power generated by the wind turbine at time t, is the wind power generation cost coefficient; The cost function of photovoltaic power generation is composed of the maintenance cost of photovoltaic power generation, and the expression is as follows: （6） Where, is the power generated by photovoltaic power generation at time t, is the photovoltaic power generation cost coefficient; The cost of interacting with the market is expressed as follows: （7） Where, and are the electricity selling price and the electricity purchasing price, Indicates selling electricity to the market. Indicates purchasing electricity from the market; S32: Construct the virtual power plant dispatch objective function with the goal of minimizing the total cost: （8） S33: Constraints for building a virtual power plant dispatch model Load balancing constraints: （9） Gas turbine constraints: （10） Where, and are the maximum and minimum values ​​of the gas turbine power generation respectively; Energy storage constraints: （11） Where, is the maximum value of energy storage charging and discharging power, and are the maximum and minimum values ​​of energy storage SOC respectively, is the state of charge of the energy storage at time t, is the state of charge of the energy storage at time t-1, is the charging and discharging efficiency of energy storage, is the capacity of energy storage; Wind power generation and photovoltaic constraints: （12） Where, is the maximum power generation of wind power at time t, is the maximum power generation of photovoltaic power at time t; Adjustable load restraint: （13）。 5. The virtual power plant scheduling method based on large language model and deep reinforcement learning according to claim 1 or 2, characterized in that: In step S4, the virtual power plant as a whole is considered as a cloud-edge collaborative multi-agent system. The virtual power plant scheduling model established in S3 is converted into a Markov game process by combining the large language model. The state space, action space, and reward function of each edge-layer agent are designed as follows: S41. Design of state space and action space of each edge layer agent; The state space and action space of the gas turbine agent are defined as follows: （14） The state space and action space of the energy storage agent are defined as follows: （15） The state space and action space of the load-reducing agent are defined as follows: （16） The state space and action space of the wind power generation agent are defined as follows: （17） The state space and action space of the photovoltaic power generation agent are defined as follows: （18） S42. Design of the reward function of the edge layer agent. The reward function of the edge layer agent includes the negative value of the objective function in the optimization scheduling model. and restraint punishment The two parts are expressed as follows: （19） （20） Where, are the weights of economic item, environmental item, system stability item and user satisfaction item respectively, is the power imbalance penalty coefficient; （21） Where, They are gas turbine output over-limit penalty, energy storage charge and discharge over-limit penalty, energy storage SOC over-limit penalty, curtailable load reduction over-limit penalty, wind power output over-limit penalty and photovoltaic output over-limit penalty; （22） Where, The penalty coefficient for gas turbine output exceeding the limit; （23） Where, The penalty coefficient for energy storage charging and discharging exceeding the limit; （24） Where, The energy storage SOC over-limit penalty coefficient; （25） Where, is the penalty coefficient for over-limit reduction of the reducible load; （26） Where, The penalty coefficient for wind power output exceeding the limit; （27） Where, It is the penalty coefficient for photovoltaic output exceeding the limit.

6. The virtual power plant scheduling method based on large language model and deep reinforcement learning according to claim 5 is characterized in that: The cloud-based agent guides the training of the edge agent through multiple rounds of dialogue based on the large language model, including: S411: Collect the state actions of each edge layer agent during the scheduling training process and the weights of each indicator in the current reward function to construct a semantic feature set that can describe the system operation status; S412: Input the above semantic feature set into a natural language prompt template function to generate a natural language prompt text that can be understood by the large language model. The natural language prompt template function is constructed as follows: （28） In the formula, b is the preset background, q is the question asked; S413: The natural language prompt text generated in S412 is input as semantics into the pre-trained large language model, and the large language model outputs the weights between the reward items in the reward function and corresponding explanations.

7. The virtual power plant scheduling method based on large language model and deep reinforcement learning according to claim 1 or 2, characterized in that: In S5, the cloud agent uses imitation learning based on the expert strategy to pre-train the policy network of the edge agent, as follows: S51: Constructing expert demonstration dataset based on expert strategy ,The expert strategy includes the dispatcher's historical operation trajectory and the optimal or suboptimal scheduling results obtained by solving the model built by S3 in typical operation scenarios; S52: Building a behavioral cloning network model and edge-layer agent strategy network model , for behavioral cloning network parameters Perform supervised learning pre-training to minimize the difference between its output action and the expert action The difference between , where the loss function is constructed as follows: （29） S53: Update the behavior cloning network parameters using the Adam optimizer To minimize the imitation loss, save the behavior cloning network parameters after pre-training ; S54: Behavior cloning network parameters after imitation learning As the initial policy network parameters of the reinforcement learning main training stage, it enters the reinforcement learning main training process.

8. The virtual power plant scheduling method based on large language model and deep reinforcement learning according to claim 1 or 2, characterized in that: In S6, based on the edge agent policy network pre-trained in S5, the cloud agent combines the large language model with the improved MAPPO algorithm to train the edge agent. The specific training process is as follows: S61: Build the value network of the edge-layer agent, determine the total scheduling period T and the maximum number of training rounds max_episode, initialize the experience pool and the neural network parameters, policy network parameters, and value network parameters of each edge-layer agent, read data from the training dataset in each episode, and initialize the basic system parameters; S62: At each time step t, the edge agent observes the current state Input the policy network to get the action distribution at this time step ; S63: Each edge layer agent takes the sampled action Execute scheduling operations to interact with the environment and obtain the state of the next time step , calculate the immediate reward ; S64: Current status ,action , reward value and the next state Composition of empirical data samples Stored in the experience replay buffer and used for model training, at each time step, the state vectors of all agents Splice into a joint state vector , and input the centralized value network to calculate the current system state value estimate , and calculate the value of the next state , using the generalized dominance estimation method to calculate the dominance value of each agent ; S65: Use the improved MAPPO algorithm to update the policy network of each agent, improve the original PPO objective function by introducing a dynamically adjustable policy entropy incentive term, and construct the following policy loss function: （30）。 9. The virtual power plant scheduling method based on large language model and deep reinforcement learning according to claim 8 is characterized in that: In step S65, The policy entropy incentive term has an entropy adjustment factor , entropy regulation factor The cloud-based intelligent agent dynamically adjusts based on the large language model through the following specific steps: S651: Every fixed training step Or when it is detected that the strategy of the edge agent is unstable, the strategy entropy value of the current training step of each edge agent is collected , policy entropy change rate and recently The average reward within training steps ; S652: Use the collected data to construct a semantic feature set, and input it into a natural language prompt template function to generate a natural language prompt text that can be understood by the large language model. The natural language prompt template function is constructed as follows: （31） In the formula, b is the preset background, q is the question asked; S653: The natural language prompt text generated by S652 is input as semantics into the pre-trained large language model, and the large language model outputs the policy entropy adjustment factor under the current training step and corresponding explanations; S66: Use Adam optimizer to optimize the policy network and value network; S67: Update the time step, repeat the above process, and determine whether the current number of training rounds n reaches the maximum number of training rounds max_episode or whether it converges. If it reaches the maximum number of training rounds max_episode or converges, save the network model parameters and end the training, and send the saved network model parameters to the edge layer. Otherwise, start a new round of training from step S62.

Images