Lightweight reinforcement learning micro-grid control method and device based on AI chip

Abstract

The invention discloses a lightweight reinforcement learning micro-grid control method and device based on an AI chip, and the method comprises the steps: obtaining the multi-source operation data of a micro-grid in real time, and constructing a real-time operation state space; obtaining a first reinforcement learning model subjected to lightweight processing; based on AI chip architecture characteristics, calculating and optimizing the first reinforcement learning model, providing a second reinforcement learning model, and deploying the second reinforcement learning model into the AI chip; and the real-time operation state space is analyzed through a second reinforcement learning model, and a micro-grid regulation and control result is given. The first reinforcement learning model is calculated and optimized according to the architecture characteristics of the AI chip to obtain and deploy the second reinforcement learning model, so that the calculation amount, memory occupation and power consumption of the model are remarkably reduced on the premise of ensuring the scheduling precision, and micro-grid intelligent control with low power consumption, high real-time performance and high robustness is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of micro-grid control, and particularly relates to a light-weight reinforcement learning micro-grid control method and device based on an AI chip. BACKGROUND

[0002] With the acceleration of global energy transformation, the proportion of renewable energy in the energy structure is rising, and the construction scale of distributed energy systems, especially micro-grids, is continuously expanding. As a small power system that can operate independently in a local area or in parallel with the public grid, micro-grids usually integrate multiple types of distributed power sources and loads such as photovoltaic, wind power, energy storage, electric vehicle charging stations, etc., and have the advantages of high flexibility, high renewable energy consumption rate, etc. However, the intermittency and volatility of renewable energy output, as well as the randomness and uncertainty of electricity load, make the operation optimization of micro-grids a dynamic problem with multiple variables, multiple constraints and strong coupling, which puts extremely high requirements on the real-time, stability and robustness of the dispatching algorithm.

[0003] Traditional micro-grid dispatching methods mostly rely on mathematical optimization, such as linear programming (LP), nonlinear programming (NLP), mixed integer linear programming (MILP), etc. This kind of method can obtain accurate solution under small-scale, deterministic operating conditions, but when the system scale expands, the operating environment becomes complex, and the input information has uncertainty, the computational complexity increases exponentially, which is difficult to meet the decision-making response demand of seconds or even milliseconds. In addition, these methods usually require accurate system parameters and operating models, which are difficult to adapt to the rapid changes of renewable energy output and load demand.

[0004] Currently, the research on deep reinforcement learning model compression on AI chips is still in the exploratory stage. In the industrial field, some manufacturers have tried to apply model compression technology to energy management systems, but they still remain at the algorithm simulation level, lacking of system adaptation to the characteristics of hardware platforms and the operating requirements of micro-grids.

[0005] The patent application CN120073869A discloses a multi-layer distributed micro-grid control system and method based on edge cloud collaborative light-weight reinforcement learning, comprising: a cloud management unit trains a DQN model using historical environmental data, generates a light-weight DQN model through pruning and quantization, distributes the light-weight DQN model to an edge node unit, and formulates daily and weekly optimization scheduling strategies based on a function layering and layering collaboration strategy; the edge node unit monitors electrical quantities and environmental parameters in real time through sensors, deploys the light-weight DQN model to monitor abnormal conditions and trigger local early warning and emergency strategies, and makes rapid decisions using the light-weight DQN model; a micro-grid subgroup unit collects data from the edge node unit. Through close cooperation of the edge node layer, the micro-grid subgroup layer and the cloud management layer, a distributed collaborative control mode of "edge node real-time decision-making + subgroup collaborative management + cloud global optimization" is realized.

[0006] How to reduce the computational complexity, memory occupation and power consumption of the model under the premise of ensuring scheduling accuracy, and realize low-power, high real-time and high-robustness micro-grid intelligent control is a problem to be solved at present. SUMMARY

[0007] In view of the defects in the prior art, the application provides a light-weight reinforcement learning micro-grid control method and device based on an AI chip, which comprises the following steps: real-time acquisition of micro-grid multi-source operation data and construction of a real-time operation state space; acquisition of a first reinforcement learning model subjected to light-weight processing; calculation optimization of the first reinforcement learning model based on the architecture characteristics of the AI chip, giving of a second reinforcement learning model, and deployment of the second reinforcement learning model to the AI chip; analysis of the real-time operation state space by the second reinforcement learning model, and giving of a micro-grid regulation result. Through calculation optimization of the first reinforcement learning model based on the architecture characteristics of the AI chip to obtain and deploy the second reinforcement learning model, the computational complexity, memory occupation and power consumption of the model are significantly reduced under the premise of ensuring scheduling accuracy, and low-power, high real-time and high-robustness micro-grid intelligent control is realized.

[0008] In the first aspect, the application provides a light-weight reinforcement learning micro-grid control method based on an AI chip, which specifically comprises the following steps: Real-time acquisition of micro-grid multi-source operation data and construction of a real-time operation state space; Acquisition of a first reinforcement learning model subjected to light-weight processing; Calculation optimization of the first reinforcement learning model based on the architecture characteristics of the AI chip, giving of a second reinforcement learning model, and deployment of the second reinforcement learning model to the AI chip; Analysis of the real-time operation state space by the second reinforcement learning model, and giving of a micro-grid regulation result.

[0009] Further, the real-time operation state space includes real-time photovoltaic power generation, real-time wind power generation, real-time energy storage system state of charge, real-time energy storage charging and discharging power, real-time main grid exchange power, real-time predicted load power, real-time bus voltage, real-time system frequency, real-time solar irradiance, real-time wind speed, and real-time ambient temperature.

[0010] Further, the first reinforcement learning model after light processing is determined by the following steps: Historical multi-source operation data is obtained, and a historical operation state space is constructed, wherein the historical multi-source operation data includes historical operation data on the power generation side, historical operation data on the energy storage side, historical operation data on the load side, and historical environmental information; A microgrid reward function is constructed according to power balance, operation cost, and safety constraints of the microgrid; Based on the policy network of the initial reinforcement learning model, the historical operation state space is analyzed to give the microgrid regulation and control actions corresponding to the historical operation state space; Based on the value network of the initial reinforcement learning model, the state-action group is analyzed to give the operation value of the state-action group; The policy network and the value network of the initial reinforcement learning model are iteratively trained in combination with the microgrid reward function and the value network error loss, to obtain the first reinforcement learning model and perform light optimization, thereby obtaining the first reinforcement learning model after light processing.

[0011] Further, the construction of the historical operation state space specifically includes: The historical multi-source operation data is timestamp-aligned to obtain first historical operation data; According to the timestamps in the first historical operation data, missing values are interpolated to obtain second historical operation data; The second historical operation data is normalized to construct the historical operation state space.

[0012] Further, obtaining the first reinforcement learning model after light processing specifically includes: According to a pruning combination strategy, the first reinforcement learning model is structurally compressed, and retrained in combination with a model optimization target to obtain a pruned reinforcement learning model; According to a quantization combination strategy, the pruned reinforcement learning model is quantization-optimized and quantization-aware trained to obtain the first reinforcement learning model after light processing.

[0013] Further, according to the pruning combination strategy, the first reinforcement learning model is structurally compressed, and retrained in combination with a model optimization target to obtain a pruned reinforcement learning model, wherein the pruning combination strategy includes structural pruning and unstructured pruning, and specifically includes: According to the current reinforcement learning model, the importance value of each convolution kernel and the local importance value corresponding to each weight in the convolution kernel are determined. In combination with the preset pruning threshold, the overall importance value of each convolution kernel is judged, and the convolution kernel in the first reinforcement learning model is screened for structured pruning. In combination with the preset pruning threshold, the local importance value corresponding to each weight in the screened convolution kernel is judged, and each weight in the convolution kernel is screened for unstructured pruning. The current weight number and the total number of weights in the current reinforcement learning model are analyzed, and the current reinforcement learning model is retrained in combination with the model optimization target, wherein the current reinforcement learning model is the first reinforcement learning model at different times during pruning. If the pruning ratio reaches the preset ratio threshold, a pruned reinforcement learning model is obtained, wherein the pruning ratio is the ratio of the current weight number to the total number of weights in the first reinforcement learning model.

[0014] Further, according to the quantization combination strategy, the pruned reinforcement learning model is quantized and optimized and quantization-aware training is performed to obtain a lightweight first reinforcement learning model, wherein the quantization combination strategy includes symmetric quantization and asymmetric quantization, and specifically includes: Based on the preset target quantization precision, the maximum absolute value of the model parameters in the pruned reinforcement learning model is combined to determine the quantization scale factor; According to the distribution state of the model parameters in the pruned reinforcement learning model, different quantization strategies are matched, and the model parameters are quantized in combination with the quantization scale factor; Based on the quantized model parameters, the pruned reinforcement learning model is quantization-aware trained in combination with the quantization loss function to obtain a lightweight first reinforcement learning model.

[0015] Further, based on the AI chip architecture characteristics, the first reinforcement learning model is calculated and optimized to obtain a second reinforcement learning model, and the second reinforcement learning model is deployed to the AI chip, specifically including: Based on the AI chip architecture characteristics, the first reinforcement learning model is derived, and the graph optimization instructions are given by using the compilation tool of the AI chip, wherein the graph optimization instructions include operator fusion instructions, constant folding instructions and memory reuse instructions; In combination with the graph optimization instructions, the derived first reinforcement learning model is calculated and optimized to obtain a second reinforcement learning model and deploy it to the AI chip.

[0016] Further, the AI chip includes a tensor operation unit, an on-chip cache, a parallel computing architecture and a low-power design; the graph optimization instructions are given by using the compilation tool of the AI chip, specifically including: Classify the operators in the first reinforcement learning model to obtain a plurality of operator sequences, and the operators in each operator sequence have data dependencies; Fuse the operators in each operator sequence to output operator fusion instructions; Identify the fixed value nodes in the first reinforcement learning model to obtain a plurality of constants of the first reinforcement learning model; Analyze the operation relationship between the plurality of constants to construct a constant operation expression; According to the constant operation expression, the constant operation result is given, and the constant folding instruction is generated to replace the corresponding fixed value node, and the constant folding of the first reinforcement learning model is completed; Using a tensor operation unit, the life cycle of each intermediate tensor in the first reinforcement learning model is obtained, and the intermediate tensors are divided according to the life cycle of the intermediate tensors to obtain a plurality of tensor subsets, wherein the life cycle of the intermediate tensors in each tensor subset does not overlap; Assign a corresponding memory space to each intermediate tensor in each tensor subset, and generate a memory reuse instruction; According to the memory reuse instruction, the current intermediate tensor in the memory space corresponding to the tensor subset is released and the next intermediate tensor is stored.

[0017] Further, the method further comprises: Cache the real-time running state space in the experience buffer area; Based on the preset update period, the update sample set is extracted from the experience buffer area, and the second reinforcement learning model is updated.

[0018] Further, based on the preset update period, the update sample set is extracted from the experience buffer area, and the second reinforcement learning model is updated, specifically including: According to the preset update period, the update sample set is extracted from the experience buffer area; Combine the stability constraint term to construct the total loss function, which is specifically represented as:

[0019] Wherein, is the total loss function, L actor is the policy network loss function, L critic is the value network error loss, λ critic is the balance coefficient corresponding to the value network error loss, L stable is the stability constraint term; Based on the update sample set, the online optimization objective function and the total loss function are combined to train the second reinforcement learning model; The disturbance simulator is integrated, random noise is injected into the microgrid operation state and microgrid regulation action obtained in the second reinforcement learning model training process, and the second reinforcement learning model is updated.

[0020] In a second aspect, the present application also provides a light-weight reinforcement learning microgrid control device based on an AI chip, which adopts the light-weight reinforcement learning microgrid control method based on an AI chip according to any one of the above, and comprises: A data acquisition module is configured to acquire real-time microgrid multi-source operation data and construct a real-time operation state space. A model acquisition module is configured to acquire a light-weight first reinforcement learning model. A calculation optimization module is configured to perform calculation optimization on the first reinforcement learning model based on the architecture characteristics of the AI chip, give a second reinforcement learning model, and deploy the second reinforcement learning model to the AI chip. A result output module is configured to analyze the real-time operation state space through the second reinforcement learning model and give a microgrid regulation result.

[0021] The light-weight reinforcement learning microgrid control method and device based on an AI chip provided by the present application have at least the following beneficial effects: (1) The second reinforcement learning model is obtained and deployed by performing calculation optimization on the light-weight first reinforcement learning model according to the architecture characteristics of the AI chip, the real-time operation state space is analyzed through the second reinforcement learning model, and a microgrid regulation result is given, which significantly reduces the calculation amount, memory occupation and power consumption of the model under the premise of ensuring the scheduling accuracy, and realizes low-power, high real-time and high-robustness microgrid intelligent management and control.

[0022] (2) Through the microgrid reward function, the model will tend to reduce power imbalance, reduce operation cost and avoid violating safety operation constraints during the training process, so as to obtain a scheduling strategy with excellent comprehensive performance.

[0023] (3) The pruning optimization is performed by adopting the pruning combination strategy combining structured pruning and unstructured pruning, and the neuron connections, convolution kernels or channels that contribute less or are irrelevant in the inference process are deleted, so that the parameter amount and calculation amount of the pruned reinforcement learning model are significantly reduced, the inference delay is reduced, and it is helpful to quickly give scheduling instructions in high real-time scenarios such as microgrid island mode and power mutation.

[0024] (4) The real-time performance of the second reinforcement learning model in microgrid scheduling is improved by adopting the methods of operator fusion, constant folding, memory reuse and delay execution during the deployment and use of the second reinforcement learning model.

[0025] (5) By online updating the second reinforcement learning model, the performance improvement of the new search strategy is ensured, and at the same time, the original control mode is not disturbed too much, so as to prevent the microgrid operation from appearing sharp fluctuation. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 A flowchart of the AI chip-based lightweight reinforcement learning microgrid control method provided in the embodiment of the present application is shown in the figure. Figure 2 A flowchart of obtaining the first reinforcement learning model provided in the embodiment of the present application is shown in the figure. Figure 3 A flowchart of performing lightweight processing on the first reinforcement learning model provided in the embodiment of the present application is shown in the figure. Figure 4 A flowchart of performing structural compression on the first reinforcement learning model provided in the embodiment of the present application is shown in the figure. Figure 5 A flowchart of performing quantization optimization on the pruned reinforcement learning model provided in the embodiment of the present application is shown in the figure. Figure 6 A flowchart of deploying the second reinforcement learning model provided in the embodiment of the present application is shown in the figure. Figure 7 A flowchart of updating the second reinforcement learning model provided in the embodiment of the present application is shown in the figure. Figure 8 A structural block diagram of the AI chip-based lightweight reinforcement learning microgrid control device provided in the embodiment of the present application is shown in the figure.

[0027] Among them, 201, data acquisition module; 202, model acquisition module; 203, calculation optimization module; 204, result output module. DETAILED DESCRIPTION

[0028] In order to better understand the above technical solutions, the above technical solutions will be described in detail in conjunction with the drawings of the specification and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0029] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Multiple" generally includes at least two.

[0030] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0031] In recent years, reinforcement learning, as an artificial intelligence method that can automatically learn optimal policies through interaction with the environment, has received increasing attention in microgrid scheduling. Reinforcement learning does not rely on explicit system models but gradually approximates the optimal control policy through iterative cycles of states, actions, and rewards, making it particularly suitable for handling complex systems with nonlinearity, high dimensionality, and strong stochasticity. However, deep reinforcement learning (DRL) faces significant computational demands and high memory consumption in practical deployments, which is especially prominent in embedded scenarios for real-time microgrid scheduling.

[0032] The development of AI chips has made real-time applications of DRL in microgrids possible. AI chips, through dedicated tensor operation units (such as MAC units), on-chip cache (SRAM), parallel computing architecture, and low-power design, can perform high-throughput inference operations under limited energy consumption. However, directly deploying unoptimized DRL models onto AI chips often leads to runtime latency, excessive power consumption, or even deployment failure due to the large size of model parameters, redundant computational paths, and excessive memory access requirements. Therefore, lightweight processing must be performed on the model before inference, including but not limited to methods such as network structure pruning, weight quantization, sparsification, and low-rank decomposition.

[0033] While compressing the model, ensuring the accuracy and stability of microgrid dispatching is a pressing issue. Excessive pruning may reduce the ability to extract key features, and low accuracy in the quantization process may lead to serious deviations in policy decisions. Therefore, a trade-off needs to be struck between compression ratio and performance preservation, and compression strategies should be designed in conjunction with the actual operating characteristics of the microgrid (such as power balance constraints, voltage / frequency stability constraints, and energy storage safety constraints). Furthermore, microgrid operation has extremely high real-time requirements. For example, in islanded mode, when load surges or renewable energy output drops sharply, the dispatching algorithm must calculate new control commands within tens of milliseconds; otherwise, voltage exceeding limits, frequency instability, or even system outages may occur. Therefore, the deployment of the lightweight DRL model on AI chips must not only consider average inference latency but also ensure that the response time meets control requirements in the worst-case scenario.

[0034] This invention proposes a lightweight reinforcement learning microgrid control method based on AI chips, taking into full account the multi-objective optimization requirements of microgrid scheduling, the computational and storage limitations of AI chips, and the principle of preserving accuracy in model compression. The method includes: acquiring multi-source operating data of the microgrid in real time and constructing a real-time operating state space; obtaining a lightweight first reinforcement learning model; optimizing the first reinforcement learning model based on the characteristics of the AI ​​chip architecture to produce a second reinforcement learning model, which is then deployed into the AI ​​chip; and analyzing the real-time operating state space using the second reinforcement learning model to provide microgrid control results. Optimizing the computation based on the AI ​​chip's architectural characteristics significantly reduces the computational load, memory usage, and power consumption of the model while ensuring scheduling accuracy, achieving low-power, high-real-time, and highly robust intelligent management and control of the microgrid.

[0035] like Figure 1 As shown, this embodiment of the invention provides a lightweight reinforcement learning microgrid control method based on an AI chip, and the specific steps are as follows: S101: Acquire multi-source operation data of microgrid in real time and construct a real-time operation status space.

[0036] Specifically, the real-time operating status space includes real-time photovoltaic power generation, real-time wind power generation, real-time state of charge of the energy storage system, real-time energy storage charging and discharging power, real-time grid switching power, real-time predicted load power, real-time bus voltage, real-time system frequency, real-time solar irradiance, real-time wind speed, and real-time ambient temperature.

[0037] It is understandable that microgrid multi-source operation data refers to relevant operation data collected during the operation of a microgrid. This data includes real-time operation data from the generation side, energy storage side, load side, and real-time information about the external environment. Based on the collected microgrid multi-source operation data, a real-time operation state space is constructed, specifically represented as follows:

[0038] Among them, P pv (t) represents the photovoltaic power generation on the power generation side at time t, which is the real-time power output from the photovoltaic array after conversion by the inverter. wind (t) represents the wind power generation on the generating side at time t, which is the real-time power after rectification and inversion by the wind turbine generator. SOC(t) represents the state of charge of the energy storage system on the energy storage side, SOC(t)∈[0,1], where 0 indicates that the energy storage side is completely empty, and 1 indicates that the energy storage side is completely full. P ess (t) represents the energy storage charging and discharging power at time t, P. ess (t) > 0 indicates that the energy storage side is charging, P ess (t) < 0 indicates that the energy storage side is discharging. P grid (t) represents the power exchanged between the load side and the main grid at time t, P grid (t) > 0 indicates purchased electricity, meaning the load side receives power from the main grid, P grid (t) < 0 indicates electricity sales, i.e., power output from the load side to the main grid. P load V(t) represents the predicted load power at time t, obtained using a short-term load forecasting model (such as LSTM or ARIMA). V(t) represents the bus voltage at time t, typically maintained within ±5% of the rated value. f(t) represents the system frequency at time t, maintained within ±0.2 Hz (or 60 Hz). G(t) represents the solar irradiance at time t, used to predict photovoltaic power output. wind (t) represents the wind speed at time t, used to predict wind power output. env (t) The ambient temperature of the external environment affects energy storage efficiency and load demand.

[0039] S102: Obtain the first reinforcement learning model with lightweight processing.

[0040] Furthermore, referring to Figure 2 The first reinforcement learning model, which is lightweight, is determined through the following steps: Acquire historical multi-source operation data and construct a historical operation status space. The historical multi-source operation data includes historical operation data of the power generation side, historical operation data of the energy storage side, historical operation data of the load side, and historical environmental information. Based on the power balance, operating costs, and security constraints of the microgrid, a reward function for the microgrid is constructed. Based on the policy network of the initial reinforcement learning model, the historical operating state space is analyzed, and the microgrid control actions corresponding to the historical operating state space are given. Based on the value network of the initial reinforcement learning model, the state-action group is analyzed, and the operational value of the state-action group is given. By combining the microgrid reward function and the value network error loss, the policy network and value network of the initial reinforcement learning model are iteratively trained to obtain the first reinforcement learning model, which is then optimized to obtain a lightweight first reinforcement learning model.

[0041] The construction of the historical operating state space specifically includes: The first historical operational data is obtained by aligning the timestamps of the historical multi-source operational data. Based on the timestamps in the first historical running data, missing values ​​are imputed to obtain the second historical running data; The second set of historical operation data is normalized to construct a historical operation state space.

[0042] It is understandable that historical multi-source operational data and microgrid multi-source operational data, as well as historical operational state space and real-time operational state space, correspond to each other, differing only in the time of data generation. After obtaining the historical multi-source operational data, the first step is to perform timestamp alignment and missing value imputation. Timestamp alignment ensures that the historical multi-source operational data form the historical operational state space at the same time, obtaining the first historical operational data. Subsequently, missing value imputation is performed on the first historical operational data to obtain the second historical operational data. Missing value imputation can employ linear interpolation, nearest neighbor interpolation, historical mean method, K-nearest neighbor imputation, or model-based predictive imputation to ensure the integrity of the historical operational state space.

[0043] During timestamp alignment, the first step is to select a stable, accurate, and calibrable clock source as the master clock. All other data source timestamps will be aligned to this master clock. In this example, the timestamps of the data source corresponding to historical photovoltaic power generation are selected as the master clock. At the instant each data source generates data, its local timestamp is recorded. The linear relationship between the local clock and the master clock for each data source is calculated. Using the calculated slope and offset, the local timestamp of each data source is converted to the master clock's timestamp. All data source timestamps are aligned to a unified time grid to obtain the first set of historical running data for further processing and fusion.

[0044] To eliminate the dimensional differences between different physical quantities and improve the convergence speed of reinforcement learning model training, historical multi-source running data is normalized. In this example, min-max normalization is used, specifically as follows:

[0045] Where x represents the second historical running data, x min Let x be the minimum value of x in the second historical running data. max x is the maximum value of x in the second historical running data, and x' is the normalized dimensionless value, x'∈[0,1].

[0046] In other examples, if the distribution of historical multi-source runtime data approximates a normal distribution, standardization can be applied, specifically as follows:

[0047] Where μ is the sample mean corresponding to the second historical running data x, and σ is the sample standard deviation corresponding to the second historical running data x. Standardization improves the numerical stability of gradient descent optimization algorithms.

[0048] After data processing, a historical operating state space is constructed, which includes historical photovoltaic power generation, historical wind power generation, historical energy storage system state of charge, historical energy storage charging and discharging power, historical grid switching power, historical predicted load power, historical bus voltage, historical system frequency, historical solar irradiance, historical ambient wind speed, and historical ambient temperature. This historical operating state space is used as input to the initial reinforcement learning model, and the policy network and value network of the initial reinforcement learning model are iteratively trained to obtain the first reinforcement learning model.

[0049] Based on the power balance, operating costs, and security constraints of the microgrid, a microgrid reward function is constructed, specifically expressed as:

[0050] Where, r t_his Let E be the function value corresponding to the microgrid reward function at time t_his. pb (t_his) is the power balance function of the microgrid at time t_his, C op (t_his) represents the operating cost of the microgrid at time t_his, including fuel cost, electricity purchase cost, and energy storage lifetime loss cost. sf (t_his) is the penalty term corresponding to the safety constraint. It is dimensionless and takes a positive value when the bus voltage or frequency exceeds the limit or when the energy storage is overcharged / overdischarged; otherwise, it is 0. P gen(t_his) represents the total power generation of the microgrid at time t_his, α is the weighting coefficient corresponding to the power balance function, β is the weighting coefficient corresponding to the operating cost, and γ sf These are the weighting coefficients corresponding to the safety constraints, used to adjust the relative importance of different optimization objectives.

[0051] By using the microgrid reward function, the model will tend to reduce power imbalance, lower operating costs, and avoid violating safety operation constraints during the training process, thereby obtaining a scheduling strategy with excellent overall performance.

[0052] It is understandable that time t_his is a historical time relative to time t in the real-time runtime state space. In the real-time runtime state space at a future time, time t_his will be transformed into time t.

[0053] Based on the policy network of the initial reinforcement learning model, the historical operating state space is analyzed, and the corresponding microgrid control actions are given, specifically as follows: The initial reinforcement learning model's policy network input is the historical running state space S. t_his Output the microgrid control action 'a' corresponding to the historical operating state space. t_his For continuous control problems, the output is the mean μ. θ (S t_his ) and variance σ θ (S t_his The specific actions are obtained through Gaussian distribution sampling:

[0054] Where θ represents the policy network parameters.

[0055] Based on the value network of the initial reinforcement learning model, the state-action group is analyzed, and its operational value is given, specifically expressed as: the value network of the initial reinforcement learning model inputs the state-action group (S). t_his ,a t_his Output the operational value corresponding to the state action group. ,in, These are the value network parameters of the value network.

[0056] Value network error loss is specifically expressed as:

[0057] Among them, L critic Let E[] be the value network error loss, and E[] be the expectation function. t_his ,a t_his ) represents the state and action group of the current historical moment t_his. The runtime value of the state-action group corresponding to the current historical moment t_his, y t_his Let S be the objective value function. t_his For the current historical moment, a t_his For the action corresponding to the current historical state, r t_his The function value of the microgrid reward function at the current historical moment t_his is given, and γ is the discount factor. To be based on search strategy π θ State S at the next historical moment t_his+1 Corresponding action a t_his+1 The expected value, α ent This is the entropy coefficient, used to encourage the exploration of search strategies. Let π be the network parameter of the target network. θ For search strategy, The state-action group (S) for the next historical moment t_his+1 t_his+1 ,a t_his+1 The corresponding operational value.

[0058] The objective value function is used to calculate the target reward of the value network at the current time step. During training, the value network utilizes the output of the objective network. Compared with the prediction network output in the current value network The differences between them are used for time-difference learning, thereby achieving a stable estimate of the returns.

[0059] The above process ensures that the value network gradually converges as the search strategy is continuously updated, which helps to improve the generalization ability and prediction accuracy of the reinforcement learning model in complex dynamic environments, thereby enhancing the stability of microgrid scheduling.

[0060] By combining the microgrid reward function and the value network error loss, the policy network and value network of the initial reinforcement learning model are iteratively trained to obtain the first reinforcement learning model, which specifically includes: First, initialize the experience playback buffer, then randomly initialize the microgrid operating state S0 in the simulation environment, according to the search strategy π. θ Generate microgrid control actions a t_his The microgrid simulation environment is then input and executed. The microgrid simulation environment returns the microgrid control action a. t The microgrid operating state S corresponding to the next moment t_his+1 and microgrid rewards r t_his , will (S t_his ,a t_his ,r t_his ,S t_his+1The data is stored in the experience replay buffer. When the amount of data in the experience replay buffer exceeds the preset data size B, a batch of data is randomly sampled from it to update the policy network and the value network. This process continues iterating until training converges, i.e., the accumulated microgrid reward r is reached. t The training objective is to minimize the value network error loss, rather than significantly increasing or meeting the preset performance threshold. The resulting first reinforcement learning model is able to generate corresponding scheduling decisions based on the current operating state and environmental conditions of the microgrid.

[0061] The aforementioned microgrid simulation environments include daily load fluctuations (significant peak-to-valley differences) and smooth load scenarios, high and low proportions of renewable energy output scenarios, sudden weather changes leading to a sharp drop in photovoltaic and wind power output scenarios, emergency charging and discharging regulation scenarios for energy storage (such as main grid accidents), and frequency stability control scenarios under islanded operation mode.

[0062] By constructing multiple microgrid simulation environments, the diversity of microgrid simulation environments is achieved, ensuring the generalization ability of the training strategy and enabling the first reinforcement learning model to maintain stable and efficient decision-making performance in complex and ever-changing microgrid operating environments.

[0063] Furthermore, referring to Figure 3 The first reinforcement learning model, which is lightweight, is obtained, specifically including: Based on the pruning and combining strategy, the first reinforcement learning model is structurally compressed and retrained in combination with the model optimization objective to obtain the pruned reinforcement learning model. Based on the quantization combination strategy, the pruning reinforcement learning model is quantized for optimization and quantization-aware training to obtain a lightweight first reinforcement learning model.

[0064] Furthermore, referring to Figure 4 Based on the pruning-combination strategy, the first reinforcement learning model is structurally compressed and retrained in conjunction with the model optimization objective to obtain a pruned reinforcement learning model. The pruning-combination strategy includes structured pruning and unstructured pruning, specifically including: Based on each convolutional kernel and each weight in the current reinforcement learning model, determine the overall importance value of the convolutional kernel and the local importance value corresponding to each weight in the convolutional kernel. Based on the preset pruning threshold, the overall importance value of each convolutional kernel is judged, and the convolutional kernels in the first reinforcement learning model are selected for structured pruning. Based on the preset pruning threshold, the local importance values ​​corresponding to each weight in the filtered convolutional kernel are judged, and each weight in the convolutional kernel is filtered and unstructured pruning is performed. Analyze the current number of weights and the total number of weights in the current reinforcement learning model, and retrain the current reinforcement learning model in combination with the model optimization objective. The current reinforcement learning model is the first reinforcement learning model at different times during the pruning process. If the pruning ratio reaches a preset threshold, a pruned reinforcement learning model is obtained, where the pruning ratio is the ratio of the current number of weights to the total number of weights in the first reinforcement learning model.

[0065] Structured pruning removes components at the kernel, channel, or neuron level, facilitating parallel acceleration on AI chips. Unstructured pruning removes components at the weight level, allowing for finer-grained reduction of model size.

[0066] In one specific implementation, the first reinforcement learning model includes multiple convolutional kernels, and each convolutional kernel includes multiple weights. The local importance value I corresponding to each weight in the convolutional kernel is then... ij Specifically, it is expressed as:

[0067] Among them, w ij For the j-th weight in the i-th convolutional kernel, I ij Let be the local importance value corresponding to the j-th weight in the i-th convolutional kernel, representing the importance of the weight magnitude to the feature contribution.

[0068] The overall importance of the convolution kernel is the average of the absolute values ​​of all weights in the kernel, specifically expressed as:

[0069] Where K is the total number of weights in the i-th convolutional kernel, I i w represents the overall importance value corresponding to the i-th convolutional kernel. ij It represents the j-th weight in the i-th convolutional kernel.

[0070] During the pruning process, convolutional kernels with an overall importance value below the pruning threshold and weights with a local importance value below the pruning threshold are removed. The pruning threshold can be adaptively determined through the validation set performance curve to ensure a balance between compression ratio and performance preservation.

[0071] Analyze the current number of weights and the total number of weights in the current reinforcement learning model, and retrain the current reinforcement learning model. This involves iterative pruning, where the model is retrained after each certain percentage (e.g., 10%) of weights and / or convolutional kernels are removed to restore model accuracy. The iterative pruning process is repeated until the cumulative pruning percentage reaches a preset pruning threshold.

[0072] After each round of pruning, the current reinforcement learning model will be retrained for a certain number of rounds. The model optimization objective is to minimize the weighted sum of policy loss and value loss, specifically expressed as:

[0073] Among them, L total For the model optimization objective, L actor Let L be the loss function of the policy network. critic For the value network error loss, λ critic Let θ be the balance coefficient corresponding to the error loss of the value network, and let θ be the policy network parameter of the policy network. These are the value network parameters of the value network.

[0074] Retraining allows the remaining non-zero weights to refit into the pruned network structure, enabling the current reinforcement learning model to recover or approach the original performance of the first model. By employing a combined pruning strategy that integrates structured and unstructured pruning, pruning optimization removes neurons, convolutional kernels, or channels that contribute little or no to inference. This significantly reduces the number of parameters and computational cost of the pruned reinforcement learning model, lowers inference latency, and facilitates rapid scheduling commands in high-real-time scenarios such as microgrid islanding and power surges. Furthermore, the reduction of redundant computation decreases chip power consumption, contributing to long-term stable operation in energy-constrained control terminals (such as remote edge nodes).

[0075] Furthermore, referring to Figure 5 Based on the quantization combination strategy, the pruned reinforcement learning model is optimized by quantization and trained with quantization awareness to obtain a lightweight first reinforcement learning model. The quantization combination strategy includes symmetric quantization and asymmetric quantization, specifically: Based on the preset target quantization accuracy, and combined with the maximum absolute value of the model parameters in the pruning reinforcement learning model, the quantization ratio factor is determined. Based on the distribution of model parameters in the pruning reinforcement learning model, different quantization strategies are matched and combined with the quantization scaling factor to quantize the model parameters. Based on the quantized model parameters and combined with the quantization loss function, the pruned reinforcement learning model is trained with quantization awareness to obtain the first reinforcement learning model with lightweight processing.

[0076] The quantitative scaling factor is specifically expressed as:

[0077] Where s is the quantization scaling factor, w f Here, n represents the model parameters in the pruning reinforcement learning model, and n is the quantization bit width, which is determined by the target quantization precision. For example, if the target quantization precision is INT8, then n=8.

[0078] If the distribution of model parameters in a pruned reinforcement learning model is symmetrical, then symmetrical quantization is used to quantize the model parameters, combined with a quantization scaling factor, as follows:

[0079] Where s is the quantization scaling factor, w f w is the model parameter in the pruning reinforcement learning model. q The parameters are the quantized model parameters, and round() is the rounding function.

[0080] If the distribution of model parameters in a pruned reinforcement learning model is asymmetric, asymmetric quantization is used to quantize the model parameters in conjunction with a quantization scaling factor, specifically as follows:

[0081] Where s is the quantization scaling factor, w f w is the model parameter in the pruning reinforcement learning model. q Here are the quantized model parameters, z is the zero-point offset, and w max w is the maximum value of the model parameters in the pruned reinforcement learning model. min q represents the minimum value of the model parameters in the pruned reinforcement learning model. max q represents the maximum value of the quantization range corresponding to the target quantization precision. min The minimum value of the quantization range corresponding to the target quantization precision.

[0082] After completing pruning optimization and restoring model accuracy, the model is quantized to convert model parameters and intermediate activation values ​​from high-precision floating-point numbers (such as FP32) to low-precision integers (such as INT8 and INT4) to further reduce storage requirements and computational complexity, thereby improving the inference speed of AI chips and reducing power consumption.

[0083] To reduce the accuracy loss caused by quantization optimization, the pruned reinforcement learning model is trained with quantization awareness based on the quantized model parameters. This is achieved by simulating quantization operations during forward propagation and updating weights based on floating-point gradients during backpropagation. Specifically, the forward propagation uses a pseudo-quantization function, expressed as follows:

[0084] Here, `clip()` is a truncation function used to restrict the quantized model parameters to an integer representation. Quantization-aware training enables pruned reinforcement learning models to gradually adapt to low-bit-precision operations during training, thereby significantly reducing the rate of accuracy degradation during deployment.

[0085] Quantization optimization enables the second reinforcement learning model deployed on the AI ​​chip to run with extremely low latency (<1ms), and can complete real-time power allocation calculations even at high sampling rates (e.g., 100Hz). At the same time, memory usage is significantly reduced, allowing edge control nodes to run multiple tasks (such as simultaneously performing load prediction, state estimation, etc.) with limited storage resources.

[0086] S103: Based on the characteristics of the AI ​​chip architecture, the first reinforcement learning model is computationally optimized to provide a second reinforcement learning model, and the second reinforcement learning model is deployed into the AI ​​chip.

[0087] Specifically, refer to Figure 6 Based on the characteristics of AI chip architecture, the first reinforcement learning model is derived, and graph optimization instructions are given using the AI ​​chip's compilation tools. The graph optimization instructions include operator fusion instructions, constant folding instructions, and memory reuse instructions. By combining graph optimization instructions, the exported first reinforcement learning model is computationally optimized to obtain the second reinforcement learning model, which is then deployed into the AI ​​chip.

[0088] Furthermore, the AI ​​chip includes tensor operation units, on-chip cache, parallel computing architecture, and low-power design; the AI ​​chip's compilation tools provide graph optimization instructions, specifically including: The operators in the first reinforcement learning model are classified to obtain multiple operator sequences, and there are data dependencies between the operators in each operator sequence; Merge the operators in each operator sequence and output the operator fusion instruction; Identify the fixed-value nodes in the first reinforcement learning model to obtain multiple constants of the first reinforcement learning model; Analyze the operational relationships between multiple constants and construct constant operation expressions; Based on the constant operation expression, the constant operation result is given, and a constant folding instruction is generated to replace the corresponding fixed value node with the constant operation result, thus completing the constant folding of the first reinforcement learning model. Tensor operation units are used to obtain the lifetime of each intermediate tensor in the first reinforcement learning model, and the intermediate tensors are divided according to their lifetimes to obtain multiple tensor sets, wherein the lifetimes of the intermediate tensors in each tensor set do not overlap. Allocate corresponding memory space for the intermediate tensors in each quantum set and generate memory reuse instructions; According to the memory reuse instruction, release the current intermediate tensor in the memory space corresponding to the tensor set and store the next intermediate tensor.

[0089] The memory space occupied by intermediate tensors with non-overlapping lifecycles during model execution can be reused. This memory space can be reused by different tensors at different points in time, thereby reducing the overall memory requirements. The lifecycles of two or more intermediate tensors do not overlap; that is, one tensor's use ends before another begins. The reusable memory space must be large enough to accommodate the size of these intermediate tensors. These intermediate tensors are logically independent, and their data content does not interfere with each other.

[0090] By employing operator fusion, constant folding, memory reuse, and delayed execution during the deployment and use of the second reinforcement learning model, the real-time performance of the model in microgrid scheduling is improved. Specifically, operator fusion combines consecutive convolution, batch normalization, and activation operations into a single operator, reducing memory accesses. Constant folding pre-compiles fixed-value expressions during the compilation phase, reducing runtime computation. Memory reuse reuses the storage space of intermediate tensors, reducing peak memory usage. Delayed execution postpones some computations until they are actually needed, reducing unnecessary computations.

[0091] The Tensor Computing Unit (TCU) is the core computing module of the AI ​​chip, undertaking highly parallel computational tasks such as matrix multiplication and addition, convolution, and activation in the second reinforcement learning model. It is a key component for achieving real-time inference. By accelerating the computation of state vectors and network weights through vectorization and pipelined design, the generation latency of control decisions can be compressed to the millisecond level.

[0092] On-chip cache is used to temporarily store state space features, intermediate activation values, and model parameters during inference operations, significantly reducing the frequency of access to off-chip memory (such as DDR). The cache reuse mechanism enables the rapid exchange of intermediate tensor results between multi-layer networks, thereby reducing memory access latency and power consumption, and ensuring smooth data transmission even in high sampling rate scenarios (such as 100 Hz power update cycles).

[0093] Parallel computing architecture enables the synchronous execution of different layers and operators in the policy network and value network through multi-core parallelism and task pipeline mechanisms. AI chips can simultaneously process multi-source data from photovoltaic, wind power, energy storage, and load side, allowing the inference process and energy dispatch instruction generation process to run in parallel, thereby achieving a high-throughput management and control mode of "perceiving and making decisions on the same side".

[0094] Low-power design includes dynamic voltage and frequency scaling (DVFS), on-chip power gating, and temperature-adaptive scheduling mechanisms. Under the long-term online operation conditions of microgrids, low-power design can suppress power consumption and heat generation while maintaining inference performance, ensuring that the chip continues to operate stably on the edge control node.

[0095] The AI ​​chip provides the core computing power through tensor operation units, solves data transmission bottlenecks through on-chip cache, enhances overall throughput through parallel computing architecture, and maintains system energy efficiency balance through low-power design. Through a hardware-software co-operation structure, it can achieve intelligent control with high real-time performance, low power consumption, and high reliability in the complex operating environment of microgrids.

[0096] In one specific implementation, the trained first reinforcement learning model is typically in PyTorch or TensorFlow format, and is exported as an intermediate representation, such as ONNX (Open Neural Network Exchange) format. Then, compilation tools provided by the AI ​​chip (such as Ascend ATC, TensorRT, or Cambricon Neuware Compiler) are used to generate graph optimization instructions. The second reinforcement learning model is obtained through these graph optimization instructions, and the compiled second reinforcement learning model is deployed to edge control nodes (such as microgrid dispatch controllers or embedded gateways).

[0097] After pruning and quantization optimization, the second reinforcement learning model was deployed to run on the AI ​​chip to achieve real-time inference and decision output for microgrid scheduling. During deployment, computation graph optimization, memory layout adjustment, and operator fusion were performed to fully leverage the hardware's performance advantages, taking into account the chip architecture characteristics. Through deployment and acceleration, the second reinforcement learning model achieved millisecond-level inference latency (<1ms), low power consumption (<1W chip power consumption), and high throughput (>50 GOPS effective computation) on the AI ​​chip, fully meeting the real-time scheduling requirements of microgrids in complex environments.

[0098] A hardware mapping strategy is adopted in the deployment of the second reinforcement learning model, and the specific process is as follows: Determine the ideal inference delay t for the second reinforcement learning model inf Specifically, it is expressed as:

[0099] Among them, C totalT represents the total computational cost of inference for the second reinforcement learning model (i.e., the number of multiply-accumulate units (MACs)). chip This refers to the computing throughput of AI chips.

[0100] Combining memory access latency and scheduling overhead, the actual latency of the second reinforcement learning model is given as follows:

[0101] Among them, t mem This refers to memory access latency, which is related to cache hit rate. sched For scheduling overhead, i.e., task scheduling and context switching latency, η util For hardware utilization, η util ∈[0,1], affected by the degree of parallelism and sparsity utilization.

[0102] To address the high real-time requirements of microgrids (typical control cycles of 10–50 ms), the AI ​​chip deployment optimizes thread scheduling, batch execution, and direct memory access (DMA) data transfer, resulting in hardware utilization exceeding 0.85 and memory access latency controlled within 5 μs, thereby achieving millisecond-level inference.

[0103] In intelligent management and control of microgrids, the hardware mapping strategy enables AI chips to quickly generate microgrid control results after collecting real-time operating state space, completing the entire process from state assessment to command issuance.

[0104] For example, when a sudden change in wind speed causes a sharp drop in wind power output, power redistribution needs to be completed within tens of milliseconds. At this time, through a hardware mapping strategy, the inference latency of the policy network is controlled within 3ms, ensuring that the AI ​​chip outputs decision commands for the energy storage system to discharge 25kW and the main grid to purchase 20kW of power within a single control cycle, thus achieving real-time response for power balance.

[0105] Meanwhile, the AI ​​chip employs parallel computing optimization. During AI chip deployment, multi-level parallel optimization is adopted based on operator characteristics and data dimensions. Specifically, this includes: splitting the input states of different power nodes (photovoltaic, wind power, energy storage, and load) into multiple computing units for parallel processing to ensure simultaneous computation of multi-source data and achieve data parallelism; executing the second reinforcement learning model in layers and segments, with different computing units simultaneously processing data from different layers of the policy network and value network, shortening the overall inference time and achieving pipeline parallelism; and combining the pruned sparse matrix storage structure (CSR / CSC format) for zero-skip operations to avoid invalid multiplication and addition, thereby achieving sparse acceleration.

[0106] The actual throughput of AI chips (T)eff Specifically, it is expressed as:

[0107] Among them, S ratio The sparsity S of the second reinforcement learning model ratio ∈[0,1], the higher the sparsity, the lower the actual computational cost.

[0108] Through multi-level parallel optimization, the second reinforcement learning model enables synchronous calculation of multiple units (source, load, and storage) for microgrid regulation. For example, during the regulation cycle, the photovoltaic prediction module, the energy storage SOC assessment module, and the main grid power flow prediction module run simultaneously in different computing units, thereby achieving parallel execution of energy balance calculation and scheduling command generation. When there is a sudden drop in solar intensity or insufficient energy storage capacity, the parallel architecture can simultaneously generate multi-dimensional control outputs such as "energy storage charging and discharging power adjustment," "main grid power purchase increase," and "load priority reordering," greatly improving decision-making efficiency and system robustness.

[0109] To ensure the real-time performance and security of microgrid scheduling, a task priority queue and a timeout rollback mechanism are introduced in addition to the AI ​​chip inference stage. The task priority queue prioritizes energy scheduling and frequency / voltage control tasks to ensure priority access to computing resources under multi-task concurrency. The timeout rollback mechanism detects whether the inference latency exceeds a set latency threshold. If the AI ​​chip detects that the inference latency exceeds the set threshold (e.g., 20ms), it invokes a simplified heuristic scheduling strategy to quickly generate backup control commands. This ensures that the microgrid can still operate stably within a safe range under abnormal conditions such as sudden high loads or chip overheating and frequency throttling.

[0110] In the microgrid islanding scenario, if the AI ​​inference exceeds the threshold due to external communication delay, the system will immediately execute the timeout rollback logic and quickly output the energy storage system power compensation command to maintain power balance and frequency stability; at the same time, the AI ​​chip continues background inference and smoothly switches back to the control mode of the second reinforcement learning model after completion.

[0111] The above mechanism realizes the three-layer linkage of "mapping-parallelism-real-time", enabling AI chips to have millisecond-level response speed and safe rollback capability under the dynamic operating conditions of microgrids. It ensures that the system can still operate stably under sudden disturbances, high loads and communication anomalies, and truly realizes the integration of intelligent scheduling and safety control.

[0112] The sparsity of the second reinforcement learning model is matched with that of the AI ​​chip. When the AI ​​chip is deployed, if the AI ​​chip supports sparse computing acceleration (such as a neural processing unit (NPU) or a digital signal processor (DSP) with a zero-skip mechanism), the higher the sparsity, the more significant the reduction in inference latency and power consumption.

[0113] Model sparsity is not only used to measure the degree of network compression, but also an important reference indicator for optimizing the computing power of AI chips. AI chips can identify pruned weights through sparse computing acceleration mechanisms (such as zero-skip or compressed sparse matrix storage), skipping zero-weight multiplication operations when performing matrix multiplication and addition, thereby significantly reducing memory access requirements and power consumption. The higher the model sparsity, the smaller the effective amount of computation that the AI ​​chip can perform, and the inference latency and energy consumption decrease linearly.

[0114] In addition, to achieve a precise match between the sparse structure of the model and the scheduling of AI chip operators, a "sparse mapping calibration" step is introduced during the deployment phase: when the sparsity reaches the set sparsity threshold, the AI ​​chip compilation tool will call the Sparse Kernel Library to implement multiply-accumulate skipping and cache compression at the hardware level.

[0115] Based on this, the second reinforcement learning model not only has the advantage of theoretical compression, but also achieves inference acceleration and energy efficiency improvement in the actual deployment of AI chips, ensuring that the microgrid scheduling system can achieve high-frequency control response under low power consumption conditions.

[0116] S104: The real-time operating state space is analyzed using the second reinforcement learning model, and the microgrid control results are given.

[0117] In one specific implementation, the second reinforcement learning model receives the real-time running state space S in real time. t The microgrid control results are generated by performing inference calculations through an AI chip and outputting actions. t It is then distributed to actuators such as energy storage converters, inverters, and circuit breakers.

[0118] The AI ​​chip utilizes a second reinforcement learning model to analyze the real-time operating state space within each control cycle. The policy network predicts the action distribution based on the real-time operating state space, the value network evaluates long-term returns, and selects the optimal action as the microgrid control outcome. t This process is equivalent to a comprehensive assessment of the current energy balance, operational economy, and safety. Microgrid control results a t A set of continuous control variables, specifically represented as:

[0119] in, For the charging and discharging power commands of the energy storage system, This is a charging power command for the energy storage system. This is the discharge power command for the energy storage system. Main network power switching command, The active / reactive power setting command for the inverter, V ref For the target voltage, f ref This is a frequency reference value.

[0120] In a specific example, under grid-connected mode, when photovoltaic output drops sharply and load increases, the AI ​​chip identifies the photovoltaic power generation P on the generation side at time t in real time. pv (t) Decrease, predicted load power P on the load side at time t load (t) increases, outputting the microgrid control result:

[0121] The microgrid control results control the energy storage system to discharge 25kW and compensate for power purchases, purchasing 20kW to achieve power balance. In islanded mode, when a frequency offset of ±0.3Hz is detected, the AI ​​chip automatically outputs the microgrid control result f. ref =50.1Hz, control the energy storage system to increase power output to restore frequency stability.

[0122] The microgrid control results are converted into standard communication protocol (such as Modbus, CAN or IEC 61850) commands through the AI ​​chip interface and sent to the field execution unit in real time, forming a complete control closed loop. This enables the AI ​​chip to complete state analysis, action generation and execution feedback in milliseconds, achieving adaptive, precise and safe operation of the microgrid under dynamic disturbances.

[0123] S105: Cache the real-time running state space in the experience cache area.

[0124] S106: Based on the preset update cycle, extract the update sample set from the experience cache and update the second reinforcement learning model.

[0125] Specifically, refer to Figure 7 According to the preset update cycle, the update sample set is extracted from the experience cache. Construct the total loss function by incorporating stability constraints; The second reinforcement learning model is trained based on the updated sample set, combined with the online optimization objective function and the total loss function; By incorporating a disturbance simulator, random noise is injected into the microgrid operating state and microgrid control actions obtained during the training of the second reinforcement learning model, thereby updating the second reinforcement learning model.

[0126] During inference, small batches of data (update sample set) are periodically sampled from the running state space in the experience cache for rapid gradient updates (e.g., update cycle of 10 minutes).

[0127] The online fine-tuning of the objective function is the same as that of offline training; that is, the online optimization of the objective function is specifically expressed as:

[0128] Among them, L total For the model optimization objective, L actor Let L be the loss function of the policy network. critic For the value network error loss, λ critic Let θ be the balance coefficient corresponding to the error loss of the value network, and let θ be the policy network parameter of the policy network. These are the value network parameters of the value network.

[0129] Due to the limited computing power of AI chips, online updates are performed in a low-frequency, small-batch, and low-learning-rate manner to avoid significant delays in real-time inference.

[0130] To improve the model's stability under sudden disturbances, a disturbance simulator is introduced during the online update phase to simulate the microgrid's operating state S. t and microgrid control actions a t Injecting random noise, specifically as follows:

[0131] in, The operating state of the microgrid after random noise injection. For microgrid control actions after random noise is injected, State noise, simulating sensor measurement error or delay. To simulate the motion noise, we use the actuator response deviation.

[0132] During training, the second reinforcement learning model needs to be able to output stable and safe scheduling instructions even under noisy conditions, thereby enhancing the robustness of the second reinforcement learning model.

[0133] To prevent strategy divergence or frequent switching caused by online updates, a stability constraint is added during the optimization process:

[0134] Where: L stable D is a stability constraint term. KL () represents the KL divergence, used to measure the difference in distribution between the old and new search strategies. For the online updated search strategy, For the search strategy before the update, η KL Here, represents the stability regularization coefficient. The total loss function during the online update process is obtained, specifically expressed as:

[0135] By updating the second reinforcement learning model online, the performance of the new search strategy is improved without causing excessive disturbance to the original control mode, thereby preventing drastic fluctuations in the operation of the microgrid.

[0136] Reference Figure 8 This invention provides a lightweight reinforcement learning microgrid control device based on an AI chip, comprising: The data acquisition module 201 is used to acquire multi-source operation data of the microgrid in real time and construct a real-time operation status space; Model acquisition module 202 is used to acquire a lightweight first reinforcement learning model; The computation optimization module 203 is used to perform computation optimization on the first reinforcement learning model based on the characteristics of the AI ​​chip architecture, provide a second reinforcement learning model, and deploy the second reinforcement learning model into the AI ​​chip. The result output module 204 is used to analyze the real-time operating state space through the second reinforcement learning model and provide the microgrid control results.

[0137] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0138] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and variations of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A lightweight reinforcement learning microgrid control method based on AI chips, characterized in that, include: Real-time acquisition of multi-source operation data of microgrids, and construction of real-time operation status space; Obtain a lightweight first reinforcement learning model; Based on the characteristics of AI chip architecture, the first reinforcement learning model is computationally optimized to provide a second reinforcement learning model, which is then deployed into the AI ​​chip. The real-time operating state space is analyzed using a second reinforcement learning model, and the microgrid control results are given.

2. The lightweight reinforcement learning microgrid control method based on AI chip as described in claim 1, characterized in that, The real-time operating status space includes real-time photovoltaic power generation, real-time wind power generation, real-time state of charge of energy storage system, real-time energy storage charging and discharging power, real-time grid switching power, real-time predicted load power, real-time bus voltage, real-time system frequency, real-time solar irradiance, real-time wind speed, and real-time ambient temperature.

3. The lightweight reinforcement learning microgrid control method based on AI chip as described in claim 1, characterized in that, The lightweight first reinforcement learning model is determined through the following steps: Acquire historical multi-source operation data and construct a historical operation status space. The historical multi-source operation data includes historical operation data of the power generation side, historical operation data of the energy storage side, historical operation data of the load side, and historical environmental information. Based on the power balance, operating costs, and security constraints of the microgrid, a reward function for the microgrid is constructed. Based on the policy network of the initial reinforcement learning model, the historical operating state space is analyzed, and the microgrid control actions corresponding to the historical operating state space are given. Based on the value network of the initial reinforcement learning model, the state-action group is analyzed, and the operational value of the state-action group is given. By combining the microgrid reward function and the value network error loss, the policy network and value network of the initial reinforcement learning model are iteratively trained to obtain the first reinforcement learning model, which is then optimized to obtain a lightweight first reinforcement learning model.

4. The lightweight reinforcement learning microgrid control method based on AI chip as described in claim 3, characterized in that, The first reinforcement learning model, which has been optimized for lightweight processing, specifically includes: Based on the pruning and combining strategy, the first reinforcement learning model is structurally compressed and retrained in combination with the model optimization objective to obtain the pruned reinforcement learning model. Based on the quantization combination strategy, the pruning reinforcement learning model is quantized for optimization and quantization-aware training to obtain a lightweight first reinforcement learning model.

5. The lightweight reinforcement learning microgrid control method based on AI chip as described in claim 4, characterized in that, Based on the pruning-combination strategy, the first reinforcement learning model is structurally compressed and retrained in conjunction with the model optimization objective to obtain a pruned reinforcement learning model. The pruning-combination strategy includes structured pruning and unstructured pruning, specifically including: Based on each convolutional kernel and each weight in the current reinforcement learning model, determine the overall importance value of the convolutional kernel and the local importance value corresponding to each weight in the convolutional kernel. Based on the preset pruning threshold, the overall importance value of each convolutional kernel is judged, and the convolutional kernels in the first reinforcement learning model are selected for structured pruning. Based on the preset pruning threshold, the local importance values ​​corresponding to each weight in the filtered convolutional kernel are judged, and each weight in the convolutional kernel is filtered and unstructured pruning is performed. Analyze the current number of weights and the total number of weights in the current reinforcement learning model, and retrain the current reinforcement learning model in combination with the model optimization objective. The current reinforcement learning model is the first reinforcement learning model at different times during the pruning process. If the pruning ratio reaches a preset threshold, a pruned reinforcement learning model is obtained, where the pruning ratio is the ratio of the current number of weights to the total number of weights in the first reinforcement learning model.

6. The lightweight reinforcement learning microgrid control method based on AI chip as described in claim 1, characterized in that, Based on the characteristics of AI chip architecture, the first reinforcement learning model is computationally optimized to produce a second reinforcement learning model, which is then deployed into the AI ​​chip. Specifically, this includes: Based on the characteristics of AI chip architecture, the first reinforcement learning model is derived, and graph optimization instructions are given using the AI ​​chip's compilation tools. These graph optimization instructions include operator fusion instructions, constant folding instructions, and memory reuse instructions. By combining graph optimization instructions, the exported first reinforcement learning model is computationally optimized to obtain the second reinforcement learning model, which is then deployed into the AI ​​chip.

7. The lightweight reinforcement learning microgrid control method based on AI chip as described in claim 6, characterized in that, AI chips include tensor operation units, on-chip cache, parallel computing architecture, and low-power design; the compilation tools for AI chips provide graph optimization instructions, specifically including: The operators in the first reinforcement learning model are classified to obtain multiple operator sequences, and there are data dependencies between the operators in each operator sequence; Merge the operators in each operator sequence and output the operator fusion instruction; Identify the fixed-value nodes in the first reinforcement learning model to obtain multiple constants of the first reinforcement learning model; Analyze the operational relationships between multiple constants and construct constant operation expressions; Based on the constant operation expression, the constant operation result is given, and a constant folding instruction is generated to replace the corresponding fixed value node with the constant operation result, thus completing the constant folding of the first reinforcement learning model. Tensor operation units are used to obtain the lifetime of each intermediate tensor in the first reinforcement learning model, and the intermediate tensors are divided according to their lifetimes to obtain multiple tensor sets, wherein the lifetimes of the intermediate tensors in each tensor set do not overlap. Allocate corresponding memory space for the intermediate tensors in each quantum set and generate memory reuse instructions; According to the memory reuse instruction, release the current intermediate tensor in the memory space corresponding to the tensor set and store the next intermediate tensor.

8. The lightweight reinforcement learning microgrid control method based on AI chip as described in claim 1, characterized in that, The method also includes: The real-time runtime state space is cached in the experience cache area; Based on a preset update cycle, an update sample set is extracted from the experience cache to update the second reinforcement learning model.

9. The lightweight reinforcement learning microgrid control method based on AI chip as described in claim 8, characterized in that, Based on a preset update cycle, an update sample set is extracted from the experience cache to update the second reinforcement learning model, specifically including: According to the preset update cycle, the update sample set is extracted from the experience cache. Combining the stability constraint term, the total loss function is constructed, specifically expressed as: ； in, Let L be the total loss function. actor Let L be the loss function of the policy network. critic For the value network error loss, λ critic L is the balance coefficient corresponding to the error loss of the value network. stable These are stability constraint terms; The second reinforcement learning model is trained based on the updated sample set, combined with the online optimization objective function and the total loss function; By incorporating a disturbance simulator, random noise is injected into the microgrid operating state and microgrid control actions obtained during the training of the second reinforcement learning model, thereby updating the second reinforcement learning model.

10. A lightweight reinforcement learning microgrid control device based on an AI chip, characterized in that, The lightweight reinforcement learning microgrid control method based on AI chips, as described in any one of claims 1-9, includes: The data acquisition module is used to acquire multi-source operation data of the microgrid in real time and construct a real-time operation status space; The model acquisition module is used to acquire the first reinforcement learning model with lightweight processing. The computational optimization module is used to perform computational optimization on the first reinforcement learning model based on the characteristics of the AI ​​chip architecture, provide a second reinforcement learning model, and deploy the second reinforcement learning model into the AI ​​chip. The results output module is used to analyze the real-time operating state space through the second reinforcement learning model and provide the microgrid control results.

Images