Storage battery energy distribution method, device and equipment and storage medium

Abstract

The invention discloses a storage battery energy distribution method, device and equipment and a storage medium, and relates to the technical field of storage batteries, and the method comprises the steps: collecting the state data of a storage battery; inputting the state data into an intelligent agent strategy network to obtain an optimal action, and determining state data at the next moment; inputting the state data and the optimal action into an intelligent agent evaluation network to obtain a reward value; generating interaction experience based on the optimal action, the state data, the reward value and the state data of the next moment, and storing the interaction experience in an experience recovery pool; and extracting target interaction experience from the experience recovery pool, and adjusting parameters of a deep reinforcement learning model based on the target interaction experience so as to realize energy distribution of the storage battery after determining a current optimal action of the storage battery under the current state data through the optimized deep reinforcement learning model. According to the invention, the optimal energy distribution of the storage battery can be realized.

Description

Technical Field

[0001] The present application relates to the field of battery technology, and in particular to a battery energy distribution method, device, equipment and storage medium. Background Art

[0002] As the market for electric vehicles and electric bicycles continues to expand, batteries, as core components of these electric devices, have performance and charge / discharge power that are directly related to the charging convenience and overall energy efficiency of electric vehicles. The energy allocation strategy of batteries plays a vital role in optimizing user experience and system operating efficiency. Currently, the industry has developed a variety of battery energy allocation methods, including strategies based on established rules, solutions based on model predictive control, approaches using machine learning technology, and strategies based on various optimization algorithms. However, these methods generally have the problem of unreasonable battery energy allocation.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a battery energy distribution method, device, equipment and storage medium, aiming to solve the technical problem of unreasonable battery energy distribution commonly existing in traditional battery energy distribution methods.

[0005] To achieve the above objectives, the present application proposes a battery energy distribution method, the battery energy distribution method comprising:

[0006] Collecting battery status data, wherein the status data includes battery charging status, voltage status, ambient temperature and load demand;

[0007] Input the state data into the agent strategy network to obtain the optimal action and determine the state data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state;

[0008] Inputting the state data and the optimal action into the agent evaluation network to obtain a reward value;

[0009] Generate interactive experience based on the optimal action, state data, reward value and next moment state data, and store the interactive experience in an experience recycling pool;

[0010] Target interaction experience is extracted from the experience recycling pool, and parameters of the deep reinforcement learning model are adjusted based on the target interaction experience, so as to achieve battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

[0011] In one embodiment, the step of inputting the state data and the optimal action into an agent evaluation network to obtain a reward value comprises:

[0012] Input the state data and the optimal action into the intelligent agent evaluation network, determine the energy utilization rate reward value through the energy utilization rate reward function, determine the battery life reward value through the battery life reward function, determine the economic benefit reward value through the economic benefit reward function, and determine the response speed reward value through the response speed reward function;

[0013] A reward value is determined based on the energy utilization rate reward value, the battery life reward value, the economic benefit reward value, and the response speed reward value.

[0014] In one embodiment, the step of determining the energy utilization rate reward value by using the energy utilization rate reward function comprises:

[0015] Determining actual charge and discharge power based on the state data and the optimal action;

[0016] An energy utilization rate reward value is determined according to a first adjustment factor, a target charge and discharge power, and the actual charge and discharge power, wherein the first adjustment factor is used to balance the penalty intensity.

[0017] In one embodiment, the step of determining the battery life bonus value by using the battery life bonus function comprises:

[0018] determining an actual charging state based on the state data and the optimal action;

[0019] Determining whether the actual charging state is within a preset reasonable range;

[0020] When the actual charging state falls within the preset reasonable range, setting the battery life reward value to a positive reward;

[0021] When the actual charging state does not belong to the preset reasonable range, the battery life reward value is set to a negative reward.

[0022] In one embodiment, the step of determining the economic benefit reward value by using the economic benefit reward function comprises:

[0023] Based on the state data and the optimal action, determining the charging power of the battery in different charging periods and the discharging power of the battery in different discharging periods;

[0024] An economic efficiency reward value is determined according to the second adjustment factor, the electricity price during the time period, the charging power, and the discharging power.

[0025] In one embodiment, the step of determining the response speed reward value by using the response speed reward function comprises:

[0026] Determining a strategy adjustment time based on the state data and the optimal action;

[0027] Determine whether the strategy adjustment time is greater than a preset adjustment time;

[0028] When it is determined that the strategy adjustment time is not greater than the preset adjustment time, setting the response speed reward value to a positive reward;

[0029] When it is determined that the strategy adjustment time is greater than the preset adjustment time, the response speed reward value is set to a negative reward.

[0030] In one embodiment, the step of adjusting the parameters of the deep reinforcement learning model based on the target interaction experience includes:

[0031] Determine the expected value of the state cumulative reward of all state data after the state data of the target interaction experience according to the state value function;

[0032] Determine the state data of the target interaction experience and the expected value of the action cumulative reward under the optimal action according to the action value function;

[0033] Determine a state loss function value and an action loss function value according to the state cumulative reward expected value and the action cumulative reward expected value;

[0034] Adjust the parameters of the deep reinforcement learning model according to the state loss function value and the action loss function value.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a battery energy distribution device, the battery energy distribution device comprising:

[0036] A collection module, used to collect battery status data, wherein the status data includes battery charging status, voltage status, ambient temperature and load demand;

[0037] An input module, used to input the state data into the agent strategy network, obtain the optimal action, and determine the state data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state;

[0038] The input module is further used to input the state data and the optimal action into the agent evaluation network to obtain a reward value;

[0039] A generation module, used to generate interaction experience based on the optimal action, state data, reward value and next moment state data, and store the interaction experience in an experience recycling pool;

[0040] A determination module is used to extract target interaction experience from the experience recycling pool and adjust the parameters of the deep reinforcement learning model based on the target interaction experience, so as to achieve battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

[0041] In addition, to achieve the above-mentioned objectives, the present application also proposes a battery energy distribution device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the battery energy distribution method described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the battery energy distribution method described above are implemented.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the battery energy distribution method described above are implemented.

[0044] One or more technical solutions proposed in this application have at least the following technical effects:

[0045] The battery energy distribution method, device, equipment and storage medium proposed in the present application collect the status data of the battery, wherein the status data includes the battery charging status, voltage status, ambient temperature and load demand; input the status data into the intelligent agent strategy network to obtain the optimal action, and determine the status data at the next moment, wherein the optimal action includes one of the charging power, discharging power or standby state; input the status data and the optimal action into the intelligent agent evaluation network to obtain the reward value; based on the optimal action, status data, reward value and status data at the next moment, generate interaction experience, and store the interaction experience in the experience recovery pool; extract the target interaction experience from the experience recovery pool, and adjust the depth based on the target interaction experience. The parameters of the deep reinforcement learning model are determined by the optimized deep reinforcement learning model to realize the battery energy distribution after the current optimal action of the battery under the current state data, which solves the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods. Compared with the existing technology, the present application realizes dynamic decision-making in complex environments through an intelligent agent strategy network, evaluates the value of dynamic strategies through an intelligent agent evaluation network, and stores the interaction experience between the intelligent agent and the environment in an experience recovery pool, so as to use the historical experience in the experience recovery pool to continuously optimize the deep reinforcement learning model in a complex environment, so that the optimized deep reinforcement learning model can find the current optimal action according to the current state data, thereby realizing the optimal battery energy distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the battery energy distribution method of the present application;

[0049] Figure 2 A schematic diagram of a flow chart provided for the second embodiment of the battery energy distribution method of the present application;

[0050] Figure 3 This is a schematic diagram of the module structure of the battery energy distribution device according to an embodiment of the present application;

[0051] Figure 4Schematic diagram of the device structure of the hardware operating environment involved in the battery energy distribution method in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: collecting the status data of the battery, wherein the status data includes the battery charging status, voltage status, ambient temperature and load demand; inputting the status data into the intelligent agent strategy network to obtain the optimal action, and determine the status data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state; inputting the status data and the optimal action into the intelligent agent evaluation network to obtain a reward value; based on the optimal action, status data, reward value and status data at the next moment, generating interaction experience, and storing the interaction experience in an experience recovery pool; extracting the target interaction experience from the experience recovery pool, and adjusting the parameters of the deep reinforcement learning model based on the target interaction experience, so as to realize the battery energy distribution after determining the current optimal action of the battery under the current status data through the optimized deep reinforcement learning model.

[0056] It can be seen from the above embodiments that the present application collects the status data of the battery, wherein the status data includes the battery charging status, voltage status, ambient temperature and load demand; inputs the status data into the intelligent agent strategy network to obtain the optimal action, and determines the status data at the next moment, wherein the optimal action includes one of the charging power, discharging power or standby state; inputs the status data and the optimal action into the intelligent agent evaluation network to obtain the reward value; based on the optimal action, status data, reward value and the status data at the next moment, generates interaction experience, and stores the interaction experience in the experience recovery pool; extracts the target interaction experience from the experience recovery pool, and adjusts the deep reinforcement learning model based on the target interaction experience Parameters are used to determine the current optimal action of the battery under the current state data through an optimized deep reinforcement learning model, and then realize battery energy distribution, which solves the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods. Compared with the prior art, this application realizes dynamic decision-making in complex environments through an intelligent agent strategy network, evaluates the value of dynamic strategies through an intelligent agent evaluation network, and stores the interaction experience between the intelligent agent and the environment in an experience recovery pool, so as to use the historical experience in the experience recovery pool to continuously optimize the deep reinforcement learning model in a complex environment, so that the optimized deep reinforcement learning model can find the current optimal action according to the current state data, thereby realizing the optimal battery energy distribution.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a battery energy distribution device, etc. The following takes battery energy distribution as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the present application embodiment provides a battery energy distribution method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the battery energy distribution method of the present application.

[0059] In this embodiment, the battery energy distribution method includes steps S10 to S50:

[0060] Step S10, collecting battery status data, wherein the status data includes battery charging status, voltage status, ambient temperature and load demand;

[0061] It should be noted that the battery status data can be collected through sensors or other data acquisition devices; the battery state of charge (SOC) refers to the ratio of the remaining capacity of the battery to its capacity in a fully charged state; the voltage state refers to the voltage level of the battery; the ambient temperature refers to the temperature of the battery and the environment; the load demand refers to the battery supply record; the historical discharge record refers to the discharge status information at the previous moment.

[0062] Step S20, inputting the state data into the agent strategy network, obtaining the optimal action, and determining the state data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state;

[0063] It should be noted that Charging Power refers to the battery charging power at the historical state moment; Discharging Power refers to the battery discharging power at the historical state moment; Standby state refers to the state where the battery is neither charging nor discharging; The agent can use the policy network to select an optimal action based on the state data; The agent policy network refers to the policy network used by the agent in the deep reinforcement learning model, which can be used to select the optimal action based on the battery state data.

[0064] In a feasible implementation manner, the step of collecting the battery status data includes: collecting the initial status data of the battery; cleaning the initial status data to remove missing values, abnormal values ​​and noise values ​​in the initial status data to obtain cleaned data; filling the missing parts in the cleaned data by interpolation to obtain filled data; normalizing the filled data to obtain status data.

[0065] It should be noted that the collected initial state data can be preprocessed, mainly including data cleaning and data normalization, etc. Data cleaning includes removing missing values, outliers and noise values ​​in the initial data, etc.; preprocessing the initial state data can ensure data quality and data consistency, so as to provide high-quality data input for model training.

[0066] Step S30, inputting the state data and the optimal action into the agent evaluation network to obtain a reward value;

[0067] It should be noted that the agent evaluation network refers to the evaluation network used by the agent in the deep reinforcement learning model, which can be used to evaluate the value of the optimal action based on the state data and the optimal action; the reward value can be evaluated based on factors such as battery energy utilization, battery life, economic benefits, and response speed. Specifically, different reward functions can be designed according to various actual working conditions to ensure that the agent in the deep reinforcement learning model can make the best decision in different scenarios.

[0068] Step S40, generating interaction experience based on the optimal action, state data, reward value and next moment state data, and storing the interaction experience in an experience recycling pool;

[0069] It should be noted that the experience recycling pool is used to store the interaction experience generated by the interaction between the agent and the environment in the deep reinforcement learning model.

[0070] In the specific implementation, assuming that the task of the intelligent agent of the deep reinforcement learning model is to manage the battery energy distribution of the electric vehicle, the intelligent agent needs to observe the environment (that is, obtain state data by observing the environment), execute decisions (that is, input the state data into the policy network to obtain the optimal action) and obtain feedback (that is, input the state data and the optimal action into the evaluation network to obtain the reward value) at each time step (for example, every minute). For example, 1. The agent observes the current battery status information St (also known as status data), including: battery state of charge (SOC), battery voltage, ambient temperature, current load and other requirements; assuming that the agent observes at time step t: SOC = 60%, voltage = 3.7V, temperature = 25°C, load demand = 20W; 2. Based on the current status data St, the agent uses a policy network to select an optimal action at (i.e., optimal action). The action space includes different charging and discharging strategies, such as selecting charging, discharging or standby state. In this example, the agent chooses "discharging 15W"; 3. The environment feedback reward rt (i.e., reward value). The environment gives feedback (reward value) based on the agent's action at (optimal action). The feedback is the reward value of the agent's action this time, indicating the contribution of the action to the goal, where the reward can be evaluated based on factors such as battery energy utilization and battery life. Assuming that the action of discharging 15W helps to maintain the health of the battery and meet the load demand, the environment gives a positive reward rt = +1; 4. Observe the next moment state St+1 of the state data (that is, the next moment state data). After performing the action of discharging 15W, the agent enters the next time step and observes the state again. For example, after 1 minute, the SOC drops to 59%, the voltage remains unchanged, the temperature rises to 26°C, and the load demand drops to 15W. The new state (i.e. the state data at the next moment) is St+1: SOC=59%, voltage=3.7V, temperature=26℃, load demand=15W; 5. The intelligent agent records the interaction experience in this interaction process as a historical experience and stores it in the experience recovery pool (st, at, rt, st+1)=(SOC=60%, voltage=3.7V, temperature=25℃, load demand=20W, discharge 15W, +1, SOC=59%, voltage=3.7V, temperature=26℃, load demand=15W); 6. The model is trained by randomly extracting the target interaction experience from the experience recovery pool to optimize the weights of the deep reinforcement learning model so that the model can gradually learn the optimal energy allocation strategy.

[0071] Step S50, extracting target interaction experience from the experience recycling pool, and adjusting the parameters of the deep reinforcement learning model based on the target interaction experience, so as to achieve battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

[0072] It should be noted that the target interaction experience can be extracted from the experience recycling pool in a random manner, so as to break the data correlation for model training; the current state data refers to the battery charging state, voltage state, ambient temperature and load demand in the current state; the current optimal action refers to the optimal action in the current state, that is, one of the charging power, discharging power or standby state.

[0073] It should be noted that in actual applications, the integrated adaptive learning mechanism enables the model to continuously optimize the strategy based on new environmental data (such as temperature changes, load changes, etc.). The model can identify new patterns in complex nonlinear spaces and dynamically adjust the energy allocation plan. At the same time, the integrated online update mechanism allows the model to update the strategy in real time during actual operation, ensuring that the best decision can still be made when the environment changes.

[0074] It should be noted that the trained model is deployed to the battery management system for testing under actual working conditions. The test content includes the energy distribution effect under different ambient temperatures and different load requirements, as well as the response speed and decision accuracy of the model. According to the test results, the model is further tuned, such as adjusting the reward function parameters and optimizing the neural network structure, to ultimately ensure the efficiency and reliability of the system.

[0075] In a feasible implementation, the step of adjusting the parameters of the deep reinforcement learning model based on the target interaction experience includes: determining the expected value of the state cumulative reward of all state data after the state data of the target interaction experience according to the state value function; determining the expected value of the action cumulative reward under the state data of the target interaction experience and the optimal action according to the action value function; determining the state loss function value and the action loss function value according to the state cumulative reward expected value and the action cumulative reward expected value; and adjusting the parameters of the deep reinforcement learning model according to the state loss function value and the action loss function value.

[0076] In the specific implementation, the state value function V(s) represents the expected value of the cumulative reward of all subsequent states under a given state s, which can be defined as follows:

[0077]

[0078] Among them, γ is the discount factor, r(s t , a t ) is the reward for each step (i.e. the reward for each state data).

[0079] In the specific implementation, the action value function Q(s, a) represents the cumulative reward under a given state s and action a:

[0080] Q(s,a)=r(s,a)+γE[Q(s′,a′)|s,a]

[0081] Among them, s' is the new state transferred to based on state s and action a. DDPG uses a neural network to approximate the Q function to cope with complex and continuous state-action space.

[0082] It can be understood that the state value function V(s) measures the expected value of the cumulative reward that can be obtained after taking any possible action starting from a certain state s. This expected value is calculated based on the model's current understanding of the environment and past interaction experience. In the target interaction experience, we focus on the state data s and calculate the expected value of the cumulative reward for all possible states in the future starting from this state.

[0083] It can be understood that the action value function Q(s, a) measures the expected value of the cumulative reward that can be obtained after taking a specific action a in a given state s; in the target interaction experience, we focus on the current state data s and the action a that is considered to be the best, and calculate the expected value of the cumulative reward of all possible states in the future starting from this state and action.

[0084] In specific implementation, the loss function is a function used in machine learning to measure the difference between the model's predicted results and the actual results. In deep reinforcement learning, we define the state loss function and the action loss function to measure the prediction errors of the state value function and the action value function respectively; the state loss function value is calculated based on the difference between the predicted value of the state value function (the expected value of the cumulative reward of the state) and the preset value; the action loss function value is calculated based on the difference between the predicted value of the action value function (that is, the expected value of the cumulative reward of the action) and the preset value; adjust the parameters of the deep reinforcement learning model according to the loss function value: after determining the state loss function value and the action loss function value, we can use these values ​​to guide the update of the model parameters. The common method is to optimize the loss function through gradient descent or its variants, that is, adjust the parameters of the model to minimize the loss function value. In this process, the model will continue to learn and improve its strategy to better adapt to the environment and maximize the cumulative reward.

[0085] This embodiment collects the state data of the battery, wherein the state data includes the battery charging state, voltage state, ambient temperature and load demand; inputs the state data into the intelligent agent strategy network to obtain the optimal action, and determines the state data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state; inputs the state data and the optimal action into the intelligent agent evaluation network to obtain a reward value; generates interaction experience based on the optimal action, state data, reward value and state data at the next moment, and stores the interaction experience in an experience recycling pool; extracts target interaction experience from the experience recycling pool, and adjusts the parameters of the deep reinforcement learning model based on the target interaction experience. The battery energy distribution is achieved by determining the current optimal action of the battery under the current state data through an optimized deep reinforcement learning model, which solves the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods. Compared with the existing technology, this application realizes dynamic decision-making in complex environments through an intelligent agent strategy network, and evaluates the value of dynamic strategies through an intelligent agent evaluation network, and stores the interaction experience between the intelligent agent and the environment in an experience recovery pool, so as to use the historical experience in the experience recovery pool to continuously optimize the deep reinforcement learning model in a complex environment, so that the optimized deep reinforcement learning model can find the current optimal action according to the current state data, thereby achieving the optimal battery energy distribution.

[0086] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , step S30 also includes steps S301-S302:

[0087] S301, inputting the state data and the optimal action into an intelligent agent evaluation network, determining an energy utilization rate reward value through an energy utilization rate reward function, determining a battery life reward value through a battery life reward function, determining an economic benefit reward value through an economic benefit reward function, and determining a response speed reward value through a response speed reward function;

[0088] It should be noted that a reasonable reward function should be designed according to the goal of battery energy distribution, such as optimizing energy utilization, extending battery life, improving economic benefits, or quickly responding to environmental changes. The reward function is used to evaluate the pros and cons of each action and directly affects the learning effect of the deep reinforcement learning model. Different reward functions are designed according to various actual working conditions to ensure that the intelligent agent can make the best decision in different scenarios.

[0089] In a feasible implementation, the step of determining the energy utilization reward value through the energy utilization reward function includes: determining the actual charge and discharge power based on the state data and the optimal action; determining the energy utilization reward value according to a first adjustment factor, the target charge and discharge power and the actual charge and discharge power, wherein the first adjustment factor is used to balance the penalty intensity.

[0090] It should be noted that the goal of the reward function for optimizing energy utilization is to maximize the energy utilization efficiency of the battery during the charging and discharging process and avoid energy waste. The energy utilization reward function is designed as follows:

[0091] 1. If the charging and discharging power is close to the ideal value (maximum use of available energy without loss), a positive reward is given;

[0092] 2. If the charging and discharging power is too high or too low, resulting in energy loss, a negative reward will be given.

[0093] The specific design is as follows:

[0094] r(s, a) = -α×(target charge / discharge power - actual charge / discharge power) 2

[0095] Among them, a is the adjustment factor (i.e., the first adjustment factor), which is used to balance the penalty intensity;

[0096] It is understandable that the smaller the difference between the actual charging and discharging power and the target charging and discharging power, the higher the reward, which means that the energy distribution is more reasonable.

[0097] In a feasible implementation, the step of determining the battery life reward value through the battery life reward function includes: determining the actual charging state based on the state data and the optimal action; judging whether the actual charging state falls within a preset reasonable range; when the actual charging state falls within the preset reasonable range, setting the battery life reward value to a positive reward; when the actual charging state does not fall within the preset reasonable range, setting the battery life reward value to a negative reward.

[0098] It should be noted that the goal of the reward function for extending battery life is to avoid overcharging and discharging, maintain the battery's health, and thus extend the battery's service life. The battery life reward function is designed as follows:

[0099] 1. Give positive rewards when the battery's state of charge (SOC) remains within a reasonable range (e.g. 20% to 80%);

[0100] 2. If the battery’s SOC is out of a reasonable range, a negative reward is given to prevent over-discharge or over-charge.

[0101] The specific design is as follows:

[0102]

[0103] Understandably, the battery life reward function encourages the agent to keep the battery within a healthy state-of-charge range, thereby extending the battery life.

[0104] In a feasible implementation, the step of determining the economic benefit reward value through the economic benefit reward function includes: determining the charging power of the battery in different charging time periods and the discharging power of the battery in different discharging time periods based on the state data and the optimal action; determining the economic efficiency reward value according to the second adjustment factor, the time period electricity price, the charging power, and the discharging power.

[0105] It should be noted that the goal of the reward function for improving economic benefits is to reduce energy consumption costs and improve the economic benefits of the system by optimizing the battery charging and discharging process. The economic benefit reward function is designed as follows:

[0106] 1. When the battery is used to generate electricity during a period of low electricity prices (reducing costs), a positive reward will be given.

[0107] 2. When the battery is discharged during a period of high electricity prices (to increase profits), a positive reward will be given.

[0108] 3. If charging is done during peak electricity price periods or discharging during low electricity price periods, a negative reward will be given.

[0109] The specific design is as follows:

[0110] r(s,a)=-β×electricity price×charging power+γ×electricity price×discharging power

[0111] Among them, β and γ are adjustment factors (second adjustment factors). This function encourages charging when the electricity price is low and discharging when it is high, thereby improving economic benefits.

[0112] In a feasible implementation, the step of determining the response speed reward value through the response speed reward function includes: determining the strategy adjustment time based on the state data and the optimal action; judging whether the strategy adjustment time is greater than the preset adjustment time; when it is determined that the strategy adjustment time is not greater than the preset adjustment time, setting the response speed reward value to a positive reward; when it is determined that the strategy adjustment time is greater than the preset adjustment time, setting the response speed reward value to a negative reward.

[0113] It should be noted that the goal of the reward function that responds quickly to environmental changes is to encourage the model to respond quickly to changes in the environment and load and to keep the system stable. The reward function is designed as follows:

[0114] 1. If the model quickly adjusts its policy when the environment changes (such as temperature or load changes), give positive rewards;

[0115] 2. If the adjustment speed is too slow or unable to adapt to the changes, negative rewards will be given.

[0116] The specific design is as follows:

[0117]

[0118] It should be noted that this reward mechanism encourages the agent to make decisions quickly when the environment changes in order to keep the system running efficiently.

[0119] S302: Determine a reward value based on the energy utilization rate reward value, the battery life reward value, the economic benefit reward value, and the response speed reward value.

[0120] It should be noted that the comprehensive reward function takes into account four factors: energy utilization, battery life, economic benefits, and response speed. The purpose of the comprehensive reward function is to weight the various target reward values ​​together and balance the weights between different design goals. The details are as follows:

[0121] r(s, a) = W1*energy utilization rate bonus + W2*battery life bonus + W3*economic benefit bonus + W4*response speed bonus

[0122] Among them, W1, W2, W3, and W4 are weight coefficients that determine the importance of each goal in the total reward, so that they can be flexibly adjusted according to the needs of different scenarios.

[0123] It should be noted that the reward function design under each goal can guide the deep reinforcement learning model to learn more reasonable charging and discharging strategies through continuous training, thereby achieving the optimal battery energy distribution plan.

[0124] This embodiment inputs the state data and the optimal action into the intelligent agent evaluation network, determines the energy utilization rate reward value through the energy utilization rate reward function, determines the battery life reward value through the battery life reward function, determines the economic benefit reward value through the economic benefit reward function, and determines the response speed reward value through the response speed reward function; based on the energy utilization rate reward value, the battery life reward value, the economic benefit reward value and the response speed reward value, the reward value is determined. In the above manner, energy utilization rate, battery life, economic benefit and response speed can be taken into account at the same time when designing the reward function, so as to ensure that the intelligent agent in the deep reinforcement learning model learning can make the best decision in different scenarios.

[0125] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the battery energy distribution method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0126] This application also provides a battery energy distribution device, please refer to Figure 3 , the battery energy distribution device comprises:

[0127] The acquisition module 10 is used to acquire the status data of the battery, wherein the status data includes the battery charging status, voltage status, ambient temperature and load demand;

[0128] An input module 20, used to input the state data into the agent strategy network, obtain the optimal action, and determine the state data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state;

[0129] The input module 20 is further used to input the state data and the optimal action into the agent evaluation network to obtain a reward value;

[0130] A generating module 30, for generating interactive experience based on the optimal action, state data, reward value and next moment state data, and storing the interactive experience in an experience recycling pool;

[0131] The determination module 40 is used to extract the target interaction experience from the experience recycling pool and adjust the parameters of the deep reinforcement learning model based on the target interaction experience, so as to realize the battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

[0132] The battery energy distribution device provided by the present application adopts the battery energy distribution method in the above embodiment, which can solve the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods. Compared with the prior art, the beneficial effects of the battery energy distribution device provided by the present application are the same as the beneficial effects of the battery energy distribution method provided by the above embodiment, and other technical features in the battery energy distribution device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0133] The present application provides a battery energy distribution device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the battery energy distribution method in the above-mentioned embodiment one.

[0134] Reference below Figure 4, which shows a schematic diagram of the structure of a battery energy distribution device suitable for implementing the embodiment of the present application. The battery energy distribution device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The battery energy distribution device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0135] like Figure 4 As shown, the battery energy distribution device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the battery energy distribution device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the battery energy distribution device to communicate wirelessly or wired with other devices to exchange data. Although the battery energy distribution device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0137] The battery energy distribution device provided by the present application adopts the battery energy distribution method in the above embodiment, which can solve the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods. Compared with the prior art, the beneficial effects of the battery energy distribution device provided by the present application are the same as the beneficial effects of the battery energy distribution method provided by the above embodiment, and the other technical features of the battery energy distribution device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the battery energy distribution method in the above-mentioned embodiment.

[0141] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0142] The computer-readable storage medium may be included in the battery energy distribution device; or may exist independently without being assembled into the battery energy distribution device.

[0143] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the battery energy distribution device, the battery energy distribution device: collects the status data of the battery, wherein the status data includes the battery charging status, voltage status, ambient temperature and load demand; inputs the status data into the intelligent agent strategy network to obtain the optimal action, and determines the status data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state; inputs the status data and the optimal action into the intelligent agent evaluation network to obtain a reward value; generates interaction experience based on the optimal action, status data, reward value and status data at the next moment, and stores the interaction experience in an experience recovery pool; extracts the target interaction experience from the experience recovery pool, and adjusts the parameters of the deep reinforcement learning model based on the target interaction experience, so as to realize the battery energy distribution after determining the current optimal action of the battery under the current status data through the optimized deep reinforcement learning model.

[0144] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0147] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned battery energy distribution method, and can solve the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the battery energy distribution method provided in the above-mentioned embodiment, and will not be repeated here.

[0148] The present application also provides a computer program product, including a computer program, which implements the steps of the battery energy distribution method as described above when executed by a processor.

[0149] The computer program product provided by the present application can solve the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the battery energy distribution method provided by the above embodiment, which will not be repeated here.

[0150] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A battery energy distribution method, characterized in that: The method comprises: Collecting battery status data, wherein the status data includes battery charging status, voltage status, ambient temperature and load demand; Input the state data into the agent strategy network to obtain the optimal action and determine the state data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state; Inputting the state data and the optimal action into the agent evaluation network to obtain a reward value; Generate interactive experience based on the optimal action, state data, reward value and next moment state data, and store the interactive experience in an experience recycling pool; Target interaction experience is extracted from the experience recycling pool, and parameters of the deep reinforcement learning model are adjusted based on the target interaction experience, so as to achieve battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

2. The method according to claim 1, characterized in that The step of inputting the state data and the optimal action into the agent evaluation network to obtain the reward value comprises: Input the state data and the optimal action into the intelligent agent evaluation network, determine the energy utilization rate reward value through the energy utilization rate reward function, determine the battery life reward value through the battery life reward function, determine the economic benefit reward value through the economic benefit reward function, and determine the response speed reward value through the response speed reward function; A reward value is determined based on the energy utilization rate reward value, the battery life reward value, the economic benefit reward value, and the response speed reward value.

3. The method according to claim 1, characterized in that The step of determining the energy utilization rate reward value by using the energy utilization rate reward function comprises: Determining actual charge and discharge power based on the state data and the optimal action; An energy utilization rate reward value is determined according to a first adjustment factor, a target charge and discharge power, and the actual charge and discharge power, wherein the first adjustment factor is used to balance the penalty intensity.

4. The method according to claim 1, characterized in that The step of determining the battery life bonus value by using the battery life bonus function comprises: determining an actual charging state based on the state data and the optimal action; Determining whether the actual charging state is within a preset reasonable range; When the actual charging state falls within the preset reasonable range, setting the battery life reward value to a positive reward; When the actual charging state does not belong to the preset reasonable range, the battery life reward value is set to a negative reward.

5. The method according to claim 1, characterized in that The step of determining the economic benefit reward value through the economic benefit reward function comprises: Based on the state data and the optimal action, determining the charging power of the battery in different charging periods and the discharging power of the battery in different discharging periods; An economic efficiency reward value is determined according to the second adjustment factor, the electricity price during the time period, the charging power, and the discharging power.

6. The method according to claim 1, characterized in that The step of determining the response speed reward value by using the response speed reward function comprises: Determining a strategy adjustment time based on the state data and the optimal action; Determine whether the strategy adjustment time is greater than a preset adjustment time; When it is determined that the strategy adjustment time is not greater than the preset adjustment time, setting the response speed reward value to a positive reward; When it is determined that the strategy adjustment time is greater than the preset adjustment time, the response speed reward value is set to a negative reward.

7. The method according to claim 1, characterized in that The step of adjusting the parameters of the deep reinforcement learning model based on the target interaction experience includes: Determine the expected value of the state cumulative reward of all state data after the state data of the target interaction experience according to the state value function; Determine the state data of the target interaction experience and the expected value of the action cumulative reward under the optimal action according to the action value function; Determine a state loss function value and an action loss function value according to the state cumulative reward expected value and the action cumulative reward expected value; Adjust the parameters of the deep reinforcement learning model according to the state loss function value and the action loss function value.

8. A battery energy distribution device, characterized in that: The device comprises: A collection module, used to collect battery status data, wherein the status data includes battery charging status, voltage status, ambient temperature and load demand; An input module, used to input the state data into the agent strategy network, obtain the optimal action, and determine the state data at the next moment, wherein the optimal action includes one of charging power, discharging power or standby state; The input module is further used to input the state data and the optimal action into the agent evaluation network to obtain a reward value; A generation module, used to generate interaction experience based on the optimal action, state data, reward value and next moment state data, and store the interaction experience in an experience recycling pool; A determination module is used to extract target interaction experience from the experience recycling pool and adjust the parameters of the deep reinforcement learning model based on the target interaction experience, so as to achieve battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

9. A battery energy distribution device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the battery energy distribution method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the battery energy distribution method according to any one of claims 1 to 7 are implemented.

Images