Battery energy allocation method, device, apparatus and storage medium

Abstract

The application discloses a battery energy distribution method and device, equipment and storage medium, relates to the technical field of battery, and includes: collecting state data of a battery; inputting the state data into an intelligent agent policy network to obtain an optimal action and determine next time state data; inputting the state data and the optimal action into an intelligent agent evaluation network to obtain a reward value; based on the optimal action, the state data, the reward value and the next time state data, generating interaction experience, and storing the interaction experience in an experience recovery pool; extracting target interaction experience from the experience recovery pool, and adjusting parameters of a deep reinforcement learning model based on the target interaction experience, so that the optimal action of the battery under the current state data is determined through the optimized deep reinforcement learning model, and the battery energy distribution is realized. The application can realize optimal battery energy distribution.

Description

Technical Field

[0001] This application relates to the field of battery technology, and in particular to battery energy distribution methods, devices, equipment and storage media. Background Technology

[0002] With the expanding market for electric vehicles and electric bicycles, batteries, as a core component of these electric devices, directly impact the convenience of charging and the overall energy efficiency of electric vehicles through their performance and charging / discharging power. Battery energy distribution strategies play a crucial role in optimizing user experience and system operating efficiency. Currently, the industry has developed various battery energy distribution methods, including rule-based strategies, model predictive control-based schemes, approaches utilizing machine learning techniques, and strategies based on various optimization algorithms. However, these methods generally suffer from the problem of unreasonable battery energy distribution.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a battery energy distribution method, device, equipment and storage medium, which aims to solve the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods.

[0005] To achieve the above objectives, this application proposes a battery energy distribution method, the battery energy distribution method comprising:

[0006] Collect battery status data, including battery charging status, voltage status, ambient temperature, and load requirements.

[0007] The state data is input into the agent policy network to obtain the optimal action and determine the state data at the next moment. The optimal action includes one of charging power, discharging power or standby state.

[0008] The state data and the optimal action are input into the agent's evaluation network to obtain a reward value;

[0009] Based on the optimal action, state data, reward value, and state data at the next moment, an interaction experience is generated and stored in the experience recycling pool.

[0010] The target interaction experience is extracted from the experience recycling pool, and the parameters of the deep reinforcement learning model are adjusted based on the target interaction experience. The optimized deep reinforcement learning model determines the current optimal action of the battery under the current state data and then realizes the battery energy distribution.

[0011] In one embodiment, the step of inputting the state data and the optimal action into the agent evaluation network to obtain a reward value includes:

[0012] The state data and the optimal action are input into the agent evaluation network. The energy utilization reward value is determined by the energy utilization reward function, the battery life reward value is determined by the battery life reward function, the economic benefit reward value is determined by the economic benefit reward function, and the response speed reward value is determined by the response speed reward function.

[0013] The reward value is determined based on the energy utilization rate reward value, the battery life reward value, the economic benefit reward value, and the response speed reward value.

[0014] In one embodiment, the step of determining the energy utilization rate reward value through the energy utilization rate reward function includes:

[0015] Based on the state data and the optimal action, the actual charging and discharging power is determined;

[0016] An energy utilization reward value is determined based on a first adjustment factor, a target charge / discharge power, and the actual charge / discharge power, wherein the first adjustment factor is used to balance the penalty intensity.

[0017] In one embodiment, the step of determining the battery life reward value using the battery life reward function includes:

[0018] Based on the state data and the optimal action, the actual charging state is determined;

[0019] Determine whether the actual charging state falls within a preset reasonable range;

[0020] When the actual charging state is within the preset reasonable range, the battery life reward value is set to a positive reward.

[0021] When the actual charging state is not within the preset reasonable range, the battery life reward value is set to a negative reward.

[0022] In one embodiment, the step of determining the economic benefit reward value through the economic benefit reward function includes:

[0023] Based on the state data and the optimal action, the charging power of the battery during different charging periods and the discharging power of the battery during different discharging periods are determined.

[0024] The economic efficiency bonus value is determined based on the second adjustment factor, the time-period electricity price, the charging power, and the discharging power.

[0025] In one embodiment, the step of determining the response speed reward value through the response speed reward function includes:

[0026] Based on the state data and the optimal action, the strategy adjustment time is determined;

[0027] Determine whether the strategy adjustment time is greater than the preset adjustment time;

[0028] When it is determined that the strategy adjustment time is not greater than the preset adjustment time, the response speed reward value is set to a positive reward;

[0029] When it is determined that the strategy adjustment time is greater than the preset adjustment time, the response speed reward value is set to a negative reward.

[0030] In one embodiment, the step of adjusting the parameters of the deep reinforcement learning model based on the target interaction experience includes:

[0031] The expected value of the cumulative state reward for all state data following the state data of the target interaction experience is determined based on the state value function.

[0032] The state data of the target interaction experience and the expected value of the cumulative reward of the action under the optimal action are determined based on the action value function.

[0033] The state loss function value and the action loss function value are determined based on the expected value of the cumulative reward of the state and the expected value of the cumulative reward of the action;

[0034] The parameters of the deep reinforcement learning model are adjusted based on the state loss function value and the action loss function value.

[0035] Furthermore, to achieve the above objectives, this application also proposes a battery energy distribution device, which includes:

[0036] The data acquisition module is used to collect the status data of the battery, including the battery charging status, voltage status, ambient temperature, and load requirements.

[0037] The input module is used to input the state data into the agent policy network to obtain the optimal action and determine the state data at the next moment. The optimal action includes one of charging power, discharging power or standby state.

[0038] The input module is also used to input the state data and the optimal action into the agent evaluation network to obtain a reward value;

[0039] The generation module is used to generate interactive experience based on the optimal action, state data, reward value and the state data at the next moment, and store the interactive experience in the experience recycling pool;

[0040] The determination module is used to extract target interaction experience from the experience recycling pool and adjust the parameters of the deep reinforcement learning model based on the target interaction experience, so as to realize battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

[0041] In addition, to achieve the above objectives, this application also proposes a battery energy distribution device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the battery energy distribution method described above.

[0042] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the battery energy distribution method described above.

[0043] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the battery energy distribution method described above.

[0044] One or more technical solutions proposed in this application have at least the following technical effects:

[0045] The battery energy distribution method, apparatus, device, and storage medium proposed in this application collect battery state data, including battery charging state, voltage state, ambient temperature, and load demand; input the state data into an intelligent agent policy network to obtain the optimal action and determine the state data for the next moment, where the optimal action includes one of charging power, discharging power, or standby state; input the state data and the optimal action into an intelligent agent evaluation network to obtain a reward value; generate interactive experience based on the optimal action, state data, reward value, and the state data for the next moment, and store the interactive experience in an experience recycling pool; extract target interactive experience from the experience recycling pool, and adjust the depth based on the target interactive experience. The parameters of the deep reinforcement learning model are optimized to determine the optimal action of the battery under the current state data, thereby achieving battery energy allocation. This solves the technical problem of unreasonable battery energy allocation in traditional battery energy allocation methods. Compared with the prior art, this application realizes dynamic decision-making in complex environments through an agent policy network and evaluates the value of dynamic policies through an agent evaluation network. The interaction experience between the agent and the environment is stored in an experience recycling pool. The historical experience in the experience recycling pool is used to continuously optimize the deep reinforcement learning model in complex environments, so that the optimized deep reinforcement learning model can find the optimal action based on the current state data, thereby achieving optimal battery energy allocation. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating an embodiment of the battery energy distribution method of this application.

[0049] Figure 2 This is a flowchart illustrating Embodiment 2 of the battery energy distribution method of this application.

[0050] Figure 3 This is a schematic diagram of the module structure of the battery energy distribution device according to an embodiment of this application;

[0051] Figure 4This is a schematic diagram of the device structure of the hardware operating environment involved in the battery energy distribution method in this application embodiment.

[0052] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0054] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of this application embodiment is as follows: Collect battery state data, including battery charging state, voltage state, ambient temperature, and load demand; input the state data into an agent policy network to obtain the optimal action and determine the state data for the next moment, wherein the optimal action includes one of charging power, discharging power, or standby state; input the state data and the optimal action into an agent evaluation network to obtain a reward value; generate interaction experience based on the optimal action, state data, reward value, and the state data for the next moment, and store the interaction experience in an experience recycling pool; extract target interaction experience from the experience recycling pool, and adjust the parameters of the deep reinforcement learning model based on the target interaction experience, so as to achieve battery energy allocation after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

[0056] As can be seen from the above embodiments, this application collects battery state data, including battery charging state, voltage state, ambient temperature, and load demand; inputs the state data into an agent policy network to obtain the optimal action and determines the state data for the next moment, where the optimal action includes one of charging power, discharging power, or standby state; inputs the state data and the optimal action into an agent evaluation network to obtain a reward value; generates interaction experience based on the optimal action, state data, reward value, and the state data for the next moment, and stores the interaction experience in an experience recycling pool; extracts target interaction experience from the experience recycling pool, and adjusts the deep reinforcement learning model based on the target interaction experience. The parameters are used to determine the optimal action of the battery under the current state data through an optimized deep reinforcement learning model to achieve battery energy allocation. This solves the technical problem of unreasonable battery energy allocation in traditional battery energy allocation methods. Compared with the prior art, this application realizes dynamic decision-making in complex environments through an agent policy network and evaluates the value of dynamic policies through an agent evaluation network. The interaction experience between the agent and the environment is stored in an experience recycling pool. The historical experience in the experience recycling pool is used to continuously optimize the deep reinforcement learning model in complex environments. The optimized deep reinforcement learning model can find the optimal action based on the current state data, thereby achieving optimal battery energy allocation.

[0057] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or battery energy distribution device capable of performing the above functions. The following description uses battery energy distribution as an example to illustrate this embodiment and the subsequent embodiments.

[0058] Based on this, embodiments of this application provide a battery energy distribution method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the battery energy distribution method of this application.

[0059] In this embodiment, the battery energy distribution method includes steps S10 to S50:

[0060] Step S10: Collect battery status data, including battery charging status, voltage status, ambient temperature, and load requirements.

[0061] It should be noted that battery status data can be collected through sensors or other data acquisition devices; battery state of charge (SOC) refers to the ratio of the battery's remaining capacity to its capacity at full charge; voltage status refers to the battery's voltage level; ambient temperature refers to the temperature of the battery and the environment; load demand refers to the battery's power supply record; historical discharge record refers to the discharge status information at the previous moment.

[0062] Step S20: Input the state data into the agent policy network to obtain the optimal action and determine the state data at the next moment. The optimal action includes one of charging power, discharging power or standby state.

[0063] It should be noted that charging power refers to the battery charging power at a historical state moment; discharging power refers to the battery discharging power at a historical state moment; standby state refers to the state in which the battery is neither charging nor discharging; the agent can use a policy network to select an optimal action based on the state data; the agent policy network refers to the policy network used by the agent in the deep reinforcement learning model, which can be used to select the optimal action based on the battery's state data.

[0064] In one feasible implementation, the step of collecting the battery status data includes: collecting initial status data of the battery; cleaning the initial status data to remove missing values, outliers, and noise values ​​to obtain cleaned data; filling in the missing parts of the cleaned data using interpolation to obtain filled data; and normalizing the filled data to obtain status data.

[0065] It should be noted that the collected initial state data can be preprocessed, mainly including data cleaning and data normalization. Data cleaning includes removing missing values, outliers, and noise values ​​from the initial data. Preprocessing the initial state data can ensure data quality and consistency, thus providing high-quality data input for model training.

[0066] Step S30: Input the state data and the optimal action into the agent evaluation network to obtain the reward value;

[0067] It should be noted that the agent evaluation network refers to the evaluation network used by the agent in the deep reinforcement learning model. It can be used to evaluate the value of the optimal action based on state data and the optimal action. The reward value can be evaluated based on factors such as battery energy utilization, battery life, economic benefits and response speed. In particular, different reward functions can be designed according to various actual working conditions to ensure that the agent in the deep reinforcement learning model can make the optimal decision in different scenarios.

[0068] Step S40: Based on the optimal action, state data, reward value, and state data at the next moment, generate interactive experience and store the interactive experience in the experience recycling pool;

[0069] It should be noted that the experience recycling pool is used to store the interaction experience generated by the agent's interaction with the environment in the deep reinforcement learning model.

[0070] In a specific implementation, it is assumed that the task of the agent in the deep reinforcement learning model is to manage the energy distribution of the battery of an electric vehicle. At each time step (e.g., every minute), the agent needs to observe the environment (i.e., obtain state data by observing the environment), make a decision (i.e., input the state data into the policy network to obtain the optimal action) and obtain feedback (i.e., input the state data and the optimal action into the evaluation network to obtain the reward value). For example: 1. The agent observes the current battery state information St (i.e., state data), including: battery state of charge (SOC), battery voltage, ambient temperature, current load requirements, etc.; Suppose the agent observes at time step t: SOC = 60%, voltage = 3.7V, temperature = 25℃, load requirement = 20W; 2. Based on the current state data St, the agent uses the policy network to select an optimal action at (i.e., optimal action). The action space includes different charging and discharging strategies, such as choosing charging, discharging, or standby states. In this example, the agent chooses "discharge 15W"; 3. The environment provides feedback reward rt (i.e., reward value). The environment provides feedback (reward value) based on the agent's action at (optimal action). This feedback is the reward value of the agent's action, representing the contribution of the action to the goal. The reward can be evaluated based on factors such as battery energy utilization rate and battery life. Assuming that discharging 15W helps maintain the battery's health and meets the load demand, the environment provides a positive reward rt = +1; 4. Observe the state data at the next time step St+1 (i.e., the state data at the next time step). After performing the 15W discharge action, the agent enters the next time step and observes the state again. For example, after 1 minute, the SOC drops to 59%, the voltage remains unchanged, the temperature rises to 26℃, and the load demand decreases to 15W. The new state (i.e., the state data at the next moment) is St+1: SOC = 59%, voltage = 3.7V, temperature = 26℃, load requirement = 15W; 5. The agent records the interaction experience during this interaction process as a historical experience and stores it in the experience recycling pool (st, at, rt, st+1) = (SOC = 60%, voltage = 3.7V, temperature = 25℃, load requirement = 20W, discharge 15W, +1, SOC = 59%, voltage = 3.7V, temperature = 26℃, load requirement = 15W); 6. The model is trained by randomly extracting target interaction experiences from the experience recycling pool to optimize the weights of the deep reinforcement learning model, so that the model can gradually learn the optimal energy allocation strategy.

[0071] Step S50: Extract target interaction experience from the experience recycling pool, and adjust the parameters of the deep reinforcement learning model based on the target interaction experience, so as to realize battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

[0072] It should be noted that target interaction experiences can be randomly extracted from the experience recycling pool, which can break the data correlation for model training; the current state data refers to the battery charging state, voltage state, ambient temperature and load demand in the current state; the current optimal action refers to the optimal action in the current state, that is, one of the charging power, discharging power or standby state.

[0073] It should be noted that in practical applications, the integrated adaptive learning mechanism enables the model to continuously optimize its strategy based on new environmental data (such as temperature changes, load changes, etc.). The model can identify new patterns in complex nonlinear spaces and dynamically adjust the energy allocation scheme. At the same time, the integrated online update mechanism enables the model to update its strategy in real time during actual operation, ensuring that it can still make optimal decisions when the environment changes.

[0074] It should be noted that the trained model was deployed to the battery management system for testing under real-world operating conditions. The tests included evaluating energy distribution under different ambient temperatures and load requirements, as well as the model's response speed and decision-making accuracy. Based on the test results, the model was further optimized, such as adjusting the reward function parameters and optimizing the neural network structure, ultimately ensuring the system's efficiency and reliability.

[0075] In one feasible implementation, the step of adjusting the parameters of the deep reinforcement learning model based on the target interaction experience includes: determining the expected value of the cumulative state reward for all state data after the state data of the target interaction experience according to the state value function; determining the expected value of the cumulative action reward for the state data and the optimal action under the target interaction experience according to the action value function; determining the state loss function value and the action loss function value according to the expected value of the cumulative state reward and the expected value of the cumulative action reward; and adjusting the parameters of the deep reinforcement learning model according to the state loss function value and the action loss function value.

[0076] In a concrete implementation, the state value function V(s) represents the expected cumulative reward for all subsequent states given a given state s, and can be defined as follows:

[0077]

[0078] Where γ is the discount factor, r(s) t a t ) is the reward for each step (i.e., the reward for each state data).

[0079] In the specific implementation, the action value function Q(s, a) represents the cumulative reward given state s and action a:

[0080] Q(s,a)=r(s,a)+γE[Q(s′,a′)|s,a]

[0081] Here, s' is the new state transitioned to based on state s and action a. DDPG uses a neural network to approximate this Q function to handle the complex and continuous state-action space.

[0082] Understandably, the state value function V(s) measures the expected value of the cumulative reward that can be obtained after taking any possible action starting from a certain state s. This expected value is calculated based on the model's current understanding of the environment and past interaction experience. In the target interaction experience, we focus on the state data s and calculate the expected value of the cumulative reward for all possible future states starting from this state.

[0083] Understandably, the action value function Q(s, a) measures the expected value of the cumulative reward that can be obtained after taking a specific action a in a given state s. In the target interaction experience, we focus on the current state data s and the action considered to be optimal a, and calculate the expected value of the cumulative reward for all possible future states starting from this state and action.

[0084] In practical implementation, the loss function is a function used in machine learning to measure the difference between the model's prediction and the actual result. In deep reinforcement learning, we define a state loss function and an action loss function to measure the prediction errors of the state value function and action value function, respectively. The state loss function value is calculated based on the difference between the predicted value of the state value function (the expected value of the cumulative reward of the state) and a preset value; the action loss function value is calculated based on the difference between the predicted value of the action value function (i.e., the expected value of the cumulative reward of the action) and a preset value. The parameters of the deep reinforcement learning model are adjusted based on the loss function values: after determining the state loss function values ​​and action loss function values, we can use these values ​​to guide the update of the model parameters. A common method is to optimize the loss function through gradient descent or its variants, that is, to adjust the model parameters to minimize the loss function value. In this process, the model continuously learns and improves its policy to better adapt to the environment and maximize the cumulative reward.

[0085] This embodiment collects battery status data, including battery charging status, voltage status, ambient temperature, and load demand. The status data is input into an agent policy network to obtain the optimal action and determine the status data for the next time step. The optimal action includes one of charging power, discharging power, or standby state. The status data and the optimal action are input into an agent evaluation network to obtain a reward value. Based on the optimal action, status data, reward value, and the status data for the next time step, interaction experience is generated and stored in an experience recycling pool. Target interaction experience is extracted from the experience recycling pool, and the parameters of the deep reinforcement learning model are adjusted based on the target interaction experience. This invention addresses the problem of unreasonable battery energy allocation in traditional battery energy allocation methods by determining the optimal action of the battery under the current state data through an optimized deep reinforcement learning model. Compared with existing technologies, this application achieves dynamic decision-making in complex environments through an agent policy network and evaluates the value of the dynamic policy through an agent evaluation network. The interaction experience between the agent and the environment is stored in an experience recycling pool. The historical experience in the experience recycling pool is used to continuously optimize the deep reinforcement learning model in complex environments, so that the optimized deep reinforcement learning model can find the optimal action based on the current state data, thereby achieving optimal battery energy allocation.

[0086] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S30 also includes steps S301 to S302:

[0087] S301, the state data and the optimal action are input into the agent evaluation network, and the energy utilization rate reward value is determined by the energy utilization rate reward function, the battery life reward value is determined by the battery life reward function, the economic benefit reward value is determined by the economic benefit reward function, and the response speed reward value is determined by the response speed reward function.

[0088] It's important to note that a reasonable reward function should be designed based on the goals of battery energy allocation, such as optimizing energy utilization, extending battery life, improving economic efficiency, or rapidly responding to environmental changes. The reward function evaluates the merits of each action and directly impacts the learning performance of the deep reinforcement learning model. Different reward functions should be designed according to various practical conditions to ensure that the agent can make optimal decisions in different scenarios.

[0089] In one feasible implementation, the step of determining the energy utilization reward value through the energy utilization reward function includes: determining the actual charging and discharging power based on the state data and the optimal action; and determining the energy utilization reward value according to the first adjustment factor, the target charging and discharging power, and the actual charging and discharging power, wherein the first adjustment factor is used to balance the penalty intensity.

[0090] It should be noted that the goal of the reward function for optimizing energy utilization is to maximize the energy utilization efficiency of the battery during charging and discharging, and avoid energy waste. The energy utilization reward function is designed as follows:

[0091] 1. If the charging and discharging power is close to the ideal value (maximizing the use of available energy without loss), a positive reward will be given;

[0092] 2. If the charging or discharging power is too high or too low, resulting in energy loss, a negative reward will be given.

[0093] The specific design is as follows:

[0094] r(s, a) = -α × (target charge / discharge power - actual charge / discharge power) 2

[0095] Where 'a' is the adjustment factor (i.e., the first adjustment factor), used to balance the intensity of the penalty;

[0096] Understandably, the smaller the difference between the actual charging / discharging power and the target charging / discharging power, the higher the reward, indicating a more reasonable energy allocation.

[0097] In one feasible implementation, the step of determining the battery life reward value through the battery life reward function includes: determining the actual charging state based on the state data and the optimal action; determining whether the actual charging state belongs to a preset reasonable range; when the actual charging state belongs to the preset reasonable range, setting the battery life reward value as a positive reward; when the actual charging state does not belong to the preset reasonable range, setting the battery life reward value as a negative reward.

[0098] It should be noted that the goal of the reward function for extending battery life is to avoid overcharging and over-discharging, maintain the battery's healthy state, and thus extend its lifespan. The battery life reward function is designed as follows:

[0099] 1. Give a positive reward when the battery's state of charge (SOC) remains within a reasonable range (e.g., 20% to 80%);

[0100] 2. If the battery's SOC exceeds a reasonable range, a negative reward will be given to prevent over-discharge or over-charge.

[0101] The specific design is as follows:

[0102]

[0103] Understandably, the battery life reward function encourages agents to keep the battery within a healthy charging range, thereby extending battery life.

[0104] In one feasible implementation, the step of determining the economic benefit reward value through the economic benefit reward function includes: determining the charging power of the battery during different charging periods and the discharging power of the battery during different discharging periods based on the state data and the optimal action; and determining the economic efficiency reward value according to the second adjustment factor, the period electricity price, the charging power, and the discharging power.

[0105] It should be noted that the objective of the reward function for improving economic efficiency is to reduce energy costs and improve the system's economic efficiency by optimizing the battery charging and discharging process. The economic efficiency reward function is designed as follows:

[0106] 1. A positive reward will be given when the battery loses power during periods of lower electricity prices (reducing costs).

[0107] 2. When the battery discharges during periods of higher electricity prices (increasing revenue), a positive reward is given.

[0108] 3. If charging occurs during peak electricity price periods or discharging occurs during off-peak periods, a negative reward will be given.

[0109] The specific design is as follows:

[0110] r(s,a)=-β×electricity price×charging power+γ×electricity price×discharging power

[0111] Here, β and γ are adjustment factors (second adjustment factors). This function encourages charging when electricity prices are low and discharging when prices are high, thereby improving economic efficiency.

[0112] In one feasible implementation, the step of determining the response speed reward value through the response speed reward function includes: determining a strategy adjustment time based on the state data and the optimal action; determining whether the strategy adjustment time is greater than a preset adjustment time; setting the response speed reward value as a positive reward when the strategy adjustment time is not greater than the preset adjustment time; and setting the response speed reward value as a negative reward when the strategy adjustment time is greater than the preset adjustment time.

[0113] It should be noted that the goal of the reward function for rapid response to environmental changes is to encourage the model to make fast responses to changes in environment and load, thereby maintaining system stability. The reward function is designed as follows:

[0114] 1. If the model quickly adjusts its strategy in response to environmental changes (such as temperature or load changes), a positive reward will be given;

[0115] 2. If the adjustment speed is too slow or the system cannot adapt to changes, a negative reward will be given.

[0116] The specific design is as follows:

[0117]

[0118] It should be noted that this reward mechanism encourages agents to make quick decisions when the environment changes in order to maintain the efficient operation of the system.

[0119] S302, determine the reward value based on the energy utilization rate reward value, the battery life reward value, the economic benefit reward value, and the response speed reward value.

[0120] It should be noted that the comprehensive reward function considers four factors simultaneously: energy utilization, battery life, economic efficiency, and response speed. The purpose of the comprehensive reward function design is to combine the reward values ​​of each objective in a weighted manner, balancing the weights between different design objectives, as detailed below:

[0121] r(s, a) = W1 * Energy Utilization Rate Bonus Value + W2 * Battery Life Bonus Value + W3 * Economic Benefit Bonus Value + W4 * Response Speed ​​Bonus Value

[0122] Among them, W1, W2, W3, and W4 are weighting coefficients that determine the importance of each objective in the total reward, so that they can be flexibly adjusted according to the needs of different scenarios.

[0123] It should be noted that the reward function design for each objective can guide the deep reinforcement learning model to learn a more reasonable charging and discharging strategy through continuous training, thereby achieving the optimal battery energy allocation scheme.

[0124] This embodiment inputs the state data and the optimal action into the agent evaluation network, determines the energy utilization reward value through an energy utilization reward function, the battery life reward value through a battery life reward function, the economic benefit reward value through an economic benefit reward function, and the response speed reward value through a response speed reward function; and determines the reward value based on the energy utilization reward value, the battery life reward value, the economic benefit reward value, and the response speed reward value. This method allows for the simultaneous consideration of energy utilization, battery life, economic benefit, and response speed when designing reward functions, ensuring that the agent in the deep reinforcement learning model makes optimal decisions in different scenarios.

[0125] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the battery energy distribution method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0126] This application also provides a battery energy distribution device; please refer to... Figure 3 The battery energy distribution device includes:

[0127] The acquisition module 10 is used to acquire the status data of the battery, wherein the status data includes the battery charging status, voltage status, ambient temperature and load requirements.

[0128] The input module 20 is used to input the state data into the agent policy network to obtain the optimal action and determine the state data at the next moment. The optimal action includes one of charging power, discharging power or standby state.

[0129] The input module 20 is also used to input the state data and the optimal action into the agent evaluation network to obtain a reward value;

[0130] The generation module 30 is used to generate interactive experience based on the optimal action, state data, reward value and the state data at the next moment, and store the interactive experience in the experience recycling pool;

[0131] The determination module 40 is used to extract target interaction experience from the experience recycling pool and adjust the parameters of the deep reinforcement learning model based on the target interaction experience, so as to realize battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model.

[0132] The battery energy distribution device provided in this application, employing the battery energy distribution method described in the above embodiments, can solve the technical problem of unreasonable battery energy distribution commonly found in traditional battery energy distribution methods. Compared with the prior art, the beneficial effects of the battery energy distribution device provided in this application are the same as those of the battery energy distribution method provided in the above embodiments, and other technical features in the battery energy distribution device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0133] This application provides a battery energy distribution device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the battery energy distribution method in the above embodiment 1.

[0134] The following is for reference. Figure 4The diagram illustrates a structural schematic suitable for implementing the battery energy distribution device of the embodiments of this application. The battery energy distribution device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The battery energy distribution device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0135] like Figure 4 As shown, the battery energy distribution device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the battery energy distribution device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the battery energy distribution device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows battery energy distribution devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0136] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0137] The battery energy distribution device provided in this application, employing the battery energy distribution method described in the above embodiments, can solve the technical problem of unreasonable battery energy distribution commonly found in traditional battery energy distribution methods. Compared with the prior art, the beneficial effects of the battery energy distribution device provided in this application are the same as those of the battery energy distribution method provided in the above embodiments, and other technical features of this battery energy distribution device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0139] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0140] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the battery energy distribution method in the above embodiments.

[0141] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0142] The aforementioned computer-readable storage medium may be included in the battery energy distribution device; or it may exist independently and not assembled into the battery energy distribution device.

[0143] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the battery energy distribution device, the battery energy distribution device performs the following actions: It collects battery state data, including battery charging state, voltage state, ambient temperature, and load demand; it inputs the state data into an agent policy network to obtain an optimal action and determines the state data for the next moment, where the optimal action includes one of charging power, discharging power, or standby state; it inputs the state data and the optimal action into an agent evaluation network to obtain a reward value; it generates interaction experience based on the optimal action, state data, reward value, and the state data for the next moment, and stores the interaction experience in an experience recycling pool; it extracts target interaction experience from the experience recycling pool and adjusts the parameters of a deep reinforcement learning model based on the target interaction experience, so that the optimized deep reinforcement learning model determines the current optimal action of the battery under the current state data and then performs battery energy distribution.

[0144] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0147] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described battery energy distribution method, which can solve the technical problem of unreasonable battery energy distribution that is common in traditional battery energy distribution methods. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the battery energy distribution method provided in the above embodiments, and will not be repeated here.

[0148] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the battery energy distribution method described above.

[0149] The computer program product provided in this application can solve the technical problem of unreasonable energy distribution in traditional battery energy distribution methods. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the battery energy distribution method provided in the above embodiments, and will not be repeated here.

[0150] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for distributing energy in a storage battery, characterized in that, The method includes: Collect battery status data, including battery charging status, voltage status, ambient temperature, and load requirements. The state data is input into the agent policy network to obtain the optimal action and determine the state data at the next moment. The optimal action includes one of charging power, discharging power or standby state. The agent policy network refers to the policy network adopted by the agent in the deep reinforcement learning model, which can be used to select the optimal action based on the state data of the battery. The state data and the optimal action are input into the agent evaluation network to obtain the reward value. The agent evaluation network refers to the evaluation network used by the agent in the deep reinforcement learning model, which can be used to evaluate the value of the optimal action based on the state data and the optimal action. Based on the optimal action, state data, reward value, and state data at the next moment, an interaction experience is generated and stored in the experience recycling pool. The target interaction experience is extracted from the experience recycling pool, and the parameters of the deep reinforcement learning model are adjusted based on the target interaction experience, so as to realize the battery energy distribution after determining the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model. The step of inputting the state data and the optimal action into the agent evaluation network to obtain the reward value includes: The state data and the optimal action are input into the agent evaluation network. The energy utilization reward value is determined by the energy utilization reward function, the battery life reward value is determined by the battery life reward function, the economic benefit reward value is determined by the economic benefit reward function, and the response speed reward value is determined by the response speed reward function. The reward value is determined based on the energy utilization rate reward value, the battery life reward value, the economic benefit reward value, and the response speed reward value.

2. The method as described in claim 1, characterized in that, The step of determining the energy utilization rate reward value through the energy utilization rate reward function includes: Based on the state data and the optimal action, the actual charging and discharging power is determined; An energy utilization reward value is determined based on a first adjustment factor, a target charge / discharge power, and the actual charge / discharge power, wherein the first adjustment factor is used to balance the penalty intensity.

3. The method as described in claim 1, characterized in that, The step of determining the battery life reward value through the battery life reward function includes: Based on the state data and the optimal action, the actual charging state is determined; Determine whether the actual charging state falls within a preset reasonable range; When the actual charging state is within the preset reasonable range, the battery life reward value is set to a positive reward. When the actual charging state is not within the preset reasonable range, the battery life reward value is set to a negative reward.

4. The method as described in claim 1, characterized in that, The step of determining the economic benefit reward value through the economic benefit reward function includes: Based on the state data and the optimal action, the charging power of the battery during different charging periods and the discharging power of the battery during different discharging periods are determined. The economic efficiency bonus value is determined based on the second adjustment factor, the time-period electricity price, the charging power, and the discharging power.

5. The method as described in claim 1, characterized in that, The step of determining the response speed reward value through the response speed reward function includes: Based on the state data and the optimal action, the strategy adjustment time is determined; Determine whether the strategy adjustment time is greater than the preset adjustment time; When it is determined that the strategy adjustment time is not greater than the preset adjustment time, the response speed reward value is set to a positive reward; When it is determined that the strategy adjustment time is greater than the preset adjustment time, the response speed reward value is set to a negative reward.

6. The method as described in claim 1, characterized in that, The step of adjusting the parameters of the deep reinforcement learning model based on the target interaction experience includes: The expected value of the cumulative state reward for all state data following the state data of the target interaction experience is determined based on the state value function. The state data of the target interaction experience and the expected value of the cumulative reward of the action under the optimal action are determined based on the action value function. The state loss function value and the action loss function value are determined based on the expected value of the cumulative reward of the state and the expected value of the cumulative reward of the action; The parameters of the deep reinforcement learning model are adjusted based on the state loss function value and the action loss function value.

7. A battery energy distribution device, characterized in that, The apparatus is used to implement the method as described in claim 1, the apparatus comprising: The data acquisition module is used to collect the status data of the battery, including the battery charging status, voltage status, ambient temperature, and load requirements. The input module is used to input the state data into the agent policy network to obtain the optimal action and determine the state data at the next moment. The optimal action includes one of charging power, discharging power or standby state. The input module is also used to input the state data and the optimal action into the agent evaluation network to obtain a reward value; The generation module is used to generate interactive experience based on the optimal action, state data, reward value and the state data at the next moment, and store the interactive experience in the experience recycling pool; The determination module is used to extract target interaction experience from the experience recycling pool and adjust the parameters of the deep reinforcement learning model based on the target interaction experience, so as to determine the current optimal action of the battery under the current state data through the optimized deep reinforcement learning model and realize the battery energy distribution.

8. A battery energy distribution device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the battery energy distribution method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the battery energy distribution method as described in any one of claims 1 to 6.

Images