Sodium-ion battery dynamic discharge strategy optimization system and method based on reinforcement learning

Abstract

The invention discloses a sodium ion battery dynamic discharge strategy optimization system and method based on reinforcement learning, and belongs to the technical field of battery discharge, the system comprises a data acquisition module, a dynamic adjustment module and a verification feedback module; the data acquisition module is used for collecting relevant data of sodium ion battery discharge in real time; and the dynamic adjustment module is used for constructing a reinforcement learning environment and a battery characteristic model, performing reinforcement learning training, optimizing a discharge strategy through local search, improving the quality of a solution, continuously executing reinforcement learning training and local search until a set number of iterations is reached, and outputting an optimized discharge strategy. Dynamic adjustment and optimization of a discharge strategy are realized; the verification feedback module is used for monitoring the discharging process of the battery in real time and feeding back the discharging process to the dynamic adjustment module, it is ensured that the battery works in the optimal state, the use efficiency of the battery is improved, and the automation and intelligence level of battery discharging is also enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of battery discharge, and relates to a system and method for optimizing a dynamic discharge strategy of a sodium ion battery based on reinforcement learning. Background Art

[0002] Electric two-wheelers and three-wheelers provide a fast and convenient way to travel with their small and flexible bodies; but due to limitations on battery capacity and technology level, their endurance is often limited and they require frequent charging, which not only affects the user experience but also limits their scope of use.

[0003] The existing Chinese patent with authorization announcement number CN118885814B discloses a battery charging and discharging optimization method, system and medium based on deep reinforcement learning. Based on historical data related to charging and discharging, a state set and a charging and discharging action strategy set are formed to train the target network model; the main network of the target network model includes a prediction actor network and a prediction critic network, wherein the prediction actor network includes a current strategy prediction network and an auxiliary strategy prediction network; the current strategy prediction network is composed of a diffusion model, which reduces the sensitivity of the model and enhances the model's strategy generation ability and strategy exploration ability; based on the auxiliary strategy prediction and the output of the target actor network, comparative learning is performed to assist in improving the model's prediction ability and solve the problem of over-estimation of the action strategy.

[0004] Although the existing technology can better cope with complex environmental changes, it does not take into account the changes in the physical and chemical properties of the battery itself, such as the battery's charge and discharge rate, temperature sensitivity, internal resistance change, self-discharge rate, and the safety and reliability of the discharge strategy in practical applications, and how to avoid safety hazards such as overcharging and over-discharging. Therefore, this application provides a system and method for optimizing the dynamic discharge strategy of sodium ion batteries based on reinforcement learning. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a system and method for optimizing the dynamic discharge strategy of sodium ion batteries based on reinforcement learning, collect battery data in real time, build a battery characteristic model, optimize the discharge strategy in combination with a reinforcement learning algorithm, and improve the quality of the solution through local search. At the same time, the battery status is monitored in real time, and the discharge strategy is adjusted in time to ensure that the battery can achieve efficient discharge under the premise of safety and stability.

[0006] To achieve the above object, the present invention provides the following technical solutions: A dynamic discharge strategy optimization system for sodium-ion batteries based on reinforcement learning, including: a data acquisition module, a dynamic adjustment module and a verification feedback module; The data acquisition module is used to collect relevant data of sodium ion battery discharge in real time; The dynamic adjustment module is used to construct a reinforcement learning environment and a battery characteristic model, perform reinforcement learning training, and optimize the discharge strategy through local search, and continuously perform reinforcement learning training and the local search until a set number of iterations is reached, and output an optimized discharge strategy; The verification feedback module is used to monitor the battery discharge process in real time and provide feedback to the dynamic adjustment module.

[0007] Furthermore, constructing a reinforcement learning environment involves defining the state space and action space , and set the reward function ; Define the state space as ,in, is the battery's state of charge, is the battery temperature, is the battery internal resistance, is the discharge current; Define the set of possible discharge current sizes as the action space ,in, is the number of possible discharge current types, For action space Middle The available discharge current; Taking into account the state of charge, temperature and discharge efficiency of the sodium-ion battery, the reward function is defined .

[0008] Furthermore, the specific steps of the reinforcement learning training include: Initialize the experience replay buffer , network and goals network , , is the network parameter; For each time step , generating the state vector , and select an action ; Execute the action , calculate the next state and rewards ; The sample Stored to the experience replay buffer middle; From the experience replay buffer Randomly select a batch of samples from Conduct training, including , Replay buffer for said experience The number of samples stored in Calculation target value ; Calculating the loss function , updated by gradient descent Network parameters , the parameters Copy to Target Network parameters middle; Set the iteration threshold to , determine the number of iterations of reinforcement learning Whether the upper limit is reached; when When , continue to carry out reinforcement learning training; when When , a local search is performed.

[0009] Furthermore, the constructed battery characteristic model includes the calculation time step State of Charge , Battery temperature And the battery internal resistance .

[0010] Furthermore, the specific steps of the local search include: Obtain the global optimal solution in the reinforcement learning training , and define the loss function is the objective function of local search ,in, is the network parameter ; In the SGD algorithm In the iterations, the objective function is calculated At the point Find the gradient , and calculate based on the gradient ; Set the gradient iteration threshold to , continue to iterate using the SGD algorithm until the gradient value is less than the gradient iteration threshold, and obtain the gradient optimization solution .

[0011] Furthermore, the specific steps of the local search also include: In the SQP algorithm In the iteration, according to the point Construct QP subproblems with constraints; Calculate search direction as well as , until the maximum number of iterations is reached, the iteration is stopped and the quadratic optimization solution is obtained ; The secondary optimization solution Add to the experience replay buffer And retrain according to the training steps of reinforcement learning; A loop is set up, and each loop performs a reinforcement learning training and a local search, and performs loop iterations until the set loop iteration limit is reached. The loop is stopped and the optimized discharge strategy of the sodium-ion battery is output.

[0012] Furthermore, the specific steps of monitoring battery discharge include: The optimized discharge strategy generated by the dynamic adjustment module is transmitted to the actual sodium ion battery through the data interface, and the battery is controlled to perform a discharge operation according to the optimized discharge strategy; Use various sensors in sodium-ion batteries to obtain data on battery parameters in real time; Set standard parameter ranges for battery parameters, including discharge rate range , Voltage Range , Current range , Temperature range ;in, , are the lower and upper limits of the discharge rate, respectively. , are the lower and upper limits of voltage respectively. , are the lower and upper limits of the current respectively, , are the lower and upper limits of temperature respectively.

[0013] Furthermore, the specific steps of monitoring battery discharge also include: like , the discharge rate is judged to be compliant; if or , it is judged that the discharge rate is abnormal; is the discharge rate; like , the voltage is determined to be compliant; if or , it is judged as voltage abnormality; is the battery voltage; like , the current is judged to be compliant; if or , it is judged as current abnormality; like , the temperature is considered to be compliant; if or , it is judged as temperature abnormality; Obtain the determination results of battery parameters. When a parameter is determined to be abnormal, an abnormal alarm is generated and abnormal adjustments are made; Record the data in the abnormal adjustment process in chronological order to generate an abnormal adjustment data set; The abnormal adjustment data set is used as a new training sample and stored in the experience playback buffer and update Parameters of the network.

[0014] The optimization method of dynamic discharge strategy of sodium-ion batteries based on reinforcement learning includes: Collect relevant data on sodium-ion battery discharge in real time; Construct a reinforcement learning environment and a battery characteristic model; the reinforcement learning environment includes the state space , Action Space And the reward function ,The battery characteristic model includes the calculation of the state of charge, battery temperature and battery internal resistance; Conduct reinforcement learning training and optimize the discharge strategy through local search until the set number of iterations is reached, and output the optimized discharge strategy; The battery discharge process is monitored in real time and fed back into reinforcement learning training.

[0015] Furthermore, the specific steps of the local search include: Get the global optimal solution , and define the objective function ; In the SGD algorithm In the iterations, the objective function is calculated At the point Find the gradient , and calculate based on the gradient ; Set the gradient iteration threshold to , continue to iterate using the SGD algorithm until the gradient value is less than the gradient iteration threshold, and obtain the gradient optimization solution ; In the SQP algorithm In the iteration, according to the point Construct QP subproblems with constraints; Calculate search direction as well as , until the maximum number of iterations is reached, the iteration is stopped and the quadratic optimization solution is obtained ; The secondary optimization solution Add to the experience replay buffer And retrain according to the training steps of reinforcement learning;

[0016] Perform loop iterations until the set loop iteration upper limit is reached, stop the loop, and output the optimized discharge strategy for the sodium-ion battery.

[0017] Beneficial effects of the present invention: Based on the reinforcement learning algorithm and combined with the battery characteristic model, a discharge strategy optimization framework with strong adaptability and high learning efficiency is constructed; by introducing the local search strategy, the local optimal problem that reinforcement learning may fall into is effectively avoided, and the quality of the discharge strategy is further improved; in the process of iterative cycles, the discharge strategy is continuously refined and optimized to ensure efficient discharge while ensuring battery safety and stability; at the same time, the battery discharge process is monitored in real time. Once abnormal parameters are found, an alarm is immediately triggered and adjustments are made to ensure that the battery is always in the best working state, which improves the battery utilization efficiency and enhances the automation and intelligence level of battery discharge. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Optimizing the system structure diagram for the dynamic discharge strategy of sodium-ion batteries based on reinforcement learning; Figure 2 Flowchart for reinforcement learning training; Figure 3 Flowchart for local search; Figure 4 Flowchart of the optimization method for dynamic discharge strategy of sodium-ion batteries based on reinforcement learning. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0020] Example 1

[0021] refer to Figures 1 to 3 As shown, this embodiment introduces a sodium ion battery dynamic discharge strategy optimization system based on reinforcement learning, including: a data acquisition module, a dynamic adjustment module, a verification feedback module and a management module; The data acquisition module is used to collect relevant data of sodium ion battery discharge in real time, including but not limited to battery voltage, current, internal resistance, temperature, state of charge, and depth of discharge, and to establish a discharge database to store historical data and real-time data for subsequent analysis and optimization; The dynamic adjustment module optimizes the sodium-ion battery discharge strategy based on the reinforcement learning algorithm, constructs a reinforcement learning environment and a battery characteristic model, wherein the battery characteristic model is used to estimate the internal state of the battery in real time and serves as the state input of the reinforcement learning environment, and performs reinforcement learning training; in some cases, reinforcement learning will converge to the local optimal solution prematurely, especially when the search space is large and there are multiple local optima. Through local search, the SGD and SQP algorithms are used to further optimize the discharge strategy, thereby improving the quality of the solution; an upper limit of the loop iteration is set, and reinforcement learning training and local search are continuously performed until the set number of iterations is reached, and the optimized discharge strategy of the sodium-ion battery is output, so as to achieve efficient discharge and dynamic adjustment and optimization of the discharge strategy while ensuring battery safety and stability; The verification feedback module is used to monitor the battery discharge process in real time and provide feedback, which helps to adjust the battery discharge strategy in time to ensure that the battery works in the best condition, and realizes automatic monitoring and adjustment. During discharge, the battery parameter data is obtained in real time and compared with the preset standard parameter range. When an abnormality is determined, an abnormal alarm is generated, and abnormal adjustments are made, and the execution results are fed back to the dynamic adjustment module. The management module is used to integrate management resources to achieve coordinated operation of vehicles and charging stations. Through real-time communication protocols such as MQTT and HTTP / HTTPS, a data interface is established with the charging station to regularly obtain detailed information about the charging station, such as location, power, and idle status, and store it in the constructed charging database to ensure the real-time and accuracy of the data. The charging reservation function is supported, and a reservation request is sent to the charging station to ensure that the vehicle can charge smoothly when it arrives at the charging station. When the vehicle arrives at the charging station, the arrival of the vehicle is confirmed and the reserved status of the charging pile is released, allowing the vehicle to start charging.

[0022] It collects information such as the location, power, and idle status of charging stations, intelligently plans charging routes and times based on vehicle location and battery status, and communicates with charging stations to implement charging reservation and scheduling functions, reduce charging waiting time, and improve charging efficiency.

[0023] Furthermore, constructing a reinforcement learning environment involves defining the state space and action space , and set the reward function ; According to the physical and chemical properties of sodium-ion batteries, the state of sodium-ion batteries is composed of multiple key parameters of sodium-ion batteries, and the state space is defined as ,in, is the battery's state of charge, is the battery temperature, is the battery internal resistance, is the discharge current; Since different discharge currents will have different effects on the discharge process of the battery, according to the actual allowable discharge current range of the sodium ion battery, the discharge current that can be taken is defined as the action. At this time, the set of discharge current sizes that can be taken is the action space ,in, is the number of types of discharge current that can be adopted; For action space Middle The available discharge current; In order to evaluate the pros and cons of an agent taking an action in a certain state, the state of charge, temperature and discharge efficiency of the sodium-ion battery are comprehensively considered. The expression of the reward function is as follows: ; ; ; ; In the formula, , , is the weight coefficient and satisfies , used to adjust the importance of different factors; for The reward function is and They are The lower and upper safety limits are Positive rewards are given when the product is within the safety lower and upper limits, and negative rewards are given when the product is too low or too high. is the reward function for temperature, is the temperature safety threshold. Negative rewards are given; is the reward function of discharge efficiency. The higher the discharge efficiency, the higher the reward. , , , is a constant.

[0024] Furthermore, the specific steps of reinforcement learning training include: Initialize the experience replay buffer , network and goals network , , is the network parameter; For each time step , obtain the current state information from the sodium ion battery detection equipment and generate the state vector , and use network and exploration strategy to select actions ; Execute an action , calculate the next state based on the battery characteristic model , and calculate the reward according to the reward function ; The sample Store to experience replay buffer It is helpful to break the correlation between data and improve the stability of training; From the Experience Replay Buffer Randomly select a batch of samples from Training is performed to reduce the correlation of data and improve the efficiency and stability of network training, among which, , Experience replay buffer The number of samples stored in Calculation target value , the expression is as follows: ; In the formula, is a discount factor used to balance short-term and long-term rewards; Define mean square error as the loss function for reinforcement learning , updated by gradient descent Network parameters , the parameters Copy to Target Network parameters In , the expression is as follows: ; ; ; In the formula, for Parameters before network update, for The updated parameters of the network, is the learning rate of reinforcement learning, For the goal Parameters before network update, For the goal The updated parameters of the network, is the update coefficient; Set the iteration threshold to , determine the number of iterations of reinforcement learning Whether the upper limit is reached, where each iteration contains multiple time steps of interaction; when When , continue to carry out reinforcement learning training; when When , a local search is performed.

[0025] Furthermore, the battery characteristic model constructed includes the calculation of the state of charge , Battery temperature And the battery internal resistance , the expression is as follows: ; ; ; ; ; In the formula, is the time step The state of charge at is the time step The discharge current at is the time step, is the rated capacity of the sodium-ion battery, is the self-discharge rate, is the time step The battery temperature at For battery quality, is the specific heat capacity of the battery, is the time step The internal resistance of is the heat generated by the battery reaction, is the heat dissipation coefficient, is the battery heat dissipation area, is the ambient temperature, for The internal resistance of the battery is is the initial internal resistance (internal resistance at reference SOC and reference temperature), and are the internal resistance changes caused by SOC and temperature, , is a constant, and For reference and reference temperature.

[0026] Furthermore, the specific steps of local search include: Obtaining the global optimal solution in reinforcement learning training , and define The network’s loss function is the objective function of local search ,in, is the network parameter ; The stochastic gradient descent (SGD) algorithm is used to guide the search direction of the global optimal individual position, update the network parameters, avoid the local optimal phenomenon, find the global optimal solution, and improve the quality of the solution; in the SGD algorithm In the iterations, the objective function is calculated At the point Find the gradient , and update the parameters according to the gradient. The expression is as follows: ; In the formula, For point The updated parameter value, is the learning rate; Set the gradient iteration threshold to , continue to use the SGD algorithm to iterate until the gradient value is less than the preset gradient iteration threshold, and the gradient optimization solution is obtained ; The sequential quadratic programming (SQP) algorithm is used to further optimize the In the iteration, according to the point The QP subproblem is constructed by using the constraints as shown below: ; ; ; In the formula, is the approximate matrix of the Hessian matrix of the Lagrangian function, is the search direction, is the equality constraint function, is the inequality constraint function; is the transpose of the search direction, is the objective function At the point The transpose of the gradient at To determine the constraints that must be met when searching for a direction, is the equality constraint function At the point The transpose of the gradient at is the inequality constraint function At the point The transpose of the gradient at ; Use the QP solver to solve the above QP subproblem and get the search direction , and update the parameters until the maximum number of iterations is reached, then stop the iteration and get the quadratic optimization solution , the expression is as follows: ; In the formula, For point The updated parameter value, is the step length; The second optimization solution The corresponding discharge strategy is converted into samples that can be used for reinforcement learning and added to the experience replay buffer and retrain according to the training steps of reinforcement learning, including sampling from the experience replay buffer, calculating the target value, and updating the network; A loop is set up, and each loop performs a reinforcement learning training and a local search. Through the cyclic iteration of reinforcement learning and local search, the performance of the discharge strategy is continuously improved, approaching the global optimal solution, until the set loop iteration upper limit is reached, the loop is stopped, and the optimized discharge strategy of the sodium-ion battery is output.

[0027] Furthermore, the specific steps of monitoring battery discharge include: The optimized discharge strategy generated by the dynamic adjustment module is transmitted to the actual sodium-ion battery through the data interface, and the battery is controlled to perform discharge operation according to the optimized discharge strategy; Using various sensors equipped in sodium ion batteries, with a certain sampling frequency Collect sensor data and obtain battery parameter data in real time. The sensors include current sensors, voltage sensors, temperature sensors and other auxiliary sensors. Battery parameters include discharge rate , battery voltage , Current and temperature ; Set standard parameter ranges for battery parameters, including discharge rate range , Voltage Range , Current range , Temperature range , and compare the real-time acquired battery parameters with the preset standard parameter range one by one; , are the lower and upper limits of the discharge rate, respectively. , are the lower and upper limits of voltage respectively. , are the lower and upper limits of the current respectively, , are the lower and upper limits of temperature respectively; For the discharge rate ,like , the discharge rate is judged to be compliant; if or , it is judged that the discharge rate is abnormal; For battery voltage ,like , the voltage is determined to be compliant; if or , it is judged as voltage abnormality; For current ,like , the current is judged to be compliant; if or , it is judged as current abnormality; For temperature ,like , the temperature is considered to be compliant; if or , it is judged as temperature abnormality; Obtain the determination results of battery parameters. When a parameter is determined to be abnormal, an abnormal alarm is generated and abnormal adjustments are made; Obtain relevant data during the abnormal adjustment process, including abnormal parameters before adjustment ( , , , ), adjusted parameters ( , , , ), the actions taken during the adjustment process (such as adjusting the proportional coefficient of the current, the connected load resistance, the power of the heat dissipation or heating system), and recording the data of each adjustment step in chronological order to generate an abnormal adjustment data set; wherein, , , , They are the discharge rate, voltage, current, and temperature before abnormal adjustment. , , , They are the discharge rate, voltage, current, and temperature after abnormal adjustment; The abnormal adjustment data set is used as a new training sample and stored in the experience playback buffer In the process, the reinforcement learning training is repeated and the parameters of the network are updated.

[0028] Example 2

[0029] See also Figure 4 Another embodiment provided by the present invention is a method for optimizing a dynamic discharge strategy of a sodium ion battery based on reinforcement learning, comprising the following steps: S1, real-time collection of sodium ion battery discharge data; S2, build a reinforcement learning environment and battery characteristic model; the reinforcement learning environment includes the state space , Action Space And the reward function ,The battery characteristic model is used to estimate the internal state of the battery in real time and serve as the state input of the reinforcement learning environment, including calculating the state of charge, battery temperature, and battery internal resistance; S3, conduct reinforcement learning training, and further optimize the discharge strategy through local search using SGD and SQP algorithms to improve the quality of the solution; set the upper limit of loop iterations, continuously perform reinforcement learning training and local search until the set number of iterations is reached, and output the optimized discharge strategy of the sodium ion battery; S4 monitors the battery discharge process in real time. During discharge, the battery parameter data is obtained in real time and compared with the preset standard parameter range. When an abnormality is determined, an abnormal alarm is generated and abnormal adjustments are made; and the execution results are fed back to the reinforcement learning training.

[0030] Furthermore, the specific steps of local search include: Obtaining the global optimal solution in reinforcement learning training , and define The network’s loss function is the objective function of local search ,in, is the network parameter ; The stochastic gradient descent (SGD) algorithm is used to guide the search direction of the global optimal individual position, update the network parameters, avoid the local optimal phenomenon, find the global optimal solution, and improve the quality of the solution; in the SGD algorithm In the iterations, the objective function is calculated At the point Find the gradient , and update the parameters according to the gradient. The expression is as follows: ; In the formula, For point The updated parameter value, is the learning rate; Set the gradient iteration threshold to , continue to use the SGD algorithm to iterate until the gradient value is less than the preset gradient iteration threshold, and the gradient optimization solution is obtained ; The sequential quadratic programming (SQP) algorithm is used to further optimize the In the iteration, according to the point The QP subproblem is constructed by using the constraints as shown below: ; ; ; In the formula, is the approximate matrix of the Hessian matrix of the Lagrangian function, is the search direction, is the equality constraint function, is the inequality constraint function; Use the QP solver to solve the above QP subproblem and get the search direction , and update the parameters until the maximum number of iterations is reached, then stop the iteration and get the quadratic optimization solution , the expression is as follows: ; In the formula, For point The updated parameter value, is the step length; The second optimization solution The corresponding discharge strategy is converted into samples that can be used for reinforcement learning and added to the experience replay buffer and retrain according to the training steps of reinforcement learning, including sampling from the experience replay buffer, calculating the target value, and updating the network; A loop is set up, and each loop performs a reinforcement learning training and a local search. Through the cyclic iteration of reinforcement learning and local search, the performance of the discharge strategy is continuously improved, approaching the global optimal solution, until the set loop iteration upper limit is reached, the loop is stopped, and the optimized discharge strategy of the sodium-ion battery is output.

[0031] In summary, the present invention utilizes a reinforcement learning algorithm, combined with a battery characteristic model, to deeply optimize the discharge strategy, and incorporates a local search strategy, and further improves the strategy quality with the help of SGD and SQP algorithms to ensure that the battery maintains its safety and stability while discharging efficiently; at the same time, the battery discharge status is monitored in real time. Once an abnormality is found, an alarm is immediately triggered and adjustments are performed to ensure that the battery is always in the best working state. The execution results are then fed back to the dynamic adjustment module to form a closed-loop optimization link.

[0032] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A dynamic discharge strategy optimization system for sodium-ion batteries based on reinforcement learning, characterized in that: include: Data acquisition module, dynamic adjustment module and verification feedback module; The data acquisition module is used to collect relevant data of sodium ion battery discharge in real time; The dynamic adjustment module is used to construct a reinforcement learning environment and a battery characteristic model, perform reinforcement learning training, and optimize the discharge strategy through local search, and continuously perform reinforcement learning training and the local search until a set number of iterations is reached, and output an optimized discharge strategy; The verification feedback module is used to monitor the battery discharge process in real time and provide feedback to the dynamic adjustment module.

2. The sodium ion battery dynamic discharge strategy optimization system based on reinforcement learning according to claim 1, characterized in that: Building a reinforcement learning environment involves defining the state space and action space , and set the reward function ; Define the state space as ,in, is the battery's state of charge, is the battery temperature, is the battery internal resistance, is the discharge current; Define the set of possible discharge current sizes as the action space ,in, is the number of types of discharge current that can be adopted; For action space Middle The available discharge current; Taking into account the state of charge, temperature and discharge efficiency of the sodium-ion battery, the reward function is defined .

3. The sodium ion battery dynamic discharge strategy optimization system based on reinforcement learning according to claim 2, characterized in that: The specific steps of the reinforcement learning training include: Initialize the experience replay buffer , network and goals network , , is the network parameter; For each time step , generating the state vector , and select an action ; Execute the action , calculate the next state and rewards ; The sample Stored to the experience replay buffer middle; From the experience replay buffer Randomly select a batch of samples from Conduct training, including , Replay buffer for said experience The number of samples stored in Calculation target value ; Calculating the loss function , updated by gradient descent Network parameters , the parameters Copy to Target Network parameters middle; Set the iteration threshold to , determine the number of iterations of reinforcement learning Whether the upper limit is reached; when When , continue to carry out reinforcement learning training; when When , a local search is performed.

4. The sodium ion battery dynamic discharge strategy optimization system based on reinforcement learning according to claim 3 is characterized in that: The constructed battery characteristic model includes the calculation time step State of Charge , Battery temperature And the battery internal resistance .

5. The sodium ion battery dynamic discharge strategy optimization system based on reinforcement learning according to claim 4 is characterized in that: The specific steps of the local search include: Obtain the global optimal solution in the reinforcement learning training , and define the loss function is the objective function of local search ,in, is the network parameter ; In the SGD algorithm In the iterations, the objective function is calculated At the point Find the gradient , and calculate based on the gradient ; Set the gradient iteration threshold to , continue to iterate using the SGD algorithm until the gradient value is less than the gradient iteration threshold, and obtain the gradient optimization solution .

6. The sodium ion battery dynamic discharge strategy optimization system based on reinforcement learning according to claim 5, characterized in that: The specific steps of the local search also include: In the SQP algorithm In the iteration, according to the point Construct QP subproblems with constraints; Calculate search direction as well as , until the maximum number of iterations is reached, the iteration is stopped and the quadratic optimization solution is obtained ; The secondary optimization solution Add to the experience replay buffer And retrain according to the training steps of reinforcement learning; A loop is set up, and each loop performs a reinforcement learning training and a local search, and performs loop iterations until the set loop iteration limit is reached. The loop is stopped and the optimized discharge strategy of the sodium-ion battery is output.

7. The sodium ion battery dynamic discharge strategy optimization system based on reinforcement learning according to claim 6, characterized in that: The specific steps to monitor battery discharge include: The optimized discharge strategy generated by the dynamic adjustment module is transmitted to the actual sodium ion battery through the data interface, and the battery is controlled to perform a discharge operation according to the optimized discharge strategy; Use various sensors in sodium-ion batteries to obtain data on battery parameters in real time; Set standard parameter ranges for battery parameters, including discharge rate range , Voltage Range , Current range , Temperature range ;in, , are the lower and upper limits of the discharge rate, respectively. , are the lower and upper limits of voltage respectively. , are the lower and upper limits of the current respectively, , are the lower and upper limits of temperature respectively.

8. The sodium ion battery dynamic discharge strategy optimization system based on reinforcement learning according to claim 7, characterized in that: The specific steps of monitoring battery discharge also include: like , the discharge rate is judged to be compliant; if or , it is judged that the discharge rate is abnormal; is the discharge rate; like , the voltage is determined to be compliant; if or , it is judged as voltage abnormality; is the battery voltage; like , the current is judged to be compliant; if or , it is judged as current abnormality; like , the temperature is considered to be compliant; if or , it is judged as temperature abnormality; Obtain the determination results of battery parameters. When a parameter is determined to be abnormal, an abnormal alarm is generated and abnormal adjustments are made; Record the data in the abnormal adjustment process in chronological order to generate an abnormal adjustment data set; The abnormal adjustment data set is used as a new training sample and stored in the experience playback buffer and update Parameters of the network.

9. A method for optimizing a dynamic discharge strategy of a sodium ion battery based on reinforcement learning, which is implemented based on a system for optimizing a dynamic discharge strategy of a sodium ion battery based on reinforcement learning as claimed in any one of claims 1 to 8, characterized in that: include: Collect relevant data on sodium-ion battery discharge in real time; Construct a reinforcement learning environment and a battery characteristic model; the reinforcement learning environment includes the state space , Action Space And the reward function ,The battery characteristic model includes the calculation of the state of charge, battery temperature and battery internal resistance; Conduct reinforcement learning training and optimize the discharge strategy through local search until the set number of iterations is reached, and output the optimized discharge strategy; The battery discharge process is monitored in real time and fed back into reinforcement learning training.

10. The method for optimizing the dynamic discharge strategy of sodium ion batteries based on reinforcement learning according to claim 9, characterized in that: The specific steps of the local search include: Get the global optimal solution , and define the objective function ; In the SGD algorithm In the iterations, the objective function is calculated At the point Find the gradient , and calculate based on the gradient ; Set the gradient iteration threshold to , continue to iterate using the SGD algorithm until the gradient value is less than the gradient iteration threshold, and obtain the gradient optimization solution ; In the SQP algorithm In the iteration, according to the point Construct QP subproblems with constraints; Calculate search direction as well as , until the maximum number of iterations is reached, the iteration is stopped and the quadratic optimization solution is obtained ; The secondary optimization solution Add to experience replay buffer And retrain according to the training steps of reinforcement learning; Perform loop iterations until the set loop iteration upper limit is reached, stop the loop, and output the optimized discharge strategy for the sodium-ion battery.

Images