Control method and device for solar energy controller in solar energy storage system

Abstract

The present invention relates to the field of energy storage control technology, and specifically provides a control method and device for a solar controller in a photovoltaic storage system, aiming to solve the problem that in the process of switching between different operating modes of the solar controller, there may be frequent switching, resulting in instability of the photovoltaic storage system. To this end, the present invention adopts a reinforcement learning method, and uses a simulation model of the photovoltaic storage system to perform multiple iterations of training on the intelligent agent of the solar controller to obtain a trained intelligent agent, and uses the trained intelligent agent to control the DC converter to operate according to a determined operating mode at the next moment. Through the above-mentioned configuration, the present invention can enable the solar controller to achieve smoother switching between different operating modes in the process of controlling the photovoltaic storage system, and also avoid frequent switching between different operating modes, so that the photovoltaic storage system can be more stable.

Description

Technical Field

[0001] The present invention relates to the field of energy storage control technology, and specifically provides a control method and device for a solar energy controller in a solar energy storage system. Background Art

[0002] In independent solar-storage systems that power satellites, solar cells are generally used as the only energy source. At the same time, in order to improve efficiency and response speed, some systems hang batteries directly on the busbar. During the illumination period, solar cells store energy through illumination to power the load and charge the battery at the same time. During the shadow period, the battery powers the load. During the operation of solar cells, the operation mode of the solar panel is often controlled by a solar controller. The solar controller switches between different operation modes based on factors such as the SOC of the battery and the power of the load. During the switching process, if the switching threshold is set unreasonably or the solar-storage system is affected by disturbances, it may cause switching back and forth between different operation modes, which will cause instability of the solar-storage system.

[0003] Accordingly, the art needs a new control scheme for a solar controller to solve the above problems. Summary of the invention

[0004] In order to solve the above technical problem, that is, to solve or partially solve the problem that when the solar controller switches between different operating modes, frequent switching may occur, resulting in instability of the solar energy storage system.

[0005] In a first aspect, the present invention provides a control method for a solar controller in a photovoltaic storage system, wherein the solar controller includes a DC converter, and the photovoltaic storage system also includes a solar panel, a storage battery, and a load, wherein a DC input side of the DC converter is connected to the solar panel, and a DC output side of the DC converter is connected to the storage battery and the load, respectively, and the storage battery is also connected to the load;

[0006] The method comprises:

[0007] Adopting a reinforcement learning method and using the simulation model of the solar energy storage system to perform multiple iterations of training on the intelligent agent of the solar energy controller to obtain a trained intelligent agent;

[0008] By using a trained intelligent agent and according to the operating mode of the DC converter in the photovoltaic storage system and the battery status parameters of the battery at the current moment, the operating mode of the DC converter at the next moment is determined and the DC converter is controlled to operate according to the determined operating mode at the next moment; wherein the battery status parameters include the input current and / or battery voltage of the battery.

[0009] In a technical solution of the control method of the solar energy controller in the above-mentioned solar energy storage system, the specific steps of each iterative training in "using a reinforcement learning method and using a simulation model of the solar energy storage system to perform multiple iterative training on the intelligent agent of the solar energy controller" include:

[0010] Step S1: randomly setting the initial SOC of the battery in the simulation model, controlling the light and temperature received by the solar panel in the simulation model to change according to the preset light curve and temperature curve respectively, and controlling the power of the load in the simulation model to change according to the preset load power curve;

[0011] Step S2: using the intelligent agent and according to the operation mode of the DC converter in the simulation model at the current moment, determining all the operation modes that the DC converter can adopt at the next moment, and using a greedy strategy to select one operation mode from all the operation modes as the final operation mode adopted by the DC converter at the next moment;

[0012] Step S3: calculating the reward value when the DC converter operates in the final operation mode according to the battery state parameters of the battery in the simulation model at the current moment; wherein the battery state parameters include the input current and / or battery voltage of the battery;

[0013] Step S4: Calculate the value of the agent according to the reward value, calculate the gradient value of the parameters of the agent according to the value, adopt the gradient descent method and update the parameters of the agent according to the gradient value to complete the current iterative training.

[0014] In a technical solution of the control method of the solar controller in the above-mentioned solar energy storage system, the operation mode includes a constant voltage mode for controlling the output voltage of the DC converter to be constant, a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be the maximum power;

[0015] The step of “calculating the reward value when the DC converter operates in the final operation mode according to the battery state parameter of the battery in the simulation model at the current moment” specifically includes:

[0016] If the operation mode of the DC converter in the simulation model at the current moment is the constant current mode or the MPPT mode switched from the constant current mode, the first reward value of the DC converter when operating in the final operation mode is calculated according to the method shown in the following formula (1):

[0017] R 1 =-(I 1 -Ir ef1 ) 2 (1)

[0018] Among them, the R 1 represents the first reward value, the I 1 represents the first input current of the battery, the I ref1 represents a preset first input current reference value, and I ref1 is a fixed constant;

[0019] If the operation mode of the DC converter in the simulation model at the current moment is the constant voltage mode or the MPPT mode switched from the constant voltage mode, the first reward value when the DC converter operates in the final operation mode is calculated according to the method shown in the following formula (2):

[0020] R 1 =-(VV ref ) 2 (2)

[0021] Among them, the R 1 represents the first reward value, V represents the battery voltage of the battery, and V ref Indicates the preset battery voltage reference value, and V ref is a fixed constant.

[0022] In a technical solution of the control method of the solar controller in the above-mentioned solar energy storage system, the operation mode includes a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power;

[0023] The step of "calculating the reward value when the DC converter operates in the final operation mode according to the battery state parameter of the battery in the simulation model at the current moment" specifically includes calculating the second reward value when the DC converter operates in the final operation mode according to the method shown in the following formula (3):

[0024] R 2 =-(I 2 -Ir ef2 ) 2 (3)

[0025] Among them, the R 2 represents the second reward value, the I 2 represents the second input current of the battery, the I ref2 Indicates a preset second input current reference value.

[0026] In a technical solution of the control method of the solar controller in the above-mentioned solar energy storage system, the method further includes obtaining the second input current reference value I according to the method shown in the following formula (4): ref2 :

[0027] I ref2 = k × e -bt (4)

[0028] Wherein, k and b are preset coefficients, and t is the charging time of the battery;

[0029] or,

[0030] The method further comprises obtaining the second input current reference value I by the following steps ref2 :

[0031] According to a minimum value and a maximum value of the SOC of the battery, dividing the SOC interval between the minimum value and the maximum value into a plurality of SOC sub-intervals;

[0032] A different second input current reference value I is set for each SOC sub-interval ref2 ;

[0033] According to the order of the values ​​corresponding to the starting endpoints of each SOC sub-interval from small to large, the second input current reference value I corresponding to each SOC sub-interval is ref2 Gradually increase.

[0034] In a second aspect, the present invention provides a control device for a solar controller in a photovoltaic storage system, wherein the solar controller includes a DC converter, and the photovoltaic storage system also includes a solar panel, a battery, and a load, wherein a DC input side of the DC converter is connected to the solar panel, and a DC output side of the DC converter is connected to the battery and the load, respectively, and the battery is also connected to the load;

[0035] The device comprises:

[0036] An intelligent agent training module, which is configured to adopt a reinforcement learning method and use the simulation model of the solar energy storage system to perform multiple iterations of training on the intelligent agent of the solar energy controller to obtain a trained intelligent agent;

[0037] An operation mode determination module is configured to use a trained intelligent agent and, based on the operation mode of the DC converter in the photovoltaic storage system and the battery status parameters of the battery at the current moment, determine the operation mode of the DC converter at the next moment and control the DC converter to operate according to the determined operation mode at the next moment; wherein the battery status parameters include the input current and / or battery voltage of the battery.

[0038] In one technical solution of the control device for a solar energy controller in a solar energy storage system, the agent training module includes:

[0039] A simulation model parameter setting unit, configured to randomly set an initial SOC of a battery in the simulation model, and to control the light and temperature received by a solar panel in the simulation model to change according to a preset light curve and temperature curve, respectively, and to control the power of a load in the simulation model to change according to a preset load power curve;

[0040] a final operation mode determination unit, configured to use the intelligent agent and determine all operation modes that the DC converter can adopt at the next moment according to the operation mode of the DC converter in the simulation model at the current moment, and select one operation mode from all the operation modes by adopting a greedy strategy as the final operation mode adopted by the DC converter at the next moment;

[0041] a reward value calculation unit, configured to calculate a reward value when the DC converter operates in the final operation mode according to a battery state parameter of the battery in the simulation model at a current moment; wherein the battery state parameter includes an input current and / or a battery voltage of the battery;

[0042] An agent parameter updating unit is configured to calculate the value of the agent according to the reward value, calculate the gradient value of the agent's parameters according to the value, adopt a gradient descent method and update the agent's parameters according to the gradient value to complete the current iterative training.

[0043] In a technical solution of the control device for a solar controller in a solar energy storage system, the operation mode includes a constant voltage mode for controlling the output voltage of the DC converter to be constant, a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be the maximum power;

[0044] The reward value calculation unit is further configured to perform the following steps:

[0045] If the operation mode of the DC converter in the simulation model at the current moment is the constant current mode or the MPPT mode switched from the constant current mode, the first reward value of the DC converter when operating in the final operation mode is calculated according to the method shown in the following formula (1):

[0046] R 1 =-(I 1 -I ref1 ) 2 (1)

[0047] Among them, the R 1 represents the first reward value, the I 1 represents the first input current of the battery, the I ref1 represents a preset first input current reference value, and Iref1 is a fixed constant;

[0048] If the operation mode of the DC converter in the simulation model at the current moment is the constant voltage mode or the MPPT mode switched from the constant voltage mode, the first reward value when the DC converter operates in the final operation mode is calculated according to the method shown in the following formula (2):

[0049] R 1 =-(VV ref ) 2 (2)

[0050] Among them, the R 1 represents the first reward value, V represents the battery voltage of the battery, and V ref Indicates the preset battery voltage reference value, and V ref is a fixed constant.

[0051] In a technical solution of the control device for a solar energy controller in a solar energy storage system, the operation mode includes a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power;

[0052] The reward value calculation unit is further configured to perform the following steps:

[0053] The second reward value when the DC converter operates in the final operation mode is calculated according to the method shown in the following formula (3):

[0054] R 2 =-(I 2 -I ref2 ) 2 (3)

[0055] Among them, the R 2 represents the second reward value, the I 2 represents the second input current of the battery, the I ref2 Indicates a preset second input current reference value.

[0056] In a technical solution of the control device for a solar energy controller in the above-mentioned solar energy storage system, the device further includes a second input current reference value first acquisition module and a second input current reference value second acquisition module, and the second input current reference value first acquisition module is configured to acquire the second input current reference value I according to the method shown in the following formula (4): ref2 :

[0057] I ref2 = k × e -bt (4)

[0058] Wherein, k and b are preset coefficients, and t is the charging time of the battery;

[0059] The second input current reference value first acquisition module is configured to acquire the second input current reference value I according to the following steps: ref2 :

[0060] According to a minimum value and a maximum value of the SOC of the battery, dividing the SOC interval between the minimum value and the maximum value into a plurality of SOC sub-intervals;

[0061] A different second input current reference value I is set for each SOC sub-interval ref2 ;

[0062] According to the order of the values ​​corresponding to the starting endpoints of each SOC sub-interval from small to large, the second input current reference value I corresponding to each SOC sub-interval is ref2 Gradually increase.

[0063] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0064] In the technical solution of the present invention, a reinforcement learning method is adopted, and the simulation model of the solar energy storage system is used to perform multiple iterations of training on the intelligent agent of the solar energy controller to obtain a trained intelligent agent, and the trained intelligent agent is used to determine the operation mode of the DC converter at the next moment according to the operation mode of the DC converter in the solar energy storage system at the current moment and the battery state parameters of the battery, and control the DC converter to operate according to the determined operation mode at the next moment. Through the above configuration, the present invention can use the trained intelligent agent to control the operation mode of the DC converter in the solar energy controller according to the operation mode of the DC converter in the solar energy storage system at the current moment and the battery state parameters of the battery, so that the solar energy controller can achieve smoother switching between different operation modes in the process of controlling the solar energy storage system, and also avoid frequent switching between different operation modes, so that the solar energy storage system can be more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. Among them:

[0066] Figure 1 is a schematic flow chart of main steps of a control method for a solar energy controller in a solar energy storage system according to an embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of the main components of a photovoltaic storage system according to an embodiment of the present invention;

[0068] Figure 3 is a schematic diagram of the main framework for training an intelligent agent according to an implementation of an embodiment of the present invention;

[0069] Figure 4 is a schematic diagram of a main framework for training an intelligent agent according to another implementation of an embodiment of the present invention;

[0070] Figure 5 is a flow chart of main steps of iteratively training an agent using a simulation model of an optical storage system according to an implementation of an embodiment of the present invention;

[0071] Figure 6 It is a main structural block diagram of a control device for a solar energy controller in a solar energy storage system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0073] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memory, and may also include software parts, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and the like. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.

[0074] See attached Figure 1 and attached Figure 2 , Figure 1 is a schematic flow chart of the main steps of a control method for a solar energy controller in a solar energy storage system according to an embodiment of the present invention. Figure 2 FIG. 1 is a schematic diagram of the main components of a photovoltaic storage system according to an embodiment of the present invention. Figure 2As shown, the solar controller in the embodiment of the present invention may include a DC converter, and the solar energy storage system may also include a solar panel, a battery and a load. The DC input side of the DC converter may be connected to the solar panel, and the DC output side of the DC converter may be connected to the battery and the load respectively, and the battery may also be connected to the load. Figure 1 As shown, the control method for a solar energy controller in a solar energy storage system in the implementation of the present invention mainly includes the following steps S101 to S102.

[0075] Step S101: adopting a reinforcement learning method and using a simulation model of a solar energy storage system to perform multiple iterations of training on the intelligent agent of the solar controller to obtain a trained intelligent agent.

[0076] In this embodiment, the solar panels and batteries in the photovoltaic storage system are connected to the load and can both supply power to the load, and the solar panels can also charge the batteries. The reinforcement learning method can be used to apply the simulation model of the photovoltaic storage system to perform multiple iterations of training on the intelligent agent of the solar controller to obtain a trained intelligent agent. Among them, the reinforcement learning method is a conventional learning method in the field of machine learning technology, which specifically refers to a method for describing and solving the problem of how the intelligent agent can achieve reward maximization or specific goals through learning strategies during the interaction process of the intelligent agent in the environment.

[0077] In one embodiment, the photovoltaic storage system can be applied to a satellite system, and the photovoltaic storage system can supply power to the satellite system. The photovoltaic storage system can adopt a method of not regulating the bus (the DC bus on the DC output side of the DC converter), that is, directly connecting the battery to the bus. During the light period, the bus voltage will be affected by the output power of the solar controller; during the shadow period, the bus voltage depends on the battery voltage.

[0078] In one embodiment, the load can be connected to the output side of the battery and the DC converter respectively through a secondary power supply. The secondary power supply refers to a device that converts the electric energy of the main power supply into another form or specification of electric energy to meet the needs of different electrical equipment. In this embodiment, the secondary power supply can be a voltage converter.

[0079] Step S102: Using the trained intelligent agent and according to the operation mode of the DC converter in the photovoltaic storage system and the battery state parameters of the battery at the current moment, determine the operation mode of the DC converter at the next moment and control the DC converter to operate according to the determined operation mode at the next moment.

[0080] In this embodiment, the intelligent agent trained in step S101 can be used to determine the operation mode of the DC converter at the next moment according to the operation mode of the DC converter in the photovoltaic energy storage system and the battery state parameters of the battery at the current moment, and control the DC converter to operate according to the determined operation mode at the next moment. The battery state parameters may include the input current of the battery and the battery voltage of the battery.

[0081] Based on the above steps S101-S102, the present invention can use a reinforcement learning method and use the simulation model of the solar energy storage system to perform multiple iterations of training on the intelligent agent of the solar energy controller to obtain a trained intelligent agent, and use the trained intelligent agent to determine the operation mode of the DC converter at the next moment according to the operation mode of the DC converter in the solar energy storage system at the current moment and the battery state parameters of the battery, and control the DC converter to operate according to the determined operation mode at the next moment. Through the above configuration, the present invention can use a trained intelligent agent to control the operation mode of the DC converter in the solar energy controller according to the operation mode of the DC converter in the solar energy storage system at the current moment and the battery state parameters of the battery. Since the intelligent agent is obtained by iterative training using a reinforcement learning method, the solar energy controller using the intelligent agent can achieve smoother switching between different operation modes in the process of controlling the solar energy storage system, and also avoid frequent switching between different operation modes, so that the solar energy storage system can be more stable.

[0082] The step S101 is further described below.

[0083] In one implementation of the embodiment of the present invention, step S101 may further include the following steps to implement each iteration training:

[0084] Step S1011: randomly setting the initial SOC of the battery in the simulation model, and controlling the light and temperature received by the solar panel in the simulation model to change according to the preset light curve and temperature curve respectively, and controlling the power of the load in the simulation model to change according to the preset load power curve.

[0085] In this embodiment, the initial SOC of the battery in the simulation model of the photovoltaic storage system can be randomly set, and the light and temperature received by the solar panel in the simulation model are controlled to change according to the preset light curve and temperature curve, respectively, and the power of the load in the simulation model is controlled to change according to the preset load power curve. That is to say, the light curve, temperature curve and load power curve can be set in the simulation model, so that the light received by the solar panel in the simulation model changes according to the light curve, the temperature changes according to the temperature curve, and the power of the load changes according to the load power curve.

[0086] Step S1012: using an intelligent agent and according to the operation mode of the DC converter in the simulation model at the current moment, determining all the operation modes that the DC converter can adopt at the next moment, and using a greedy strategy to select one operation mode from all the operation modes as the final operation mode adopted by the DC converter at the next moment.

[0087] In this embodiment, an intelligent agent can be used to determine all the operating modes that the DC converter can adopt at the next moment according to the operating mode of the DC converter in the simulation model at the current moment, and a greedy strategy can be further applied to select an operating mode from all the determined operating modes as the final operating mode adopted by the DC converter at the next moment. Among them, the greedy strategy or greedy algorithm is a conventional problem-solving algorithm in the field of mathematical technology, which always makes the best choice at the moment when solving the problem. For the sake of brevity, the specific working principle or algorithm of the greedy strategy or greedy algorithm will not be described here.

[0088] Step S1013: Calculate the reward value when the DC converter operates in the final operation mode according to the battery state parameters of the battery in the simulation model at the current moment; wherein the battery state parameters include the input current and / or battery voltage of the battery.

[0089] In this implementation, the reward value when the DC converter operates in the final operation mode may be calculated based on the battery state parameters of the battery in the simulation model at the current moment.

[0090] In one embodiment, the operation modes of the DC converter include a constant voltage mode for controlling the output voltage of the DC converter to be constant, a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT (Maximum Power Point Tracking) mode for controlling the output power of the DC converter to be maximum power. When the load demand in the photovoltaic storage system is large, or the output power of the solar panel is insufficient, the DC converter can operate in the MPPT mode to meet the needs of the photovoltaic storage system as much as possible.

[0091] In this implementation, step S1013 may further include the following steps:

[0092] Step S10131: If the operation mode of the DC converter in the simulation model at the current moment is the constant current mode or the MPPT mode switched from the constant current mode, the first reward value when the DC converter operates in the final operation mode can be calculated according to the method shown in the following formula (1):

[0093] R 1 =-(I 1 -I ref1 ) 2(1)

[0094] Among them, R 1 Represents the first reward value, I 1 Represents the first input current of the battery, I ref1 represents a preset first input current reference value, and I ref1 is a fixed constant;

[0095] Step S10132: If the operation mode of the DC converter in the simulation model at the current moment is the constant voltage mode or the MPPT mode switched from the constant voltage mode, the first reward value when the DC converter operates in the final operation mode can be calculated according to the method shown in the following formula (2):

[0096] R 1 =-(VV ref ) 2 (2)

[0097] Among them, R 1 represents the first reward value, V represents the battery voltage of the battery, V ref Indicates the preset battery voltage reference value, and V ref is a fixed constant.

[0098] Specifically, in this embodiment, the battery can be charged in a constant current mode first and then in a constant voltage mode, that is, when the battery SOC is small, the constant current mode is used for charging, and when the battery voltage of the battery reaches a preset first voltage threshold, the constant current mode is switched to the constant voltage mode. Similarly, during the charging process, the battery may also supply power to the load at the same time, and when the battery voltage of the battery drops to a preset second voltage threshold, the constant voltage mode can be switched to the constant current mode for charging.

[0099] If the operation mode of the DC converter in the simulation model at the current moment is the constant current mode, or when switching between the constant current mode and the MPPT mode, the first reward value when the DC converter operates in the final operation mode can be calculated according to the method shown in formula (1).

[0100] If the operation mode of the DC converter in the simulation model at the current moment is the constant voltage mode, or when switching between the constant voltage mode and the MPPT mode, the first reward value when the DC converter operates in the final operation mode can be calculated according to the method shown in formula (2).

[0101] The first voltage threshold is greater than the second voltage threshold. Those skilled in the art can determine the first voltage threshold, the second voltage threshold, the first input current reference value and the battery voltage reference value according to parameters provided by the battery manufacturer.

[0102] In one embodiment, the operation mode of the DC converter may include a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power. In this embodiment, step S1013 may be further configured to calculate the second reward value when the DC converter operates in the final operation mode according to the method shown in the following formula (3):

[0103] R 2 =-(I 2 -I ref2 ) 2 (3)

[0104] Among them, R 2 Represents the second reward value, I 2 Represents the second input current of the battery, I ref2 Indicates a preset second input current reference value.

[0105] Specifically, in this embodiment, the operation mode of the DC converter includes a constant current mode for controlling the output current of the DC converter to be constant and an MPPT mode for controlling the output power of the DC converter to be maximum power. The second reward value when the DC converter operates in the final operation mode can be calculated according to the method shown in formula (3).

[0106] In one implementation, the second input current reference value I can be obtained according to the method shown in the following formula (4): ref2 :

[0107] I ref2 = k × e -bt (4)

[0108] Wherein, k and b are preset coefficients, and t is the charging time of the battery.

[0109] In this embodiment, the second input current reference value is an exponential function of the charging time, and the second input current reference value can be calculated according to the method shown in formula (4).

[0110] In one implementation, the second input current reference value I can be obtained by the following steps: ref2 :

[0111] Step S103: dividing the SOC interval between the minimum value and the maximum value into a plurality of SOC sub-intervals according to the minimum value and the maximum value of the SOC of the battery;

[0112] Step S104: Setting a different second input current reference value I for each SOC sub-interval ref2 Wherein, according to the order of the values ​​corresponding to the starting endpoints of each SOC sub-interval from small to large, the second input current reference value I corresponding to each SOC sub-intervalref2 Gradually increase.

[0113] In this embodiment, the second input current reference value is a step function between the minimum and maximum values ​​of the SOC of the battery. That is, the discrete second input current reference value can be set according to the SOC interval of the battery. min To SOC max , then SOC min ~SOC 1 , I ref2 =I 1 ;SOC 1 ~SOC 2 , I ref2 =I 2 ;……;SOC n ~SOC max , I ref2 =I n+1 Those skilled in the art can determine the second input current reference value, the minimum value and the maximum value of the SOC for each SOC interval according to the battery parameters provided by the battery manufacturer.

[0114] Step S1014: Calculate the value of the agent according to the reward value, calculate the gradient value of the agent's parameters according to the value, use the gradient descent method and update the agent's parameters according to the gradient value to complete the current iterative training.

[0115] In this embodiment, the value of the agent can be calculated according to the reward value obtained in step S1013, and the gradient value of the agent's parameters can be further calculated according to the value, and the gradient descent method is used to update the agent's parameters according to the gradient value to complete the current iterative training. Then jump to step S1011 for the next iterative training.

[0116] Specifically, the elements of the reinforcement learning method are usually represented as a tuple (S, A, P, R), where S represents the set of states, A represents the set of actions, P represents the probability of state transition, and R represents the reward function. The reinforcement learning process observes the current state of the environment s∈S and determines the action a∈A to be taken based on the state. A state may correspond to multiple actions, and each action has a probability p∈P of being selected, and p is determined by the strategy π. The function of the strategy π can be expressed by formula (5):

[0117] π(a|s)=p[A t =a|S t =s] (5)

[0118] Among them, π(a|s) is the strategy when the current state is s and the action to be taken is a, A t is the set of actions in the t-th iteration training, St The set of states in the tth iteration of training.

[0119] After selecting an action according to the state, a reward is generated. The value of reinforcement learning is composed of the current reward and the discounted future reward. If the reward is positive, the probability of executing the reward increases. In state s, the value corresponding to action a is shown in formula (6):

[0120] Q π (s,a)≡E[R 1 +γR 2 +…|S 0 =s,A 0 =a,π] (6)

[0121] Among them, γ∈[0,1] is the discount factor, E represents the expected function of the reward, S 0 Represents the set of states of the current iteration training, A 0 Represents the set of actions for the current iteration training, R 1 Represents the reward for the first iteration of training, R 2 Represents the reward for the second iteration of training.

[0122] Using a greedy strategy or greedy algorithm, you can select the action with the maximum current value. Then the current maximum value is shown in formula (7):

[0123] Q * (s,a)=max π Q π (s,a) (7)

[0124] Among them, Q * (s,a) is the current maximum value.

[0125] However, in actual applications, the probability P of a state being selected and the immediate reward corresponding to each action or state may be unknown. Q-learning is a reinforcement learning algorithm based on an environment model and a value function. Its main idea is to define the value function of the state and action, and after randomly initializing the parameters, substitute the observed data (the state, action, reward of the current iterative training, and the state of the next iterative training) into the value function, and perform iterative training according to the method shown in formula (8).

[0126]

[0127] Among them, s t and t+1are the states in the current iteration training (t-th iteration training) and the next iteration training (t+1-th iteration training), respectively. In the implementation of the embodiment of the present invention, the state may include the operation mode of the DC converter and the battery state of the battery. t is the action of the current iterative training. In the implementation of the embodiment of the present invention, the action may be to select an operation mode from all the operation modes that the DC converter can adopt as the final operation mode adopted by the DC converter at the next moment, α is the learning rate, R t represents the reward for the t-th iteration of training.

[0128] When the states and actions are finite, the Q-learning method can generate a table. However, when the states or actions are continuous, the strategy of using the table is not feasible. Using the parameterized value function Q(s t ,a t |θ) as a function approximator can handle situations where states or actions are continuous. Among them, the function approximator means that for a more complex function, a relatively simple and easy-to-calculate function can be set first. Through continuous iterative training, the parameters of this relatively simple function are updated so that the relatively simple function is constantly close to the more complex function. This relatively simple function is the function approximator, and θ is the parameter of the function approximator. In reinforcement learning, this more complex function can be called the target network, and the function approximator is called an online network. The updating method is the gradient descent method, so that the value of the current action is close to the target value of the reward.

[0129] In this embodiment, the gradient descent method can be applied to update the parameters of the agent according to the following formula (9):

[0130]

[0131] Among them, θ t represents the parameters of the agent in the tth iteration training, θ t+1 represents the parameters of the agent in the t+1th iteration training, S t represents the set of states of the photovoltaic storage system in the t-th iteration training. The state of the photovoltaic storage system may include the operation mode of the DC converter and the battery state of the battery. t represents the action set in the t-th iteration training. The action of the agent may include selecting an operation mode from all the operation modes that the DC converter can adopt as the final operation mode adopted by the DC converter at the next moment. Q(S t ,A t θ t ) represents the action set A in the tth iteration training t The value function of represents the target value of the reward output by the agent, α represents the preset learning rate, Represents the parameter θ calculated according to the value function in the t-th iteration training t The gradient value of the agent's parameters is calculated so that the gradient value develops in a gradually decreasing direction. The agent's parameters are updated according to the gradient value through back propagation to achieve the training of the agent.

[0132] The purpose of training the agent is to make the reward value output by the agent close to the target value of the reward. In the general reinforcement learning process, the target value of the reward can be obtained according to the following formula (10):

[0133]

[0134] in, is the target value of the reward in the general reinforcement learning process.

[0135] In the Deep Q-Network (DQN), considering the impact of the target network on the target value of the reward, the parameter θ is increased. - , where θ - is the parameter of the target network, θ - It is copied from the parameters θ of the online network every few iterations of training. The training method through the DQN network is a training method that replays the experience (i.e., copies the parameters of the online network to the target network), which can make the training process easier to converge and more stable. Among them, the target value of the reward of the DQN network can be obtained according to formula (11):

[0136]

[0137] in, is the target value of the reward of the DQN network, R t+1 represents the reward for the t+1th iteration training, Represents the value of the target network with the state of the t+1th iteration training, the parameters of the target network, and the action with the maximum value as parameters.

[0138] In this embodiment, the agent is iteratively trained so that the reward value of the DC converter controlled by the agent when it runs in the final operation mode is closer to

[0139] In one embodiment, when the operating mode includes a constant voltage mode for controlling the output voltage of the DC converter to be constant, a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power, the value of the intelligent agent can be calculated based on the first reward value.

[0140] In another embodiment, when the operation mode includes a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power, the value of the agent can be calculated based on the second reward value.

[0141] In one embodiment, when the battery is charged in a constant voltage mode, the intelligent agent can compensate the battery according to the difference between the current battery voltage and the reference value of the battery voltage, so that the battery voltage of the compensated battery is as close to the battery voltage reference value as possible. When the battery is charged in a constant current mode, the intelligent agent can compensate the battery according to the difference between the current input current of the battery (the first input current or the second input current) and the reference value of the input current (the first input current reference value or the second input current reference value), so that the input current of the compensated battery is as close to the input current reference value as possible. That is, the input current of the battery is used as the control quantity, the difference between the input current and the reference value is used as the feedback quantity, and the input current of the battery is negatively feedback controlled, so that the input current of the battery continuously tracks the reference value of the input current. In this embodiment, a conventional control method in the field of automatic control technology can be used, and the input current of the battery is used as the control quantity, and the difference between the input current and the reference value is used as the feedback quantity, and the input current of the battery is negatively feedback controlled. For example, a PI control method or a PID control method can be used to negatively feedback control the input current of the battery.

[0142] In one implementation, some general rewards can be set to protect the battery and also help speed up the iterative training process of the agent. max , the minimum input current I min , the maximum value of the battery state of charge SOC max , minimum state of charge SOC min If one of the following situations occurs, a large penalty is imposed on the agent and the iterative training is terminated immediately to protect the battery: I>I max ; I<I min ; SOC>SOC max ; SOC<SOC min The absolute value of the penalty must be greater than the absolute value of the reward for completing one iteration of training in the worst case.

[0143] See the attached Figure 3 , Attachment Figure 4 and attached Figure 5 The main steps of iterative training of the agent are described in detail. Figure 3 is a schematic diagram of the main framework for training an intelligent agent according to an implementation of an embodiment of the present invention; Figure 4 is a schematic diagram of a main framework for training an intelligent agent according to another implementation of an embodiment of the present invention; Figure 5 It is a flowchart of the main steps of iteratively training an intelligent agent using a simulation model of a photovoltaic storage system according to an implementation of an embodiment of the present invention.

[0144] like Figure 3 As shown, the DQN model (simulation model) obtains the current operating mode (state) of the DC converter according to the environment of the photovoltaic storage system, as well as all the operating modes that the DC converter can adopt at the next moment and the rewards corresponding to each operating mode, and selects one operating mode from all operating modes as the final operating mode (mode selection) adopted by the DC converter at the next moment, and compensates according to the actual input current or battery voltage of the battery and the corresponding reference value. The DC converter obtains the output signal required according to the output signal (PWM), operating mode, duty cycle and compensation of the solar panel. When the DC converter is in constant current mode, the signal output by the DC converter is the input current of the battery; when the DC converter is in constant voltage mode, the signal output by the DC converter is the battery voltage of the battery; when the DC converter is in MPPT mode, the DC converter outputs the voltage across the solar panel. As shown Figure 4 As shown in the figure, when the battery needs to be charged at a constant voltage, if the output power of the solar panel is insufficient, the actual voltage output by the DC converter is insufficient to meet the requirement of continuing to use the constant voltage mode. L ) and the battery voltage reference value (V ref ) there is a difference between them, you can use Figure 4 The method shown is performed by the agent on V ref With V L The difference is compensated, and the PWM signal output by the solar panel is controlled according to the compensation result through the PI control method, and the constant voltage signal is output through the DC converter (solar array converter) to charge the battery at a constant voltage. When the battery needs to be charged with a constant current, if the output power of the solar panel is insufficient, the actual current output by the DC converter is insufficient to meet the continuous use of the constant current mode, and the actual current output by the DC converter (I L ) and the input current reference value (I ref ) there is a difference between them, you can use Figure 4 The way in which the agent ref with I LThe difference is compensated, and the PWM signal output by the solar panel is controlled according to the compensated result through PI control, and a constant current signal is output through the DC converter (solar array converter) to charge the battery with a constant current. When the DC converter is in MPPT mode, no compensation is required, and the PWM signal corresponding to the MPPT mode is directly output to realize the function of the solar controller. Among them, PI control refers to a method of controlling the controlled object by linearly combining the proportion and integral of the deviation to form a control quantity based on the control deviation formed by the given value and the actual value.

[0145] In one embodiment, if Figure 4 As shown in Figure 1, a simulation model including solar panels, DC converters, batteries and loads can be constructed first, and then Figure 5 The following steps are shown to iterate the training of the agent:

[0146] Step S201: Setting the certainty information of the simulation model.

[0147] In this embodiment, the deterministic information of the simulation model may include the voltage curve and current curve of the solar panel, the capacity, internal resistance and charging curve of the battery, etc. The deterministic information is determined according to the manufacturer parameters of the solar panel and the battery.

[0148] Step S202: Setting reinforcement learning agent parameters and training parameters, n=0.

[0149] In this embodiment, the agent parameters, including learning rate, optimizer, memory size, and iterative training batch size, can be randomly initialized, and the number of initialization training times n = 0. The iterative training batch size refers to the upper limit of the time of iterative cycle training.

[0150] Step S203: Initialize the uncertainty information of the simulation model.

[0151] In this embodiment, the uncertain information includes the light and temperature curves received by the solar panel, the initial SOC of the battery, the power curve of the load, etc. The uncertainty information is randomly generated within a reasonable range at the beginning of each iterative cycle training.

[0152] Step S204: According to the external information of the system, all parameters of the system are obtained by the simulation model, t=0.

[0153] In this embodiment, all parameters of the photovoltaic storage system including the output power of the solar panel and the input (or output) power of the battery can be determined based on the determined information obtained in step S201 and the uncertain information obtained in step S203.

[0154] Specifically, without considering the conversion efficiency, the relationship between the output power of the solar panel and the input (or output) power of the battery can be determined according to formula (12):

[0155] P pv +P bat =P load (12)

[0156] Among them, P pv is the output power of the solar panel, and its value must be positive; P bat is the battery input (or output) power, when P bat >0, indicating that the battery is discharged. bat <0 means the battery is charged; P load is the load power, and its value must be positive.

[0157] In one embodiment, the DC converter may be a buck converter, whose output voltage V L1 By formula (13), we can obtain:

[0158] V L1 =V PV ×α 1 (13)

[0159] Among them, V L1 is the output voltage of the buck converter, V PV is the output voltage of the solar panel, α 1 is the duty cycle.

[0160] The battery voltage of the battery can be approximately obtained according to the following formula (14):

[0161] V bat =f(SOC)+I bat ×r(14)

[0162] Among them, f(SOC) is a function with SOC as a variable, I bat It represents the input current of the battery. When the current flows into the battery, the input current of the battery is positive. When the current flows out of the battery, the input current of the battery is negative. r is the internal resistance of the battery. If the steady-state of other impedances is ignored, it can be considered that V L1 =V bat .

[0163] In practical applications, the simulation model can be determined according to the parameters of the DC converter. If conditions permit, a more accurate model can be established by using data-driven modeling. Data-driven modeling refers to the method of collecting massive amounts of data, organizing the data into information, and then integrating and refining the relevant information, and then training and fitting the data to form an automated decision-making model.

[0164] Step S205: Obtain observation information at the current time t.

[0165] In this embodiment, the observation information at the current time t may be obtained, where the observation information may include certain information and uncertain information.

[0166] Step S206: Determine whether there is a deterministic optimal mode based on experience. If yes, jump to step S212; if no, jump to step S207.

[0167] In this embodiment, it can be determined empirically whether there is a deterministic optimal mode at the current time t. For example, when the battery adopts a charging mode of constant current first and constant voltage later, it can be determined that the DC converter executes a certain mode in most ranges according to the SOC of the battery and the battery voltage of the battery. For example, when the SOC of the battery is very small, the constant current mode or the MPPT mode must be executed.

[0168] Step S207: Select a mode according to the greedy strategy.

[0169] In this embodiment, a greedy strategy selection probability λ (0<λ<1, and the value of λ is generally relatively small) can be set to select the operating mode with the maximum reward with a probability of 1-λ, and randomly select the operating mode with a probability of λ.

[0170] Step S208: Update the memory library.

[0171] In this embodiment, the above-mentioned state and mode information stored in the memory of the intelligent agent can be updated according to the state of the optical storage system, the selected operation mode and the reward value corresponding to the operation mode.

[0172] Step S209: extract the minimum batch to update the agent parameters.

[0173] In this embodiment, the minimum batch in the memory bank can be extracted to update the parameters of the agent. Minimum batch means that small batches of data are uniformly extracted during the reinforcement learning process to update the parameters of the agent.

[0174] Step S210: Determine whether t≥T|| satisfies the end condition? If yes, jump to step S211; if no, jump to step S214.

[0175] In this embodiment, it is determined whether the current time t of the iterative loop training in the current iterative loop training reaches the upper limit T of the time of the current iterative loop training, or whether the current iterative training meets the end condition. If yes, jump to step S211; if no, jump to step S214.

[0176] Step S211: n≥N? If yes, then end the iterative training; if no, then jump to step S213.

[0177] In this embodiment, it can be determined whether the number n of iterative cycle training reaches the upper limit N of the total number of iterative cycle training. If so, the iterative training is terminated; if not, the process jumps to step S213.

[0178] Step S212: Select and determine the strategy, and jump to step S208.

[0179] In this embodiment, if a deterministic optimal strategy exists, the deterministic optimal strategy may be selected.

[0180] Step S213: n=n+1, and jump to step S203.

[0181] In this implementation, the number of times of the current iterative cycle training may be increased by 1 before entering the next iterative cycle training.

[0182] Step S214: t=t+1, and jump to step S205.

[0183] In this embodiment, the current time t of the iterative loop training in the current iterative loop training is increased by 1, and the process jumps to step S205 to perform the next iterative training.

[0184] In one embodiment, the trained agent can be written into an executable embedded chip, and cooperated with the chip controlling the DC converter to realize the function of the solar controller, so that the DC converter can switch between different operating modes. In the actual switching process, the actual operating data is collected to further update the agent parameters.

[0185] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0186] Furthermore, the present invention also provides a control device for a solar energy controller in a solar energy storage system.

[0187] See attached Figure 6, Figure 6 FIG. 1 is a main structural block diagram of a control device of a solar energy controller in a solar energy storage system according to an embodiment of the present invention. Figure 6 As shown, the solar controller in the embodiment of the present invention may include a DC converter, and the photovoltaic storage system may also include a solar panel, a battery and a load. The DC input side of the DC converter is connected to the solar panel, and the DC output side of the DC converter is connected to the battery and the load respectively, and the battery is also connected to the load. The control device of the solar controller in the photovoltaic storage system may include an agent training module and an operation mode determination module. In this embodiment, the agent training module may be configured to use a reinforcement learning method and use a simulation model of the photovoltaic storage system to perform multiple iterations of training on the agent of the solar controller to obtain a trained agent. The operation mode determination module may be configured to use a trained agent and determine the operation mode of the DC converter at the next moment according to the operation mode of the DC converter in the photovoltaic storage system and the battery state parameters of the battery at the current moment, and control the DC converter to operate according to the determined operation mode at the next moment; wherein the battery state parameters may include the input current and / or battery voltage of the battery.

[0188] In one embodiment, the agent training module may include a simulation model parameter setting unit, a final operation mode determination unit, a reward value calculation unit, and an agent parameter updating unit. In this embodiment, the simulation model parameter setting unit may be configured to randomly set the initial SOC of the battery in the simulation model, and control the light and temperature received by the solar panel in the simulation model to change according to the preset light curve and temperature curve, and control the power of the load in the simulation model to change according to the preset load power curve. The final operation mode determination unit may be configured to use the agent and determine all the operation modes that the DC converter can adopt at the next moment according to the operation mode of the DC converter in the simulation model at the current moment, and adopt a greedy strategy to select an operation mode from all the operation modes as the final operation mode adopted by the DC converter at the next moment. The reward value calculation unit may be configured to calculate the reward value when the DC converter operates in the final operation mode according to the battery state parameters of the battery in the simulation model at the current moment; wherein the battery state parameters may include the input current and / or battery voltage of the battery. The agent parameter updating unit can be configured to calculate the value of the agent according to the reward value, calculate the gradient value of the agent's parameters according to the value, adopt the gradient descent method and update the agent's parameters according to the gradient value to complete the current iterative training.

[0189] In one embodiment, the operation mode may include a constant voltage mode for controlling the output voltage of the DC converter to be constant, a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power; the reward value calculation unit may be further configured to perform the following steps: if the operation mode of the DC converter in the simulation model at the current moment is a constant current mode or an MPPT mode switched from a constant current mode, then the first reward value when the DC converter is operated in the final operation mode may be calculated according to the method shown in the following formula (1):

[0190] R 1 =-(I 1 -I ref1 ) 2 (1)

[0191] Among them, R 1 Represents the first reward value, I 1 Represents the first input current of the battery, I ref1 represents a preset first input current reference value, and I ref1 is a fixed constant;

[0192] If the operation mode of the DC converter in the simulation model at the current moment is the constant voltage mode or the MPPT mode switched from the constant voltage mode, the first reward value when the DC converter operates in the final operation mode can be calculated according to the method shown in the following formula (2):

[0193] R 1 =-(VV ref ) 2 (2)

[0194] Among them, R 1 represents the first reward value, V represents the battery voltage of the battery, V ref Indicates the preset battery voltage reference value, and V ref is a fixed constant.

[0195] In one embodiment, the operation mode may include a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power; the reward value calculation unit may be further configured to perform the following steps:

[0196] The second reward value when the DC converter operates in the final operation mode is calculated according to the method shown in the following formula (3):

[0197] R 2 =-(I 2 -I ref2 ) 2 (3)

[0198] Among them, R2 Represents the second reward value, I 2 Represents the second input current of the battery, I ref2 Indicates a preset second input current reference value.

[0199] In one embodiment, the control device for a solar energy controller in a solar energy storage system may further include a second input current reference value first acquisition module and a second input current reference value second acquisition module. The second input current reference value first acquisition module may be configured to acquire the second input current reference value I according to the method shown in the following formula (4): ref2 :

[0200] I ref2 = k × e -bt (4)

[0201] Wherein, k and b are preset coefficients, and t is the charging time of the battery;

[0202] The second input current reference value first acquisition module may be configured to acquire the second input current reference value I according to the following steps: ref2 : According to the minimum and maximum values ​​of the SOC of the battery, the SOC interval between the minimum and maximum values ​​is divided into a plurality of SOC sub-intervals; a different second input current reference value I is set for each SOC sub-interval ref2 Wherein, according to the order of the values ​​corresponding to the starting endpoints of each SOC sub-interval from small to large, the second input current reference value I corresponding to each SOC sub-interval ref2 Gradually increase.

[0203] The control device for the solar energy controller in the solar energy storage system is used to execute Figure 1 and Figure 5 The control method embodiment of the solar energy controller for the photovoltaic storage system shown in the figure is similar in technical principles, technical problems solved and technical effects produced. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working process and related instructions of the control device of the solar energy controller for the photovoltaic storage system can refer to the contents described in the control method embodiment of the solar energy controller for the photovoltaic storage system, which will not be repeated here.

[0204] It is understood by those skilled in the art that the present invention implements all or part of the processes in the method of the above embodiment, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0205] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0206] Those skilled in the art will appreciate that the modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present invention.

[0207] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A control method for a solar energy controller in a solar energy storage system, characterized in that: The solar controller includes a DC converter, and the solar energy storage system also includes a solar panel, a battery and a load. The DC input side of the DC converter is connected to the solar panel, and the DC output side of the DC converter is connected to the battery and the load respectively, and the battery is also connected to the load. The method comprises: Adopting a reinforcement learning method and using the simulation model of the solar energy storage system to perform multiple iterations of training on the intelligent agent of the solar energy controller to obtain a trained intelligent agent; Using a trained intelligent agent and according to the operation mode of the DC converter in the photovoltaic storage system and the battery state parameters of the storage battery at the current moment, determining the operation mode of the DC converter at the next moment and controlling the DC converter to operate according to the determined operation mode at the next moment; wherein the battery state parameters include the input current and / or battery voltage of the storage battery; In "using a reinforcement learning method and using the simulation model of the solar energy storage system to perform multiple iterations of training on the intelligent agent of the solar energy controller", the specific steps of each iteration of training include: Step S1: randomly setting the initial SOC of the battery in the simulation model, controlling the light and temperature received by the solar panel in the simulation model to change according to the preset light curve and temperature curve respectively, and controlling the power of the load in the simulation model to change according to the preset load power curve; Step S2: using the intelligent agent and according to the operation mode of the DC converter in the simulation model at the current moment, determining all the operation modes that the DC converter can adopt at the next moment, and using a greedy strategy to select one operation mode from all the operation modes as the final operation mode adopted by the DC converter at the next moment; Step S3: calculating the reward value when the DC converter operates in the final operation mode according to the battery state parameters of the battery in the simulation model at the current moment; wherein the battery state parameters include the input current and / or battery voltage of the battery; Step S4: calculating the value of the agent according to the reward value, calculating the gradient value of the parameters of the agent according to the value, using the gradient descent method and updating the parameters of the agent according to the gradient value to complete the current iterative training; The operation modes include a constant voltage mode for controlling the output voltage of the DC converter to be constant, a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power; The step of "calculating the reward value when the DC converter operates in the final operation mode according to the battery state parameters of the battery in the simulation model at the current moment" specifically includes: If the operation mode of the DC converter in the simulation model at the current moment is the constant current mode or the MPPT mode switched from the constant current mode, the first reward value of the DC converter when operating in the final operation mode is calculated according to the method shown in the following formula (1): R1=-(I1-I ref1 ) 2 (1) Wherein, R1 represents the first reward value, I1 represents the first input current of the battery, and I ref1 represents a preset first input current reference value, and I ref1 is a fixed constant; If the operation mode of the DC converter in the simulation model at the current moment is the constant voltage mode or the MPPT mode switched from the constant voltage mode, the first reward value when the DC converter operates in the final operation mode is calculated according to the method shown in the following formula (2): R1=-(VV ref ) 2 (2) Wherein, R1 represents the first reward value, V represents the battery voltage of the battery, and V ref Indicates the preset battery voltage reference value, and V ref is a fixed constant; or, The operation mode includes a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power; The step of "calculating the reward value when the DC converter operates in the final operation mode according to the battery state parameters of the battery in the simulation model at the current moment" specifically includes calculating the second reward value when the DC converter operates in the final operation mode according to the method shown in the following formula (3): R2=-(I2-I ref2 ) 2 (3) Wherein, R2 represents the second reward value, I2 represents the second input current of the battery, and I ref2 Indicates a preset second input current reference value.

2. The control method for a solar energy controller in a solar energy storage system according to claim 1, characterized in that: The method further includes obtaining the second input current reference value I according to the method shown in the following formula (4): ref2 : I ref2 =k×e -bt (4) Wherein, k and b are preset coefficients, and t is the charging time of the battery; or, The method further comprises obtaining the second input current reference value I by the following steps ref2 : According to a minimum value and a maximum value of the SOC of the battery, dividing the SOC interval between the minimum value and the maximum value into a plurality of SOC sub-intervals; A different second input current reference value I is set for each SOC sub-interval ref2 ; According to the order of the values ​​corresponding to the starting endpoints of each SOC sub-interval from small to large, the second input current reference value I corresponding to each SOC sub-interval is ref2 Gradually increase.

3. A control device for a solar energy controller in a solar energy storage system, characterized in that: The solar controller includes a DC converter, and the solar energy storage system also includes a solar panel, a battery and a load. The DC input side of the DC converter is connected to the solar panel, and the DC output side of the DC converter is connected to the battery and the load respectively, and the battery is also connected to the load. The device comprises: An intelligent agent training module, which is configured to adopt a reinforcement learning method and use the simulation model of the solar energy storage system to perform multiple iterations of training on the intelligent agent of the solar energy controller to obtain a trained intelligent agent; An operation mode determination module is configured to use a trained intelligent agent and determine the operation mode of the DC converter at the next moment according to the operation mode of the DC converter in the photovoltaic storage system and the battery state parameters of the storage battery at the current moment, and control the DC converter to operate according to the determined operation mode at the next moment; wherein the battery state parameters include the input current and / or battery voltage of the storage battery; The agent training module includes: A simulation model parameter setting unit, configured to randomly set an initial SOC of a battery in the simulation model, and to control the light and temperature received by a solar panel in the simulation model to change according to a preset light curve and temperature curve, respectively, and to control the power of a load in the simulation model to change according to a preset load power curve; a final operation mode determination unit, configured to use the intelligent agent and determine all operation modes that the DC converter can adopt at the next moment according to the operation mode of the DC converter in the simulation model at the current moment, and select one operation mode from all the operation modes by adopting a greedy strategy as the final operation mode adopted by the DC converter at the next moment; a reward value calculation unit, configured to calculate a reward value when the DC converter operates in the final operation mode according to a battery state parameter of the battery in the simulation model at a current moment; wherein the battery state parameter includes an input current and / or a battery voltage of the battery; An agent parameter updating unit, configured to calculate the value of the agent according to the reward value, calculate the gradient value of the agent's parameter according to the value, adopt a gradient descent method and update the agent's parameter according to the gradient value to complete the current iterative training; The operation modes include a constant voltage mode for controlling the output voltage of the DC converter to be constant, a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power; The reward value calculation unit is further configured to perform the following steps: If the operation mode of the DC converter in the simulation model at the current moment is the constant current mode or the MPPT mode switched from the constant current mode, the first reward value of the DC converter when operating in the final operation mode is calculated according to the method shown in the following formula (1): R1=-(I1-I ref1 ) 2 (1) Wherein, R1 represents the first reward value, I1 represents the first input current of the battery, and I ref1 represents a preset first input current reference value, and I ref1 is a fixed constant; If the operation mode of the DC converter in the simulation model at the current moment is the constant voltage mode or the MPPT mode switched from the constant voltage mode, the first reward value when the DC converter operates in the final operation mode is calculated according to the method shown in the following formula (2): R1=-(VV ref ) 2 (2) Wherein, R1 represents the first reward value, V represents the battery voltage of the battery, and V ref Indicates the preset battery voltage reference value, and V ref is a fixed constant; The operation mode includes a constant current mode for controlling the output current of the DC converter to be constant, and an MPPT mode for controlling the output power of the DC converter to be maximum power; The reward value calculation unit is further configured to perform the following steps: The second reward value when the DC converter operates in the final operation mode is calculated according to the method shown in the following formula (3): R2=-(I2-I ref2 ) 2 (3) Wherein, R2 represents the second reward value, I2 represents the second input current of the battery, and I ref2 Indicates a preset second input current reference value.

4. The control device for a solar energy controller in a solar energy storage system according to claim 3, characterized in that: The device further includes a first second input current reference value acquisition module and a second input current reference value second acquisition module, wherein the first second input current reference value acquisition module is configured to acquire the second input current reference value I according to the method shown in the following formula (4): ref2 : I ref2 =k×e -bt (4) Wherein, k and b are preset coefficients, and t is the charging time of the battery; The second input current reference value first acquisition module is configured to acquire the second input current reference value I according to the following steps: ref2 : According to a minimum value and a maximum value of the SOC of the battery, dividing the SOC interval between the minimum value and the maximum value into a plurality of SOC sub-intervals; A different second input current reference value I is set for each SOC sub-interval ref2 ; According to the order of the values ​​corresponding to the starting endpoints of each SOC sub-interval from small to large, the second input current reference value I corresponding to each SOC sub-interval is ref2 Gradually increase.

Images