Electric vehicle charging method and device based on multi-agent deep reinforcement learning

Abstract

The invention discloses an electric vehicle charging method and device based on multi-agent deep reinforcement learning, and the method comprises the steps: building a charging control model based on a Markov game framework and a real-time charging scene, and the charging control model comprises a state variable, an action variable, a reward function and a state transfer function; constructing a control strategy network based on the charging control model, and constructing a parameter updating model of the control strategy network; and constructing an evaluation network, and optimizing the control strategy network based on the evaluation network and the parameter updating model to obtain an optimal control strategy. A charging control model is constructed through a Markov game framework, so that multi-agent collaborative optimization of vehicle charging control is constructed, charging resources of each electric vehicle are dynamically allocated, it is ensured that under limited power grid resources, the charging requirements of all vehicles are met to the maximum extent, and the charging efficiency and economic benefits in different scenes are planned as a whole.

Description

Technical Field

[0001] The present invention relates to the field of automobile control technology, and in particular to an electric vehicle charging method and device based on multi-agent deep reinforcement learning. Background Art

[0002] At present, with the increasing popularity of electric vehicles, the load on the power system will increase, which may lead to overload of distribution transformers. This is because when many electric vehicles are charged at the same time, they consume a lot of energy and cause more current to pass through the transformer. This high load may cause the transformer to overheat and shorten its service life. Most of the smart grid technologies and adaptive control algorithms are used in the market to monitor and manage network loads in real time. For example, models such as centralized aggregator collaborative optimization models, fuzzy logic algorithms, and centralized model light models based on temperature limits optimize the load distribution during charging in real time and avoid overuse of a single transformer, and perform intelligent scheduling based on data analysis and prediction models to prevent transformer overload.

[0003] Most of the existing electric vehicle charging optimization methods are based on models to optimize the charging process. However, model-based methods rely on specific charging environments and cannot be applied to different environments. They are difficult to respond to changing conditions, resulting in low vehicle charging efficiency. Summary of the invention

[0004] The present invention provides an electric vehicle charging method and device based on multi-agent deep reinforcement learning to comprehensively improve the charging efficiency and economic benefits of electric vehicles.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides an electric vehicle charging method based on multi-agent deep reinforcement learning, comprising:

[0006] Building a charging control model based on a Markov game framework and a real-time charging scenario, wherein the charging control model includes state variables, action variables, reward functions, and state transfer functions;

[0007] Building a control strategy network based on the charging control model, and building a parameter update model of the control strategy network;

[0008] An evaluation network is constructed, and the control strategy network is optimized based on the evaluation network and the parameter update model to obtain an optimal control strategy.

[0009] The present invention constructs a charging control model through a Markov game framework to construct a multi-agent collaborative optimization of vehicle charging control, dynamically allocate charging resources (such as charging power and time) for each electric vehicle, and ensure that the charging needs of all vehicles are maximized under limited power grid resources. At the same time, by real-time monitoring of power grid load and transformer status, intelligent scheduling of charging strategies can avoid transformer overload caused by excessive centralized charging and extend the service life of transformers, thereby making charging strategies more flexible and able to adapt to more diverse and complex scenarios, so as to timely adjust charging strategies based on different scenarios and improve charging efficiency.

[0010] Furthermore, the charging control model is constructed based on the Markov game framework and the real-time charging scenario, including:

[0011] Construct state variables based on the hottest point temperature of the transformer winding, the total load of the transformer, the predicted value of the electricity price, the battery state of charge, the charging stop time and the expected state of charge;

[0012] The state variables include:

[0013]

[0014] in, Indicates the hottest point temperature of the transformer winding at time t; represents the total load on the transformer at time t-1; EP t represents the predicted electricity price at time t; SOC t Indicates the state of charge of the electric vehicle battery; t dep represents the charging stop time of the jth electric vehicle; Indicates the expected state of charge at the time of charging stop.

[0015] Furthermore, the charging control model is constructed based on the Markov game framework and the real-time charging scenario, including:

[0016] The action variables are constructed based on the charging and discharging power of the electric vehicle. The action variables are specifically:

[0017] a t =P t

[0018] in, is the set of all agent action spaces, P t Indicates the charge and discharge power.

[0019] Furthermore, the charging control model is constructed based on the Markov game framework and the real-time charging scenario, including:

[0020] The total charging price is calculated based on the electricity price and the charging and discharging power, and the transformer charging loss is calculated based on the transformer life and the loss of the transformer-powered electric vehicle charging power;

[0021] A reward function is constructed based on the total charging price and the transformer charging loss.

[0022] Furthermore, the charging control model is constructed based on the Markov game framework and the real-time charging scenario, including:

[0023] constructing a first transfer function based on the battery power, and constructing a second transfer function based on a random conditional probability model;

[0024] A state transfer function is constructed based on the first transfer function and the second transfer function.

[0025] In a second aspect, the present invention provides an electric vehicle charging device based on multi-agent deep reinforcement learning, including: a charging control model construction module, a strategy network construction module and a strategy optimization module;

[0026] The charging control model building module is used to build a charging control model based on the Markov game framework and the real-time charging scenario, and the charging control model includes state variables, action variables, reward functions and state transfer functions;

[0027] The strategy network construction model is used to construct a control strategy network based on the charging control model, and to construct a parameter update model of the control strategy network;

[0028] The strategy optimization module is used to construct an evaluation network, and optimize the control strategy network based on the evaluation network and the parameter update model to obtain the optimal control strategy.

[0029] Furthermore, the charging control model construction module is used to:

[0030] Construct state variables based on the hottest point temperature of the transformer winding, the total load of the transformer, the predicted value of the electricity price, the battery state of charge, the charging stop time and the expected state of charge;

[0031] The state variables include:

[0032]

[0033] in, Indicates the hottest point temperature of the transformer winding at time t; Indicates the total load on the transformer at time t-6; EP t represents the predicted electricity price at time t; SOC t Indicates the state of charge of the electric vehicle battery; t deprepresents the charging stop time of the jth electric vehicle; Indicates the expected state of charge at the time of charging stop.

[0034] Furthermore, the charging control model construction module is used to:

[0035] The action variables are constructed based on the charging and discharging power of the electric vehicle. The action variables are specifically:

[0036] a t =P t

[0037] in, is the set of all agent action spaces, P t Indicates the charge and discharge power.

[0038] Furthermore, the charging control model construction module is used to:

[0039] The total charging price is calculated based on the electricity price and the charging and discharging power, and the transformer charging loss is calculated based on the transformer life and the loss of the transformer-powered electric vehicle charging power;

[0040] A reward function is constructed based on the total charging price and the transformer charging loss.

[0041] Furthermore, the charging control model construction module is used to:

[0042] constructing a first transfer function based on the battery power, and constructing a second transfer function based on a random conditional probability model;

[0043] A state transfer function is constructed based on the first transfer function and the second transfer function. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic flow chart of an electric vehicle charging method based on multi-agent deep reinforcement learning provided in an embodiment of the present invention.

[0045] Figure 2 A schematic diagram of the structure of an electric vehicle charging device based on multi-agent deep reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0047] The terms "first" and "second" and the like in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0048] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] First, some terms in this application are explained to facilitate understanding by those skilled in the art.

[0050] (1) Markov Game is a framework that extends the Markov decision process (MDP) and is used to deal with multi-agent decision-making problems.

[0051] Example 1

[0052] See also Figure 1 , Figure 1 The present invention provides a method for restoring fuel cell performance, including steps 101 to 104, as follows:

[0053] Step 101: constructing a charging control model based on a Markov game framework and a real-time charging scenario, wherein the charging control model includes state variables, action variables, a reward function, and a state transfer function;

[0054] In this embodiment, the total number of cars in the real-time charging scenario, the battery state of charge of each car, the charging stop time and the expected state of charge, the hottest point temperature of the transformer winding, the total load of the transformer, and the predicted electricity price are obtained, and an intelligent agent is built for each electric vehicle based on the Markov game framework to generate a charging control model containing multiple intelligent agents.

[0055] In this embodiment, a charging control model for multiple vehicles is constructed using a Markov game framework, and the real-time charging control of electric vehicles is constructed into a four-tuple model including state variables, action variables, reward functions, and state transfer functions.

[0056] In this embodiment, the charging control model is a four-tuple in is the set of all agent state spaces is the set of all agent action spaces is the set of all agent reward functions Represents the probability function of executing an action in the current state to transfer to a new state

[0057] In this embodiment, the charging control model is constructed based on the Markov game framework and the real-time charging scenario, including:

[0058] Construct state variables based on the hottest point temperature of the transformer winding, the total load of the transformer, the predicted value of the electricity price, the battery state of charge, the charging stop time and the expected state of charge;

[0059] The state variables include:

[0060]

[0061] in, Indicates the hottest point temperature of the transformer winding at time t; represents the total load on the transformer at time t-1; EP t represents the predicted electricity price at time t; SOC t Indicates the state of charge of the electric vehicle battery; t dep represents the charging stop time of the jth electric vehicle; Indicates the expected state of charge at the time of charging stop.

[0062] In this embodiment, the state of the jth electric car is defined as:

[0063]

[0064] in, Indicates the hottest point temperature of the transformer winding at time t; represents the total load on the transformer at time t-1; EP t represents the predicted value of electricity price at time t; represents the state of charge of the j-th electric vehicle battery; represents the time when the jth electric car leaves home; Indicates the expected state of charge when leaving home.

[0065] In this embodiment, the state variables of each intelligent agent are constructed through multiple aspects such as the hottest point temperature of the transformer winding, the total load of the transformer, the predicted value of the electricity price, the battery state of charge, the charging stop time and the expected state of charge, so that the charging strategy is more adapted to each intelligent agent.

[0066] In this embodiment, the hottest point temperature of the transformer winding is used to monitor the health of the transformer in real time, thereby preventing damage caused by overheating, improving the safety and reliability of the transformer, and extending its service life.

[0067] In this embodiment, the total load of the transformer is used to detect the current load of the power grid to prevent overload of the power grid caused by a large number of vehicles charging at the same time, so as to optimize resource allocation, ensure stable operation of the power grid, and avoid equipment failure caused by overload.

[0068] In this embodiment, the best charging period is selected according to the fluctuation of electricity price, so as to reduce the charging cost and improve the economic benefit.

[0069] In this embodiment, based on charge information such as the battery state of charge, the time when charging stops, and the expected state of charge, it is ensured that the vehicle reaches the expected power level at the time of charging stops, thereby meeting the user's travel needs.

[0070] In this embodiment, by monitoring the hottest point temperature and total load of the transformer winding, transformer overload can be prevented in advance, its service life can be extended, and the reliability of the power grid can be improved; at the same time, by using the electricity price forecast value, the intelligent body can charge during periods of lower electricity prices, thereby reducing charging costs and improving economic benefits. Based on information such as the battery charge state, charging stop time and expected charge state, it is ensured that the vehicle reaches the required charge state at the user's expected charging stop time.

[0071] In this embodiment, state variables are constructed based on the hottest point temperature of the transformer winding, the total load of the transformer, the predicted electricity price, the battery state of charge, the charging stop time and the expected state of charge, which can optimize the charging strategy in multiple dimensions, reduce charging costs, and improve efficiency and grid stability.

[0072] In this embodiment, the charging control model is constructed based on the Markov game framework and the real-time charging scenario, including:

[0073] The action variables are constructed based on the charging and discharging power of the electric vehicle. The action variables are specifically:

[0074] a t =P t (3)

[0075] in, is the set of all agent action spaces, P tIndicates the charge and discharge power.

[0076] In this embodiment, the action variable is a variable to be optimized, that is, the charging / discharging behavior of the electric vehicle is the action variable, and the charging / discharging power of the electric vehicle is used to represent the charging / discharging behavior of the electric vehicle. Specifically:

[0077]

[0078] in, Represents the charging and discharging power of the jth electric vehicle.

[0079] In this embodiment, the charging and discharging power is used as the action variable to provide a flexible energy management strategy for electric vehicles. The intelligent agent can flexibly adjust the charging and discharging power according to environmental changes and user needs to improve the adaptability and robustness of the system. At the same time, by dynamically adjusting the charging and discharging power according to the grid load, electricity price changes and battery status, the optimal energy management strategy can be achieved to avoid overload, optimize charging costs and improve system efficiency.

[0080] In this embodiment, the charging control model is constructed based on the Markov game framework and the real-time charging scenario, including:

[0081] The total charging price is calculated based on the electricity price and the charging and discharging power, and the transformer charging loss is calculated based on the transformer life and the loss of the transformer-powered electric vehicle charging power;

[0082] A reward function is constructed based on the total charging price and the transformer charging loss.

[0083] In this embodiment, the total charging price is calculated based on the electricity price and the charging and discharging power, specifically:

[0084]

[0085] Among them, EP t represents the electricity price at time t; represents the charging and discharging power of electric vehicle j at time t; Δt represents the time gap.

[0086] In this embodiment, the transformer charging loss is calculated based on the transformer life and the loss of the charging power of the electric vehicle powered by the transformer, specifically:

[0087]

[0088] in, It represents the impact of electric vehicle charging on transformer life loss, that is, transformer charging loss; W LOL Indicates the life of the transformer, provided by the transformer equipment supplier; It represents the loss of charging power of electric vehicles powered by transformers, which can be obtained by statistics or calculation.

[0089] In this embodiment, a reward function is constructed based on the total charging price and the transformer charging loss, specifically:

[0090]

[0091] Where N represents the total number of electric vehicles.

[0092] In this embodiment, by calculating the total charging price, the intelligent agent can select the most economical charging plan during the charging process, reduce charging costs, and improve economic benefits. At the same time, by considering the transformer charging loss, the intelligent agent can avoid transformer overload and loss caused by overcharging, extend the service life of the transformer, and improve equipment reliability, so as to comprehensively consider the charging cost and equipment loss, achieve comprehensive optimization of multiple objectives, and improve the overall efficiency and stability of the system.

[0093] In this embodiment, the charging control model is constructed based on the Markov game framework and the real-time charging scenario, including:

[0094] constructing a first transfer function based on the battery power, and constructing a second transfer function based on a random conditional probability model;

[0095] A state transfer function is constructed based on the first transfer function and the second transfer function.

[0096] In this embodiment, the first transfer function is a deterministic transfer portion, and the second transfer function state is a transfer function with an unknown transfer probability.

[0097] In this embodiment, the state transfer function is:

[0098]

[0099] Among them, a t It will only affect the deterministic transfer part of the state variable, that is, the first transfer function. t Only affects the battery level E t+1 ,E t With E t+1 The relationship is E t+1 =E t +a t In the random part, the transition function with unknown transition probability follows the random conditional probability The random conditional probability is affected by the randomness of electricity prices and the commuting behavior of EV owners.

[0100] In this embodiment, by constructing an intelligent agent of a charging control model including state variables, action variables, reward functions and state transfer functions, intelligent scheduling of electric vehicle charging is achieved. By constructing a reward function for the total charging price and transformer charging loss, the economic efficiency of charging and the impact of equipment life are coordinated, and action variables are constructed based on the charging and discharging power, the charging and discharging behavior of each electric vehicle can be dynamically adjusted to ensure efficient use of multiple electric vehicles sharing grid resources and avoid waste of resources.

[0101] Step 102: constructing a control strategy network based on the charging control model, and constructing a parameter update model of the control strategy network;

[0102] In this embodiment, the control strategy network is an actor network, which is used to output the charging control strategy, that is, the charging and discharging power of the electric vehicle.

[0103] In this embodiment, consider an N-agent environment with a parameterized continuous strategy, the control strategy network of the jth agent is With parameter θ j , then the control strategy network for:

[0104]

[0105] in, represents the state variable of the Nth agent at time t; K p is the input of the pth hidden layer; p=1,...,P-1 represents the weight parameters and bias of the pth hidden layer of the first microgrid controller; represents the output of the Pth hidden layer; as well as Represents the output decision of the first microgrid controller The weight parameters and bias terms.

[0106] In this embodiment, a parameter updating model of the control strategy network is constructed based on an action value function.

[0107] In this embodiment, the parameter update model is specifically:

[0108]

[0109] in Represents the memory buffer, is a centralized action-value function with parameter φ j，1 .

[0110] In this embodiment, based on the action value function Eliminate the negative impact of overestimation, thereby enhancing the decision-making performance of the algorithm.

[0111] In this embodiment, the normal distribution and variance are used to update the action value function. as well as Updates follow:

[0112]

[0113]

[0114]

[0115] Among them, ε represents noise, specifically:

[0116]

[0117] in represents normal distribution, σ represents variance, and c represents noise margin.

[0118] Step 103: constructing an evaluation network, and optimizing the control strategy network based on the evaluation network and the parameter update model to obtain an optimal control strategy;

[0119] In this embodiment, an evaluation network is constructed based on the critic network, the control strategy network is evaluated based on the critic network, and the parameters in the action value function and the parameters in the control strategy network are updated.

[0120] In this embodiment, the control strategy network is optimized based on the evaluation network and the parameter update model, specifically:

[0121]

[0122] Among them, φ′ represents the parameters of the target critic network; θ′ represents the parameters of the target actor network; τ represents a coefficient much smaller than 1.

[0123] In this embodiment, based on formula (18), the parameter φ′ of the target critic network is j,i＝1,2 and θ′ j Perform a soft update operation.

[0124] In this embodiment, the control strategy network is used to output decisions, and the evaluation network is used to guide the control strategy network to approach the optimal strategy, so as to obtain the optimal control strategy to control the charging and discharging power of each electric vehicle in the real-time scenario, thereby comprehensively improving the overall charging and discharging efficiency and economic benefits.

[0125] The present invention constructs a charging control model through a Markov game framework to construct a multi-agent collaborative optimization of vehicle charging control, dynamically allocate charging resources (such as charging power and time) for each electric vehicle, and ensure that the charging needs of all vehicles are maximized under limited power grid resources. At the same time, by real-time monitoring of power grid load and transformer status, intelligent scheduling of charging strategies can avoid transformer overload caused by excessive centralized charging and extend the service life of transformers, thereby making charging strategies more flexible and able to adapt to more diverse and complex scenarios, so as to timely adjust charging strategies based on different scenarios and improve charging efficiency.

[0126] Please refer to Figure 2 , Figure 2 A schematic diagram of a structure of an electric vehicle charging device based on multi-agent deep reinforcement learning provided by an embodiment of the present invention, comprising: a charging control model construction module 201, a strategy network construction module 202 and a strategy optimization module 203;

[0127] The charging control model construction module 201 is used to construct a charging control model based on a Markov game framework and a real-time charging scenario, wherein the charging control model includes state variables, action variables, a reward function, and a state transfer function;

[0128] The strategy network building module 202 is used to build a control strategy network based on the charging control model, and to build a parameter update model of the control strategy network;

[0129] The strategy optimization module 203 is used to construct an evaluation network, and optimize the control strategy network based on the evaluation network and the parameter update model to obtain an optimal control strategy.

[0130] In this embodiment, the charging control model construction module is used to:

[0131] Construct state variables based on the hottest point temperature of the transformer winding, the total load of the transformer, the predicted value of the electricity price, the battery state of charge, the charging stop time and the expected state of charge;

[0132] The state variables include:

[0133]

[0134] in, Indicates the hottest point temperature of the transformer winding at time t; Indicates the total load on the transformer at time t-6; EP t represents the predicted electricity price at time t; SOC t Indicates the state of charge of the electric vehicle battery; t dep represents the charging stop time of the jth electric vehicle; Indicates the expected state of charge at the time of charging stop.

[0135] In this embodiment, the charging control model construction module is used to:

[0136] The action variables are constructed based on the charging and discharging power of the electric vehicle. The action variables are specifically:

[0137] a t =P t

[0138] in, is the set of all agent action spaces, P t Indicates the charge and discharge power.

[0139] In this embodiment, the charging control model construction module is used to:

[0140] The total charging price is calculated based on the electricity price and the charging and discharging power, and the transformer charging loss is calculated based on the transformer life and the loss of the transformer-powered electric vehicle charging power;

[0141] A reward function is constructed based on the total charging price and the transformer charging loss.

[0142] In this embodiment, the charging control model construction module is used to:

[0143] constructing a first transfer function based on the battery power, and constructing a second transfer function based on a random conditional probability model;

[0144] A state transfer function is constructed based on the first transfer function and the second transfer function.

[0145] In an embodiment of the present invention, a terminal device is also provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the above-mentioned electric vehicle charging method based on multi-agent deep reinforcement learning is implemented.

[0146] In an embodiment of the present invention, a computer-readable storage medium is also provided, and the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned electric vehicle charging method based on multi-agent deep reinforcement learning.

[0147] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0148] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor, a memory, and a display. Those skilled in the art will appreciate that the above components are merely examples of the terminal device and do not constitute a limitation on the terminal device. The terminal device may include more or fewer components than the components, or may combine certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0149] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0150] The memory can be used to store computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0151] Among them, if the module based on the processing of the multi-device access platform is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0152] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An electric vehicle charging method based on multi-agent deep reinforcement learning, characterized in that: include: Building a charging control model based on a Markov game framework and a real-time charging scenario, wherein the charging control model includes state variables, action variables, reward functions, and state transfer functions; Building a control strategy network based on the charging control model, and building a parameter update model of the control strategy network; An evaluation network is constructed, and the control strategy network is optimized based on the evaluation network and the parameter update model to obtain an optimal control strategy.

2. The electric vehicle charging method based on multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: The charging control model is constructed based on the Markov game framework and the real-time charging scenario, including: Construct state variables based on the hottest point temperature of the transformer winding, the total load of the transformer, the predicted value of the electricity price, the battery state of charge, the charging stop time and the expected state of charge; The state variables include: in, Indicates the hottest point temperature of the transformer winding at time t; represents the total load on the transformer at time t-1; EP t represents the predicted electricity price at time t; SOC t Indicates the state of charge of the electric vehicle battery; t dep represents the charging stop time of the jth electric vehicle; Indicates the expected state of charge at the time of charging stop.

3. The electric vehicle charging method based on multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: The charging control model is constructed based on the Markov game framework and the real-time charging scenario, including: The action variables are constructed based on the charging and discharging power of the electric vehicle. The action variables are specifically: a t =P t in, is the set of all agent action spaces, P t Indicates the charge and discharge power.

4. The electric vehicle charging method based on multi-agent deep reinforcement learning as claimed in claim 3, characterized in that: The charging control model is constructed based on the Markov game framework and the real-time charging scenario, including: The total charging price is calculated based on the electricity price and the charging and discharging power, and the transformer charging loss is calculated based on the transformer life and the loss of the transformer-powered electric vehicle charging power; A reward function is constructed based on the total charging price and the transformer charging loss.

5. The electric vehicle charging method based on multi-agent deep reinforcement learning as claimed in claim 4, characterized in that: The charging control model is constructed based on the Markov game framework and the real-time charging scenario, including: constructing a first transfer function based on the battery power, and constructing a second transfer function based on a random conditional probability model; A state transfer function is constructed based on the first transfer function and the second transfer function.

6. An electric vehicle charging device based on multi-agent deep reinforcement learning, characterized in that: include: Charging control model building module, strategy network building module and strategy optimization module; The charging control model building module is used to build a charging control model based on the Markov game framework and the real-time charging scenario, and the charging control model includes state variables, action variables, reward functions and state transfer functions; The strategy network construction model is used to construct a control strategy network based on the charging control model, and to construct a parameter update model of the control strategy network; The strategy optimization module is used to construct an evaluation network, and optimize the control strategy network based on the evaluation network and the parameter update model to obtain the optimal control strategy.

7. The electric vehicle charging device based on multi-agent deep reinforcement learning as claimed in claim 6, characterized in that: The charging control model building module is used to: Construct state variables based on the hottest point temperature of the transformer winding, the total load of the transformer, the predicted value of the electricity price, the battery state of charge, the charging stop time and the expected state of charge; The state variables include: in, Indicates the hottest point temperature of the transformer winding at time t; Indicates the total load on the transformer at time t-6; EP t represents the predicted electricity price at time t; SOC t Indicates the state of charge of the electric vehicle battery; t dep represents the charging stop time of the jth electric vehicle; Indicates the expected state of charge at the time of charging stop.

8. The electric vehicle charging device based on multi-agent deep reinforcement learning as claimed in claim 6, characterized in that: The charging control model building module is used to: The action variables are constructed based on the charging and discharging power of the electric vehicle. The action variables are specifically: a t =P t in, is the set of all agent action spaces, P t Indicates the charge and discharge power.

9. The electric vehicle charging device based on multi-agent deep reinforcement learning as claimed in claim 8, characterized in that: The charging control model building module is used to: The total charging price is calculated based on the electricity price and the charging and discharging power, and the transformer charging loss is calculated based on the transformer life and the loss of the transformer-powered electric vehicle charging power; A reward function is constructed based on the total charging price and the transformer charging loss.

10. The electric vehicle charging device based on multi-agent deep reinforcement learning as claimed in claim 9, characterized in that: The charging control model building module is used to: constructing a first transfer function based on the battery power, and constructing a second transfer function based on a random conditional probability model; A state transfer function is constructed based on the first transfer function and the second transfer function.

Images