A distributed road surface micro-energy optimization system and method based on a deep deterministic policy gradient algorithm

Abstract

The application discloses a kind of distributed road surface micro-energy optimization systems based on deep deterministic policy gradient algorithm, it includes hardware deployment module and is laid out micro-energy collection device and distributed energy storage unit at each node, constructs energy transmission topology;State vector generation module carries out standardization processing to environmental state data, is combined into state vector;Model construction module constructs executor network and critic network, obtains charge-discharge action value according to state vector by executor network, obtains future cumulative return prediction value according to state vector and charge-discharge action value by critic network, forms deep deterministic policy gradient model;Model training module updates network parameters using dataset samples, obtains optimal model after iteration to convergence;Online control module obtains charge-discharge control instruction using optimal model according to real-time state vector.The application can realize global optimal energy scheduling under continuous action space, improve energy utilization and load balancing degree.

Description

Technical Field

[0001] This invention relates to the field of road surface micro-energy management and intelligent control technology, specifically to a distributed road surface micro-energy optimization system and method based on a deep deterministic strategy gradient algorithm. Background Technology

[0002] The main problem that current distributed pavement micro-energy optimization methods need to solve is the difficulty in energy scheduling caused by the strong randomness of pavement micro-energy acquisition and the continuous action space. The core objective is to achieve efficient acquisition, storage, and distribution of pavement micro-energy, thereby improving energy utilization and load balancing. In existing technologies, fixed threshold control methods cannot adapt to the randomness of pavement micro-energy acquisition, cannot adjust strategies according to real-time environmental conditions, and have low energy utilization. Simple rule-based scheduling methods lack comprehensive consideration of the global energy state, making it difficult to achieve optimal energy allocation and prone to local energy surplus or shortage. Model predictive control methods require accurate system models, but pavement micro-energy systems are complex, making it difficult to accurately establish models, and the computational load is large, resulting in poor real-time performance. Optimization methods based on genetic algorithms have slow convergence speeds, are prone to getting trapped in local optima, and cannot well handle optimization problems in continuous action spaces. Summary of the Invention

[0003] The purpose of this invention is to provide a distributed road micro-energy optimization system and method based on a deep deterministic strategy gradient algorithm. This invention can achieve global optimal energy scheduling in a continuous action space, thereby improving energy utilization and load balancing.

[0004] To achieve this objective, the present invention presents a distributed road micro-energy optimization system based on a deep deterministic policy gradient algorithm, comprising: The hardware deployment module is used to divide the road surface into nodes at set intervals, and to deploy micro-energy harvesting devices and distributed energy storage units at each node to build an energy transmission topology between nodes. The state vector generation module is used to standardize the environmental state data collected by each node at a preset sampling frequency, and combine the standardized environmental state data into a state vector. The model building module is used to build an actuator network and a commentator network. The actuator network obtains the charging and discharging action value based on the state vector, and the commentator network obtains the predicted value of future cumulative return based on the state vector and the charging and discharging action value, thereby forming a deep deterministic policy gradient model based on a two-layer neural network architecture. The model training module is used to update the parameters of the executor network and the critic network using dataset samples, and obtains the optimal deep deterministic policy gradient model after iteration until convergence. The online control module is used to obtain charging and discharging control commands based on the real-time state vector and the optimal depth deterministic strategy gradient model.

[0005] Preferably, the specific method for dividing the road surface into nodes at predetermined intervals, and deploying micro-energy harvesting devices and distributed energy storage units at each node to construct the energy transmission topology between nodes includes: In the road surface layer, nodes are divided according to a set spacing, and the number of nodes N is based on the total length of the road segment and the set spacing. The ratio is determined, and each node includes a set of micro-energy harvesting devices, a distributed energy storage unit, and an embedded controller. The spacing is calculated based on the traffic flow density per unit area of ​​the road segment and the spacing adjustment coefficient, expressed as: in, This is the spacing adjustment factor. Traffic flow density per unit area of ​​the road segment. The set spacing between micro-energy harvesting devices; The state of charge (SOC) of a distributed energy storage unit is determined by the ratio of the real-time voltage to the maximum energy storage capacity, expressed as: in, The state of charge of the distributed energy storage unit. This refers to the real-time voltage of the distributed energy storage unit. This represents the maximum energy storage capacity of the distributed energy storage unit. Using a distributed mesh structure, the transmission power between any two nodes is determined by the product of the available power of the sending node and the transmission efficiency coefficient between the two nodes, as expressed in the following expression: in, For sending node i Available power, For sending node i With receiving node j The transmission efficiency coefficient between them; This completes the construction of the energy transmission topology between nodes.

[0006] Preferably, the environmental status data includes vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature.

[0007] Preferably, the specific method for standardizing the environmental state data collected by each node at a preset sampling frequency and combining the standardized environmental state data into a state vector includes: The environmental condition data collected at each node according to the preset sampling frequency are standardized to calculate the mean and standard deviation of vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature. The standardization formula is as follows: in, For variables The mean, For variables standard deviation For variables The standardized value; The standardized environmental state data are combined into a state vector in a fixed order. s The state vector is defined as: in, This represents the standardized vehicle crushing frequency. This represents the standardized state of charge of the energy storage unit. This represents the standardized real-time power demand of the surrounding load. This indicates the standardized road ambient temperature.

[0008] Preferably, the specific process of constructing an actuator network and a critic network, whereby the actuator network obtains charging / discharging action values ​​based on the state vector, and the critic network obtains predicted future cumulative returns based on the state vector and the charging / discharging action values, thereby forming a deep deterministic policy gradient model based on a two-layer neural network architecture, includes: Construct a two-layer neural network architecture consisting of an Actor network and a Critic network; The actuator network adopts a deep neural network architecture, and the input layer of the actuator network receives the state vector. s The output layer of the actuator network generates deterministic continuous charge and discharge action values. a The actuator network is represented as ,in Let μ be the set of weight parameters for the Actor network; The critic network is based on a deep neural network design and receives a state vector through an input layer. s and the charge / discharge action values ​​output by the actuator network a The predicted cumulative return is obtained through the output layer. The commentator network is represented as... ,in, Let Q be the set of weight parameters for the Critic network, and let Q represent the predicted value of future cumulative returns. By comparing the predicted future cumulative returns output by the Critic network with the target value, the loss function of the Critic network is defined based on the time-series difference error, and its expression is: in, For loss function, E Let s represent the desired state vector and a represent the charging / discharging action value. r s represents the immediate reward, and s' represents the state vector at the next time step. D This represents the experience replay pool. y Indicates the target value; Target value y The target value is obtained by adding the immediate reward to the discounted predicted cumulative reward for the next time step. This predicted cumulative reward is calculated by the target network based on the state vector s' for the next time step. The target network includes a target Actor network and a target Critic network. y The calculation formula is as follows: in, As a discount factor, For the target Critic network, For the target Actor network, For the weight parameter set of the target Actor network, The set of weight parameters for the target Critic network; The weight parameter set of the Critic network is updated by minimizing the loss function, so that the predicted future cumulative return value output by the Critic network continuously approaches the target value calculated based on the target network. After the weight parameter set of the Critic network is updated, the weight parameter set of the Actor network is updated using the deterministic policy gradient method based on the updated Critic network. The expression is as follows: in, Describe the objective function J Regarding the weight parameter set of the Actor network gradient, Indicates from the experience replay pool D The mathematical expectation of the extracted state vector s. This represents the partial derivative of the predicted future cumulative return Q from the Critic network output with respect to the charge / discharge action value a. This represents the charging and discharging action values ​​output by the Actor network with respect to the weight parameter set. The partial derivatives; By calculating the gradient feedback of the action policy using the chain rule, the policy network is driven to iteratively update in the direction of maximizing the predicted value of future cumulative returns, thereby constructing a deep deterministic policy gradient model based on a two-layer neural network architecture.

[0009] Preferably, the specific process of updating the parameters of the actuator network and the critic network using dataset samples, and iterating until convergence to obtain the optimal deep deterministic policy gradient model includes: Historical operational data is collected from each node, including the current state, current action, immediate reward, and next state, and a quadruple is formed and stored in the experience replay pool. During the training loop, a batch of dataset samples is randomly sampled from the experience replay pool. Based on these dataset samples, the target value of the Critic network is calculated using the following formula: in, y For the target value, r For instant rewards, As a discount factor, Represents the target Critic network. Represents the target Actor network. This represents the state vector at time t+1. This indicates that the target Actor network is in the state vector. The output charging and discharging action values; Based on the mean square error between the predicted future cumulative return value and the target value of the Critic network, the loss function of the Critic network is calculated. The loss function of the Critic network is minimized through the backpropagation algorithm, thereby completing the update of the Critic network parameters. Using the updated Critic network, the policy gradient of the Actor network is calculated using the chain rule, and the parameters of the Actor network are updated using the gradient ascent method. After each update of the Critic and Actor networks, a soft update operation is performed on the target network, expressed as: in, For the weight parameter set of the target Critic network, For the weight parameter set of the target Actor network, This is the soft update coefficient. For the updated set of weight parameters of the Critic network, This is the set of weight parameters for the updated Actor network; The parameters of the updated Critic and Actor networks are then fused into the corresponding target network in a certain proportion, and iterative training is performed until convergence to obtain the optimal deep deterministic policy gradient model.

[0010] Preferably, the specific process of obtaining the charge / discharge control command based on the real-time state vector using the optimal depth deterministic strategy gradient model includes: The Actor network in the optimal deep deterministic strategy gradient model is deployed to the embedded controller of the distributed energy storage unit at each node. The embedded controller collects environmental state data in real time at a preset sampling frequency, including vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature, forming a real-time state vector. The Actor network in the optimal deep deterministic strategy gradient model obtains charging and discharging action values ​​based on the real-time state vector, executes the charging and discharging action values ​​as charging and discharging control commands, and makes predictions through the Critic network in the optimal deep deterministic strategy gradient model to obtain the predicted future cumulative return value of the charging and discharging action value.

[0011] A distributed road microenergy optimization method based on a deep deterministic policy gradient algorithm includes the following steps: The road surface is divided into nodes at set intervals, and micro-energy harvesting devices and distributed energy storage units are deployed at each node to construct the energy transmission topology between nodes. The environmental state data collected by each node at a preset sampling frequency is standardized, and the standardized environmental state data is combined into a state vector. An actuator network and a commentator network are constructed. The actuator network obtains the charging and discharging action value based on the state vector, and the commentator network obtains the predicted value of future cumulative return based on the state vector and the charging and discharging action value, thereby forming a deep deterministic policy gradient model based on a two-layer neural network architecture. The parameters of the actuator network and the critic network are updated using dataset samples, and the optimal deep deterministic policy gradient model is obtained after iteration until convergence. Based on the real-time state vector, the charge and discharge control commands are obtained using the optimal depth deterministic strategy gradient model.

[0012] A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0013] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0014] The beneficial effects of this invention are: This invention constructs a deep deterministic policy gradient model based on a two-layer neural network architecture, which solves the problem that fixed thresholds or rule-based control cannot cope with the strong randomness of road surface micro-energy collection. It achieves globally optimal energy scheduling in a continuous action space, improving energy utilization and load balance. This invention reduces computational complexity by using data-driven reinforcement learning, and ensures the stability and fast convergence of the deep deterministic policy gradient model training by using experience replay and target network soft update mechanisms. Based on the real-time state vector of the road, this invention outputs charging and discharging control commands adapted to the current traffic flow, load, temperature, and energy storage unit charge state, which can adaptively respond to road dynamic operating condition fluctuations, realize the collaborative real-time scheduling of distributed energy storage across the entire road segment, and improve the overall comprehensive utilization rate and operational reliability of road surface micro-energy. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 A distributed road micro-energy optimization system based on a deep deterministic policy gradient algorithm, such as Figure 1 As shown, it includes: The hardware deployment module is used to divide the road surface into nodes at set intervals. At each node, micro-energy harvesting devices (micro-energy refers to electrical energy captured from the small deformations or vibrations of the road surface caused by vehicle traffic using the piezoelectric effect or electromagnetic induction principle) and distributed supercapacitor energy storage units are deployed to construct the energy transmission topology between nodes. This design divides the nodes at set intervals and can optimize the distribution of devices according to traffic flow density. The distributed mesh topology supports bidirectional energy flow, which allows energy to be flexibly allocated between nodes. It realizes the orderly layout of the physical layer energy harvesting and storage network, ensuring that energy can be efficiently transmitted and shared between nodes, thereby avoiding the problem of local energy excess or deficiency. In some preferred embodiments, a node refers to a physical location point in the road surface layer divided at a set interval. Each node is equipped with a set of micro-energy harvesting devices (such as piezoelectric devices or electromagnetic devices) and connected to a distributed supercapacitor energy storage unit. The distributed supercapacitor energy storage unit is installed in a roadside protective box. Each node is also equipped with an embedded controller for performing data acquisition, communication and algorithm calculation. Nodes are connected to each other through energy transmission topology and CAN bus communication protocol to form a distributed collaborative control network.

[0017] The state vector generation module is used to standardize the environmental state data collected by each node at a preset sampling frequency (the preset sampling frequency refers to a pre-set, fixed sampling time interval, which can be set from 10Hz to 100Hz), and combine the standardized environmental state data into a state vector. This design ensures that the environmental state data of each dimension are on the same order of magnitude through standardization, thus ensuring data consistency and scalability. The model building module is used to construct an Actor network and a Critic network. The Actor network obtains the charging and discharging action values ​​based on the state vector, and the Critic network obtains the predicted value of the future cumulative reward based on the state vector and the charging and discharging action values. This forms a deep deterministic policy gradient model based on a two-layer neural network architecture. This design uses the Actor network to output deterministic continuous charging and discharging action values ​​and the Critic network to predict the future cumulative reward of the charging and discharging action values. The two are trained together so that the policy can be iteratively optimized towards higher rewards, achieving the globally optimal energy scheduling in the continuous action space. The model training module is used to update the parameters of the executor network and the critic network using dataset samples. After iterating until convergence, the optimal deep deterministic policy gradient model is obtained. This design avoids overfitting caused by temporal correlation by randomly sampling dataset samples, and further ensures the smooth change of network parameters through iteration, thereby improving the generalization ability of the policy. The online control module is used to obtain charging and discharging control commands based on the real-time state vector using the optimal depth deterministic strategy gradient model. This design achieves real-time adaptive regulation of road surface micro-energy by obtaining charging and discharging control commands based on the real-time state vector using the optimal depth deterministic strategy gradient model, which can quickly respond to changes in traffic flow and environment and maintain efficient operation.

[0018] In the above technical solution, the specific method for dividing the road surface into nodes at set intervals, and deploying micro-energy harvesting devices and distributed energy storage units at each node to construct the energy transmission topology between nodes includes: In the road surface layer, nodes are divided according to a set spacing, and the number of nodes N is based on the total length of the road segment and the set spacing. The ratio is determined, and each node includes a set of micro-energy harvesting devices, a distributed energy storage unit, and an embedded controller. The spacing is calculated based on the traffic flow density per unit area of ​​the road segment and the spacing adjustment coefficient, expressed as: in, This is the spacing adjustment factor. Traffic flow density per unit area of ​​the road segment. The set spacing between micro-energy harvesting devices; The state of charge (SOC) of a distributed energy storage unit is determined by the ratio of the real-time voltage to the maximum energy storage capacity, expressed as: in, The state of charge of the distributed energy storage unit. This refers to the real-time voltage of the distributed energy storage unit. This represents the maximum energy storage capacity of the distributed energy storage unit. Using a distributed mesh structure, the transmission power between any two nodes is determined by the product of the available power of the sending node and the transmission efficiency coefficient between the two nodes, as expressed in the following expression: in, For sending node i Available power, For sending node i With receiving node j The transmission efficiency coefficient between them; In a specific implementation of the present invention, the transmitting node refers to a node that has surplus power at the current moment and outputs power to neighboring nodes; the receiving node refers to a node that has a power shortage at the current moment and receives power from neighboring nodes; any node can act as a transmitting node or a receiving node at different times according to its own state of charge and load requirements. This completes the construction of the energy transmission topology between nodes. The above design clarifies the quantitative basis for node division, energy storage status monitoring, and energy transmission, making the hardware deployment operable and scientific, and ensuring the rationality of the distribution of micro energy harvesting devices, the accuracy of energy storage status assessment, and the controllability of energy transmission efficiency.

[0019] In the above technical solution, the environmental state data includes vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature. In the above design, vehicle rolling frequency reflects energy harvesting potential, energy storage unit state of charge reflects energy storage capacity, real-time power demand of surrounding loads reflects power demand, and road ambient temperature affects equipment performance and energy loss, ensuring that the state vector can comprehensively reflect the key factors affecting energy dispatch and improve the accuracy of strategy decision-making.

[0020] Regarding specific methods for collecting environmental condition data, some optimized technical solutions include: vehicle traffic frequency is detected by a piezoelectric sensor array; the piezoelectric sensor outputs a pulse signal, which is converted into a numerical value by a frequency counter, representing the number of vehicle passes per unit time; the state of charge of the energy storage unit is measured using the voltage and current integration method; the voltage and current of the energy storage unit are obtained through a high-precision analog-to-digital converter module, and the percentage of state of charge is calculated based on the ratio of the integral value of the current over time to the rated capacity; the real-time power demand of the surrounding load is monitored in real time by power sensors or current and voltage transformers to obtain the active power consumed by the load, in watts (W); the road ambient temperature is collected by a digital temperature sensor, and the temperature value is output in degrees Celsius (°C).

[0021] In the above technical solution, the specific method for standardizing the environmental state data collected by each node at a preset sampling frequency and combining the standardized environmental state data into a state vector includes: The environmental condition data collected at each node according to the preset sampling frequency are standardized to calculate the mean and standard deviation of vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature. The standardization formula is as follows: in, For variables The mean, For variables standard deviation For variables The standardized value; The standardized environmental state data are combined into a state vector in a fixed order. s The state vector is defined as: in, This represents the standardized vehicle crushing frequency. This represents the standardized state of charge of the energy storage unit. This represents the standardized real-time power demand of the surrounding load. This represents the standardized road ambient temperature. The above design eliminates dimensional differences and data distribution offsets by standardizing each environmental state data and combining them into a fixed-order state vector, thus ensuring the stability and consistency of the data.

[0022] In the above technical solution, the specific process of constructing an actuator network and a commentator network, obtaining charging and discharging action values ​​through the actuator network based on the state vector, and obtaining future cumulative return prediction values ​​through the commentator network based on the state vector and the charging and discharging action values, thereby forming a deep deterministic policy gradient model based on a two-layer neural network architecture, includes: Construct a two-layer neural network architecture consisting of an Actor network and a Critic network; In a preferred embodiment, the deep deterministic policy gradient model adopts a multilayer perceptron structure. The input layer receives the state vectors of the nodes, the hidden layer uses the ReLU activation function, and the output layer uses the tanh activation function to output continuous charging and discharging action values. The Critic network simultaneously receives the state vectors of the nodes and the charging and discharging action values ​​output by the actuator network. The multilayer fully connected structure combined with the ReLU activation function outputs the predicted value of the future cumulative reward of the action. The actuator network adopts a deep neural network architecture, and the input layer of the actuator network receives the state vector. s The output layer of the actuator network generates deterministic continuous charge and discharge action values. a The actuator network is represented as ,in Let μ be the set of weight parameters for the Actor network, and μ represent the actuator network. The optimization objective of the Actor network is to maximize the future cumulative reward predicted by the Critic network. In a specific implementation of this invention, to prevent the charging and discharging control commands output by the Actor network from exceeding the limits of the physical devices, an activation function limiting module is provided at the end of the output layer of the Actor network. Specifically, for the charging and discharging characteristics of the distributed energy storage unit, the activation function limiting module is configured to map and limit the unbounded continuous value output by the network to a preset maximum charging power and maximum discharging power, ensuring that the output charging and discharging control commands are always within the physical safety range allowed by the actual hardware devices. The preset maximum charging power and maximum discharging power are jointly determined based on the peak output capability of the micro-energy harvesting device at the node, the rated charging and discharging current of the distributed energy storage unit, and the peak power requirements of the surrounding loads.

[0023] The critic network is based on a deep neural network design and receives a state vector through an input layer. s and the charge / discharge action values ​​output by the actuator network a The predicted cumulative return is obtained through the output layer. The commentator network is represented as... ,in, is the set of weight parameters for the Critic network (representing all trainable weights and bias parameters within the Critic network), and Q represents the predicted cumulative return value in the future. By comparing the predicted future cumulative returns output by the Critic network with the target value, the loss function of the Critic network is defined based on the time-series difference error, and its expression is: in, The loss function (used to measure the difference between the current predicted future cumulative returns of the Critic network and the target value, guiding the update of network parameters) E Let s represent the expected value (the average error of samples drawn from the experience replay pool), s represent the state vector, and a represent the charging / discharging action value (the currently executed action). r s represents the immediate reward (the reward signal fed back by the environment after the action is performed), and s' represents the state vector at the next moment. D This represents the experience replay pool. y This represents the target value (a pre-calculated baseline value). Target value y The target value is obtained by adding the immediate reward to the discounted predicted cumulative reward for the next time step. This predicted cumulative reward is calculated by the target network based on the state vector s' for the next time step. The target network includes a target Actor network and a target Critic network. y The calculation formula is as follows: in, As a discount factor, For the target Critic network, For the target Actor network, For the weight parameter set of the target Actor network, The set of weight parameters for the target Critic network; The weight parameter set of the Critic network is updated by minimizing the loss function, so that the predicted future cumulative return value output by the Critic network continuously approaches the target value calculated based on the target network. This enables the Critic network to accurately evaluate the value of actions. After the weight parameter set of the Critic network is updated, the weight parameter set of the Actor network is updated using the deterministic policy gradient method based on the updated Critic network. The expression is as follows: in, Describe the objective function J (The policy objective function refers to the sum of future cumulative returns) Regarding the Actor network weight parameter set gradient, Indicates from the experience replay pool D The mathematical expectation of the extracted state vector s. This represents the partial derivative of the predicted future cumulative return Q output by the Critic network with respect to the charge / discharge action value a (reflecting the degree of influence of the action change in the current state on the future cumulative return; the charge / discharge action value a is taken as the action output by the Actor network based on the current state during the calculation). This represents the charging and discharging action values ​​output by the Actor network with respect to the weight parameter set. The partial derivatives; By calculating the gradient feedback of the action policy using the chain rule, the policy network is driven to iteratively update in the direction of maximizing the predicted value of future cumulative rewards, thereby constructing a deep deterministic policy gradient model based on a two-layer neural network architecture. The above design defines the structure, loss function, target value calculation, policy gradient update, and chain rule application of the Actor network and Critic network. The loss function driven by temporal difference error enables the Critic network to accurately evaluate the value of actions, and the deterministic policy gradient method enables the Actor network to update in the direction of high rewards, thus obtaining a deep deterministic policy gradient model.

[0024] In the above technical solution, the specific process of updating the parameters of the actuator network and the critic network using dataset samples, and iterating until convergence to obtain the optimal deep deterministic policy gradient model includes: Historical operational data is collected from each node, including the current state, current action, immediate reward, and next state, and is formed into a quadruple and stored in the experience replay pool. The experience replay pool adopts a first-in-first-out strategy, and the capacity of the experience replay pool is set to a preset value that covers a variety of traffic scenarios. During the training loop, a batch of dataset samples is randomly sampled from the experience replay pool. Based on these dataset samples, the target value of the Critic network is calculated using the following formula: in, y For the target value, r For instant rewards, As a discount factor, Represents the target Critic network. Represents the target Actor network. This represents the state vector at time t+1. This indicates that the target Actor network is in the state vector. The output charging and discharging action values; Based on the mean square error between the predicted future cumulative return value and the target value of the Critic network, the loss function of the Critic network is calculated. The loss function of the Critic network is minimized through the backpropagation algorithm, thereby completing the update of the Critic network parameters. Using the updated Critic network, the policy gradient of the Actor network is calculated using the chain rule, and the parameters of the Actor network are updated using the gradient ascent method. After each update of the Critic and Actor networks, a soft update operation is performed on the target network, expressed as: in, For the weight parameter set of the target Critic network, For the weight parameter set of the target Actor network, This is the soft update coefficient (soft update refers to the slow and gradual synchronization of network parameters). For the updated set of weight parameters of the Critic network, This is the set of weight parameters for the updated Actor network; The parameters of the updated Critic and Actor networks are then fused into the corresponding target network in a certain proportion, and iterative training is performed until convergence to obtain the optimal deep deterministic policy gradient model. The above design ensures data timeliness through an empirical replay pool, breaks data correlation through random sampling, and controls the update speed of the Critic and Actor networks through soft update coefficients, thereby improving the training efficiency and stability of the deep deterministic policy gradient model.

[0025] In a specific implementation of this invention, the convergence determination of the training process is based on the changing trend of the loss function or the policy reward value. Specifically, when the average rate of change of the mean squared error loss value of the Critic network is less than a preset first threshold (the preset first threshold can be 0.001 to 0.01) within m consecutive training rounds (m can be 50 to 100), or the obtained average cumulative reward value is greater than a preset second threshold (the preset second threshold can be 90% to 95% of the historical highest cumulative reward value) and remains stable, it is determined that the deep deterministic policy gradient model has converged, the parameter iteration update is stopped, and the final network parameters are used as the parameters of the optimal deep deterministic policy gradient model and distributed to each node.

[0026] In the above technical solution, the specific process of obtaining the charge / discharge control command based on the real-time state vector and using the optimal depth deterministic strategy gradient model includes: The Actor network in the optimal depth deterministic strategy gradient model is deployed to the embedded controllers of the distributed energy storage units at each node. The embedded controllers collect environmental state data in real time at a preset sampling frequency, including vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature, forming a real-time state vector. The Actor network in the optimal depth deterministic strategy gradient model obtains charging and discharging action values ​​(i.e., specific charging and discharging power commands) based on the real-time state vector. These charging and discharging action values ​​are used as charging and discharging control commands, which are then sent to the distributed energy storage units for execution. The Critic network in the optimal depth deterministic strategy gradient model is used for prediction to obtain the future cumulative return prediction value of the charging and discharging action value, which is used for performance evaluation. The above design achieves real-time and precise control of road surface micro-energy by deploying the Actor network in the optimal depth deterministic strategy gradient model to the embedded controllers at each node, collecting real-time state vectors at a preset sampling frequency and outputting charging and discharging control commands, and using the Critic network to predict the future cumulative return of the charging and discharging action values ​​in the charging and discharging control commands. This allows for rapid adaptation to environmental changes.

[0027] Example 2 A distributed road microenergy optimization method based on deep deterministic policy gradient algorithm, such as Figure 2 As shown, micro-energy harvesting devices and distributed energy storage units are deployed at each node to construct an energy transmission topology; environmental state data is standardized and combined into a state vector; an actuator network and a critic network are constructed. The actuator network obtains charging and discharging action values ​​based on the state vector, and the critic network obtains future cumulative return prediction values ​​based on the state vector and charging and discharging action values, forming a deep deterministic policy gradient model; the network parameters are updated using dataset samples, and the optimal model is obtained after iteration until convergence; charging and discharging control commands are obtained using the optimal model based on the real-time state vector.

[0028] The specific method for distributed road micro-energy optimization based on deep deterministic policy gradient algorithm includes the following steps: The road surface is divided into nodes at set intervals, and micro-energy harvesting devices and distributed energy storage units are deployed at each node to construct the energy transmission topology between nodes. The environmental state data collected by each node at a preset sampling frequency is standardized, and the standardized environmental state data is combined into a state vector. An actuator network and a commentator network are constructed. The actuator network obtains the charging and discharging action value based on the state vector, and the commentator network obtains the predicted value of future cumulative return based on the state vector and the charging and discharging action value, thereby forming a deep deterministic policy gradient model based on a two-layer neural network architecture. The parameters of the actuator network and the critic network are updated using dataset samples, and the optimal deep deterministic policy gradient model is obtained after iteration until convergence. Based on the real-time state vector, the charge and discharge control commands are obtained using the optimal depth deterministic strategy gradient model.

[0029] Example 3 A computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 2.

[0030] Example 4 A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in Embodiment 2.

[0031] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0032] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0033] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0034] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.

[0036] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A distributed road micro-energy optimization system based on a deep deterministic policy gradient algorithm, characterized in that, It includes: The hardware deployment module is used to divide the road surface into nodes at set intervals, and to deploy micro-energy harvesting devices and distributed energy storage units at each node to build an energy transmission topology between nodes. The state vector generation module is used to standardize the environmental state data collected by each node at a preset sampling frequency, and combine the standardized environmental state data into a state vector. The model building module is used to build an actuator network and a commentator network. The actuator network obtains the charging and discharging action value based on the state vector, and the commentator network obtains the predicted value of future cumulative return based on the state vector and the charging and discharging action value, thereby forming a deep deterministic policy gradient model based on a two-layer neural network architecture. The model training module is used to update the parameters of the executor network and the critic network using dataset samples, and obtains the optimal deep deterministic policy gradient model after iteration until convergence. The online control module is used to obtain charging and discharging control commands based on the real-time state vector and the optimal depth deterministic strategy gradient model.

2. The distributed road micro-energy optimization system based on deep deterministic policy gradient algorithm according to claim 1, characterized in that: The specific methods for constructing the energy transmission topology between nodes by dividing the road surface layer into nodes at predetermined intervals and deploying micro-energy harvesting devices and distributed energy storage units at each node include: In the road surface layer, nodes are divided according to a set spacing, and the number of nodes N is based on the total length of the road segment and the set spacing. The ratio is determined, and each node includes a set of micro-energy harvesting devices, a distributed energy storage unit, and an embedded controller. The spacing is calculated based on the traffic flow density per unit area of ​​the road segment and the spacing adjustment coefficient, expressed as: in, This is the spacing adjustment factor. Traffic flow density per unit area of ​​the road segment. The set spacing between micro-energy harvesting devices; The state of charge (SOC) of a distributed energy storage unit is determined by the ratio of the real-time voltage to the maximum energy storage capacity, expressed as: in, The state of charge of the distributed energy storage unit. This refers to the real-time voltage of the distributed energy storage unit. This represents the maximum energy storage capacity of the distributed energy storage unit. Using a distributed mesh structure, the transmission power between any two nodes is determined by the product of the available power of the sending node and the transmission efficiency coefficient between the two nodes, as expressed in the following expression: in, For sending node i Available power, For sending node i With receiving node j The transmission efficiency coefficient between them; This completes the construction of the energy transmission topology between nodes.

3. The distributed road micro-energy optimization system based on deep deterministic policy gradient algorithm according to claim 1, characterized in that: The environmental status data includes vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature.

4. A distributed road micro-energy optimization system based on a deep deterministic policy gradient algorithm according to claim 3, characterized in that: The specific methods for standardizing the environmental state data collected by each node at a preset sampling frequency and combining the standardized environmental state data into a state vector include: The environmental condition data collected at each node according to the preset sampling frequency are standardized to calculate the mean and standard deviation of vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature. The standardization formula is as follows: in, For variables The mean, For variables standard deviation For variables The standardized value; The standardized environmental state data are combined into a state vector in a fixed order. s The state vector is defined as: in, This represents the standardized vehicle crushing frequency. This represents the standardized state of charge of the energy storage unit. This represents the standardized real-time power demand of the surrounding load. This indicates the standardized road ambient temperature.

5. A distributed road micro-energy optimization system based on a deep deterministic policy gradient algorithm according to claim 4, characterized in that: The specific process of constructing an actuator network and a critic network, obtaining charging and discharging action values ​​from the state vector through the actuator network, and obtaining future cumulative return prediction values ​​from the state vector and the charging and discharging action values ​​through the critic network, thereby forming a deep deterministic policy gradient model based on a two-layer neural network architecture, includes: Construct a two-layer neural network architecture consisting of an Actor network and a Critic network; The actuator network adopts a deep neural network architecture, and the input layer of the actuator network receives the state vector. s The output layer of the actuator network generates deterministic continuous charge and discharge action values. a The actuator network is represented as ,in Let μ be the set of weight parameters for the Actor network; The critic network is based on a deep neural network design and receives a state vector through an input layer. s and the charge / discharge action values ​​output by the actuator network a The predicted cumulative return is obtained through the output layer. The commentator network is represented as... ,in, Let Q be the set of weight parameters for the Critic network, and let Q represent the predicted value of future cumulative returns. By comparing the predicted future cumulative returns output by the Critic network with the target value, the loss function of the Critic network is defined based on the time-series difference error, and its expression is: in, For loss function, E Let s represent the desired state vector and a represent the charging / discharging action value. r s represents the immediate reward, and s' represents the state vector at the next time step. D This represents the experience replay pool. y Indicates the target value; Target value y The target value is obtained by adding the immediate reward to the discounted predicted cumulative reward for the next time step. This predicted cumulative reward is calculated by the target network based on the state vector s' for the next time step. The target network includes a target Actor network and a target Critic network. y The calculation formula is as follows: in, As a discount factor, For the target Critic network, For the target Actor network, For the weight parameter set of the target Actor network, The set of weight parameters for the target Critic network; The weight parameter set of the Critic network is updated by minimizing the loss function, so that the predicted future cumulative return value output by the Critic network continuously approaches the target value calculated based on the target network. After the weight parameter set of the Critic network is updated, the weight parameter set of the Actor network is updated using the deterministic policy gradient method based on the updated Critic network. The expression is as follows: in, Describe the objective function J Regarding the weight parameter set of the Actor network gradient, Indicates from the experience replay pool D The mathematical expectation of the extracted state vector s. This represents the partial derivative of the predicted future cumulative return Q from the Critic network output with respect to the charge / discharge action value a. This represents the charging and discharging action values ​​output by the Actor network with respect to the weight parameter set. The partial derivatives; By calculating the gradient feedback of the action policy using the chain rule, the policy network is driven to iteratively update in the direction of maximizing the predicted value of future cumulative returns, thereby constructing a deep deterministic policy gradient model based on a two-layer neural network architecture.

6. A distributed road micro-energy optimization system based on a deep deterministic policy gradient algorithm according to claim 5, characterized in that: The specific process of updating the parameters of the actuator network and the critic network using dataset samples, and iterating until convergence to obtain the optimal deep deterministic policy gradient model includes: Historical operational data is collected from each node, including the current state, current action, immediate reward, and next state, and a quadruple is formed and stored in the experience replay pool. During the training loop, a batch of dataset samples is randomly sampled from the experience replay pool. Based on these dataset samples, the target value of the Critic network is calculated using the following formula: in, y For the target value, r For instant rewards, As a discount factor, Represents the target Critic network. Represents the target Actor network. This represents the state vector at time t+1. This indicates that the target Actor network is in the state vector. The output charging and discharging action values; Based on the mean square error between the predicted future cumulative return value and the target value of the Critic network, the loss function of the Critic network is calculated. The loss function of the Critic network is minimized through the backpropagation algorithm, thereby completing the update of the Critic network parameters. Using the updated Critic network, the policy gradient of the Actor network is calculated using the chain rule, and the parameters of the Actor network are updated using the gradient ascent method. After each update of the Critic and Actor networks, a soft update operation is performed on the target network, expressed as: in, For the weight parameter set of the target Critic network, For the weight parameter set of the target Actor network, This is the soft update coefficient. For the updated set of weight parameters of the Critic network, This is the set of weight parameters for the updated Actor network; The parameters of the updated Critic and Actor networks are then fused into the corresponding target network in a certain proportion, and iterative training is performed until convergence to obtain the optimal deep deterministic policy gradient model.

7. A distributed road micro-energy optimization system based on a deep deterministic policy gradient algorithm according to claim 1, characterized in that: The specific process of obtaining the charge / discharge control command based on the real-time state vector and using the optimal depth deterministic strategy gradient model includes: The Actor network in the optimal deep deterministic strategy gradient model is deployed to the embedded controller of the distributed energy storage unit at each node. The embedded controller collects environmental state data in real time at a preset sampling frequency, including vehicle rolling frequency, energy storage unit state of charge, real-time power demand of surrounding loads, and road ambient temperature, forming a real-time state vector. The Actor network in the optimal deep deterministic strategy gradient model obtains charging and discharging action values ​​based on the real-time state vector, executes the charging and discharging action values ​​as charging and discharging control commands, and makes predictions through the Critic network in the optimal deep deterministic strategy gradient model to obtain the predicted future cumulative return value of the charging and discharging action value.

8. A distributed road micro-energy optimization method based on a deep deterministic policy gradient algorithm, characterized in that, It includes the following steps: The road surface is divided into nodes at set intervals, and micro-energy harvesting devices and distributed energy storage units are deployed at each node to construct the energy transmission topology between nodes. The environmental state data collected by each node at a preset sampling frequency is standardized, and the standardized environmental state data is combined into a state vector. An actuator network and a commentator network are constructed. The actuator network obtains the charging and discharging action value based on the state vector, and the commentator network obtains the predicted value of future cumulative return based on the state vector and the charging and discharging action value, thereby forming a deep deterministic policy gradient model based on a two-layer neural network architecture. The parameters of the actuator network and the critic network are updated using dataset samples, and the optimal deep deterministic policy gradient model is obtained after iteration until convergence. Based on the real-time state vector, the charge and discharge control commands are obtained using the optimal depth deterministic strategy gradient model.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in claim 8.

Images