Electric vehicle operation simulation intelligent scheduling method and system based on reinforcement learning

Abstract

The application provides a kind of electric vehicle operation simulation intelligent scheduling method and system based on reinforcement learning, it is related to intelligent transportation technical field, the steps of the method include obtaining the road network data of target area, community data, charging station data and vehicle data, each data is unified to the same plane projection coordinate system and establishes spatial correlation;Based on the unified data, an electric vehicle operation simulation model is constructed, a reinforcement learning strategy network is set in the simulation model, and a vehicle energy consumption model and a vehicle state machine model are introduced;The reinforcement learning strategy network is trained in the simulation model to obtain the optimal scheduling strategy network;In each simulation step, the real-time state of the vehicle is obtained, the vehicle state machine model is called, and the remaining power updated in real time is combined with the vehicle energy consumption model to determine whether the vehicle needs to be charged;The current state of the vehicle to be charged is input into the optimal scheduling strategy network, the target charging station is matched for the vehicle, and the charging scheduling instruction is output and executed.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to an intelligent scheduling method and system for electric vehicle operation simulation based on reinforcement learning. Background Technology

[0002] With the rapid growth of global electric vehicle ownership and the widespread use of commercial vehicles such as ride-hailing and taxis, efficient management of electric vehicle fleets, optimization of charging strategies, and improvement of operational efficiency have become significant challenges for the industry. During actual operation, electric vehicles experience dynamic changes in multiple dimensions, including vehicle status, charging station status, and traffic conditions. Traditional fixed-rule or single-factor threshold-based charging decision-making methods are ill-suited to this complex and ever-changing operating environment, easily leading to problems such as vehicles stopping mid-journey due to insufficient battery power, uneven utilization of charging station resources, and high operating costs.

[0003] Currently, most existing electric vehicle scheduling solutions employ simple charging decision rules based on battery level thresholds. Charging is triggered when the battery level falls below a preset value, often neglecting key factors such as charging station distance, electricity price fluctuations, and the vehicle's current task status. This results in a one-sided scheduling strategy lacking comprehensive awareness of the operating environment. When faced with real-time changes in traffic flow, battery status, charging station availability, and electricity prices, existing technologies lack an integrated solution that can deeply fuse reinforcement learning policy networks with high-precision traffic simulation models, completing policy training and scheduling execution within the same framework. This makes it impossible to automatically learn the optimal charging scheduling strategy through repeated trial and error and reward feedback in a simulation environment, thus hindering the provision of adaptive and efficient intelligent scheduling support for operational decisions. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for intelligent scheduling of electric vehicle operation simulation based on reinforcement learning, so as to solve the problem mentioned in the background art that the existing technology is difficult to provide truly adaptive and efficient intelligent scheduling support for operation decision-making.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning, comprising the following steps: acquiring road network data, community data, charging station data, and vehicle data of a target area, unifying the data to the same planar projected coordinate system and establishing spatial correlation; constructing an electric vehicle operation simulation model based on the unified data, setting a reinforcement learning strategy network in the electric vehicle operation simulation model, and introducing a vehicle energy consumption model and a vehicle state machine model; training the reinforcement learning strategy network in the electric vehicle operation simulation model to obtain an optimal scheduling strategy network; acquiring the real-time vehicle status in each simulation step, calling the vehicle state machine model, and combining the remaining battery power updated in real time by the vehicle energy consumption model to determine whether the vehicle needs charging; when it is determined that the vehicle needs charging, inputting the current status of the vehicle to be charged into the optimal scheduling strategy network, matching the vehicle with a target charging station, and outputting and executing a charging scheduling command.

[0006] Optionally, the step of unifying the various data into the same planar projection coordinate system and establishing spatial association specifically includes: converting the geographic coordinates of the community data and the charging station data into planar projection coordinates; performing coordinate correction using the offset parameters obtained from the road network data; and establishing spatial association between the corrected community data, the charging station data, and the nearest road in the road network data by calculating the Euclidean distance.

[0007] Optionally, the vehicle state machine model defines the vehicle states as follows: offline state, initial trip state, customer search state, customer transport state, heading to charging state, and charging state. The vehicle state switching strategy includes: when the vehicle is offline and appears in the simulation environment or reaches the preset shift start time, switching to the initial trip state; when the vehicle is in the initial trip state and the current location reaches the road network road in the initial trip, switching to the customer search state; when the vehicle is in the customer search state and successfully finds a customer based on the probability model simulation, switching to the customer transport state; when the vehicle is in the customer search state or the customer transport state and the real-time battery status is lower than the preset charging threshold, switching to the heading to charging state; when the vehicle is in the heading to charging state and the current location reaches the road network road where the target charging station is located, switching to the charging state; when the vehicle is in the charging state and the real-time battery status reaches the preset charging completion threshold, switching to the customer search state; when the vehicle is in the customer transport state and the current trip is the last shift of the day or the current time exceeds the shift end time, switching to the offline state.

[0008] Optionally, in the step of determining whether the vehicle needs to be charged, charging anxiety value and time anxiety value are also introduced as the basis for judgment: when the weighted sum of the vehicle's remaining battery power, charging anxiety value, and time anxiety value exceeds a preset threshold, a state switch to charging is triggered even if the battery power is not lower than the charging threshold; the charging anxiety value is calculated by weighting the vehicle's battery power, distance, and time factors; the time anxiety value is calculated by weighting the charging time and electricity price factors.

[0009] Optionally, the training process of the reinforcement learning policy network specifically includes: defining a state space, an action space, and a reward function; the state space includes the vehicle's real-time battery level, location, current state, charging station occupancy rate, and electricity price information; the action space includes charging decision actions and charging station selection actions; the reward function includes operational efficiency rewards, charging efficiency rewards, cost control rewards, and service quality rewards, and the weighted sum of these rewards is used as the comprehensive reward value; wherein, the operational efficiency reward is calculated based on the number of passenger trips or mileage completed, the charging efficiency reward is calculated based on the rationality of charging timing and charging station utilization efficiency, the cost control reward is calculated based on a negative weighted average of operating costs and charging costs, and the service quality reward is calculated based on a negative weighted average of passenger waiting time and transportation time; the reinforcement learning policy network is iteratively trained with the goal of maximizing the comprehensive reward value to obtain the optimal scheduling policy network.

[0010] Optionally, the target charging station is matched with the vehicle to be charged based on a multi-attribute utility function, taking into account distance, electricity price, and availability factors. The standardized utility values ​​of the distance, electricity price, and availability factors are weighted and summed to obtain the comprehensive utility value of the candidate charging station for the current vehicle. The charging station with the highest comprehensive utility value is selected as the target charging station.

[0011] Optionally, the vehicle's remaining battery status is updated in real time using the vehicle energy consumption model. Specifically, this includes: obtaining the vehicle's rolling resistance coefficient, vehicle mass, and gravitational acceleration; multiplying the rolling resistance coefficient, vehicle mass, and gravitational acceleration to obtain rolling resistance; obtaining air density, vehicle air resistance coefficient, vehicle frontal area, and vehicle speed relative to the air; multiplying the air density, air resistance coefficient, and frontal area, and then multiplying by half the square of the vehicle's speed relative to the air to obtain air resistance; obtaining the vehicle mass and vehicle acceleration; multiplying the vehicle mass and acceleration to obtain acceleration resistance; summing the rolling resistance, air resistance, and acceleration resistance; multiplying by the vehicle speed; and dividing by the vehicle's propulsion efficiency to obtain the vehicle's propulsion power; and calculating the vehicle's battery consumption within each simulation step based on the propulsion power and the simulation step length, and updating the vehicle's remaining battery status.

[0012] On the other hand, the present invention also provides an intelligent scheduling system for electric vehicle operation simulation based on reinforcement learning, comprising: an acquisition module for acquiring road network data, community data, charging station data, and vehicle data of a target area, unifying the data into the same planar projection coordinate system and establishing spatial correlation; a simulation module for constructing an electric vehicle operation simulation model based on the unified data, setting a reinforcement learning strategy network in the electric vehicle operation simulation model, and introducing a vehicle energy consumption model and a vehicle state machine model; a training module for training the reinforcement learning strategy network in the electric vehicle operation simulation model to obtain an optimal scheduling strategy network; a judgment module for acquiring the real-time status of the vehicle at each simulation step, calling the vehicle state machine model, and combining the remaining power updated in real time by the vehicle energy consumption model to determine whether the vehicle needs to be charged; and a scheduling module for inputting the current status of the vehicle to be charged into the optimal scheduling strategy network when it is determined that the vehicle needs to be charged, matching the vehicle with a target charging station, and outputting and executing a charging scheduling command.

[0013] On the other hand, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning.

[0014] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This application eliminates spatial inconsistencies in multi-source heterogeneous data by unifying road network, community, charging station, and vehicle data into the same planar projected coordinate system and establishing spatial relationships. By directly setting and training a reinforcement learning policy network in the electric vehicle operation simulation model, the simulation environment and decision-making algorithm are integrated. This allows the policy network to autonomously explore the optimal combination of charging timing and charging station selection during continuous interactive simulation. The policy network simultaneously perceives multi-dimensional information such as vehicle remaining battery power, real-time location, current operating status, charging station occupancy rate, and electricity price, and continuously optimizes its decision-making behavior based on the long-term reward maximization objective, forming an adaptive, scenario-aware intelligent scheduling capability. In each simulation step, by calling the vehicle state machine model and combining it with the remaining power updated in real time by the energy consumption model, the vehicle to be charged and its current state can be accurately identified. After inputting the current state of the vehicle to be charged into the trained optimal scheduling strategy network, the matching target charging station and corresponding scheduling instructions can be output in real time. This realizes closed-loop control from state perception to instruction execution, ensuring that charging decisions meet the requirements of operational continuity while taking into account charging costs and resource utilization efficiency. This significantly improves the scientific nature and real-time performance of charging decisions, optimizes charging resource allocation, and reduces operating costs. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method steps of the present invention.

[0017] Figure 2 This is a schematic diagram of the system structure of the present invention.

[0018] In the diagram: 10 - Acquisition module, 20 - Simulation module, 30 - Training module, 40 - Judgment module, 50 - Scheduling module. Detailed Implementation

[0019] The present invention will now be clearly and completely described in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0022] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0023] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Please refer to Figure 1This invention discloses an intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning. The method includes the following steps: acquiring road network data, community data, charging station data, and vehicle data for a target area; unifying all data into the same planar projected coordinate system and establishing spatial relationships; constructing an electric vehicle operation simulation model based on the unified data; setting a reinforcement learning strategy network in the electric vehicle operation simulation model and introducing a vehicle energy consumption model and a vehicle state machine model; training the reinforcement learning strategy network in the electric vehicle operation simulation model to obtain an optimal scheduling strategy network; acquiring the real-time vehicle status at each simulation step; calling the vehicle state machine model and combining it with the real-time updated remaining battery power from the vehicle energy consumption model to determine whether the vehicle needs charging; when it is determined that the vehicle needs charging, inputting the current status of the vehicle to be charged into the optimal scheduling strategy network, matching the vehicle with a target charging station, and outputting and executing a charging scheduling command.

[0026] Specifically, by combining the SUMO simulation system with electric vehicle operation decision-making, the following data is first obtained for the target area: road network data (SUMO format), community data (CSV format, including community ID, name, latitude and longitude, and population information), charging station data (CSV format, including charging station location and electricity pricing strategy with 24-hour electricity price information), and vehicle data (CSV format, including vehicle attributes and shift schedule).

[0027] The coordinates of communities and charging stations in the geographic coordinate system are converted to a planar projected coordinate system, and the system offset between their internal coordinates and real-world coordinates is analyzed and corrected by reading the SUMO road network file. By calculating the Euclidean distance, the nearest road network edge (i.e., the road in the road network) is matched for each community and each charging station to establish spatial association.

[0028] An electric vehicle operation simulation model is constructed based on unified data. This model is based on SUMO traffic simulation software and realizes real-time interactive control through the TraCI interface. A reinforcement learning policy network to be trained is set in the simulation model, and a vehicle energy consumption model (used to calculate the power consumption during vehicle operation) and a vehicle state machine model (used to manage the transition of vehicle operating states) are introduced.

[0029] The reinforcement learning policy network is trained in the simulation model. By defining the state space, action space and reward function, the PPO algorithm is used to iteratively update the network parameters until the reward converges, thus obtaining the optimal scheduling policy network.

[0030] In each simulation step, the real-time status (position, speed, remaining battery power, current roadside location, etc.) of all vehicles is obtained through the TraCI interface. The vehicle state machine model is invoked, and the remaining battery power is updated in real time based on the vehicle energy consumption model to determine whether the vehicle needs charging. When the determination result is that charging is needed, the current status of the vehicle to be charged (including battery power, location, current status type, information on nearby charging stations, etc.) is input into the optimal scheduling strategy network. This network outputs the matching target charging station identifier and generates a charging scheduling instruction. The system executes the instruction to send the vehicle to the charging station.

[0031] This application eliminates spatial inconsistencies in multi-source heterogeneous data by unifying road network, community, charging station, and vehicle data into the same planar projected coordinate system and establishing spatial relationships. By directly setting and training a reinforcement learning policy network in the electric vehicle operation simulation model, the simulation environment and decision-making algorithm are integrated. This allows the policy network to autonomously explore the optimal combination of charging timing and charging station selection during continuous interactive simulation. The policy network simultaneously perceives multi-dimensional information such as vehicle remaining battery power, real-time location, current operating status, charging station occupancy rate, and electricity price, and continuously optimizes its decision-making behavior based on the long-term reward maximization objective, forming an adaptive, scenario-aware intelligent scheduling capability. In each simulation step, by calling the vehicle state machine model and combining it with the remaining power updated in real time by the energy consumption model, the vehicle to be charged and its current state can be accurately identified. After inputting the current state of the vehicle to be charged into the trained optimal scheduling strategy network, the matching target charging station and corresponding scheduling instructions can be output in real time. This realizes closed-loop control from state perception to instruction execution, ensuring that charging decisions meet the requirements of operational continuity while taking into account charging costs and resource utilization efficiency. This significantly improves the scientific nature and real-time performance of charging decisions, optimizes charging resource allocation, and reduces operating costs.

[0032] In some embodiments, the step of unifying the various data into the same planar projection coordinate system and establishing spatial association specifically includes: converting the geographic coordinates of the community data and the charging station data into planar projection coordinates; performing coordinate correction using offset parameters obtained from the road network data; and establishing spatial association between the corrected community data, the charging station data, and the nearest road in the road network data by calculating the Euclidean distance.

[0033] Specifically, the pyproj library is used to convert the WGS84 geographic coordinates in the community data and charging station data to UTM50N planar projected coordinates. The SUMO road network file is read, and the systematic offset between the internal coordinates of the road network and the real-world coordinates is parsed from it. This offset is then applied to the planar coordinates converted in the previous step to complete the coordinate correction.

[0034] Finally, spatial relationships are established by calculating Euclidean distances: for each corrected community coordinate and each charging station coordinate, the Euclidean distance between it and the starting coordinates of all road network edges is calculated in batches using the broadcast feature of the NumPy library. The road network edge with the smallest distance is selected as the matching edge of the community or charging station, and the matching relationship is stored in the static data manager, thus laying the foundation for subsequent path planning and location query.

[0035] This application converts the geographic coordinates of communities and charging stations into planar projected coordinates and uses offset parameters from the SUMO road network file for systematic error correction. This ensures that spatial data from different sources can be accurately analyzed and distance calculated in a unified coordinate system, eliminating coordinate drift between the real world and the simulation environment. Furthermore, by employing a nearest-path matching algorithm based on Euclidean distance, a precise association between each community and each charging station and the road network topology is established. This allows subsequent route planning and location queries to directly utilize the topological relationships of road network edges, avoiding invalid paths or inaccessible charging stations caused by a disconnect between spatial location and the traffic network.

[0036] In some embodiments, the vehicle state machine model defines the vehicle states as follows: offline state, initial trip state, customer search state, customer transport state, heading to charging state, and charging state. The vehicle state switching strategy includes: when the vehicle is offline and appears in the simulation environment or reaches a preset shift start time, switching to the initial trip state; when the vehicle is in the initial trip state and its current location reaches a road in the initial trip, switching to the customer search state; when the vehicle is in the customer search state and successfully finds a customer based on a probability model simulation, switching to the customer transport state; when the vehicle is in the customer search state or the customer transport state and its real-time battery level is lower than a preset charging threshold, switching to the heading to charging state; when the vehicle is in the heading to charging state and its current location reaches a road in the target charging station, switching to the charging state; when the vehicle is in the charging state and its real-time battery level reaches a preset charging completion threshold, switching to the customer search state; when the vehicle is in the customer transport state and its current trip is the last trip of the day or the current time exceeds the shift end time, switching to the offline state.

[0037] Specifically, the vehicle state machine model defines six vehicle states: offline state, initial trip state, customer search state, customer transport state, heading to charging state, and charging state, as shown in Table 1 below: Table 1: Vehicle Status and Meaning.

[0038]

[0039] The state transition strategy is executed according to the following rules: When the vehicle is offline and appears in the simulation environment or reaches the preset start time of the shift, it switches to the initial trip state; when the vehicle is in the initial trip state and its current location reaches the target road network specified in the initial trip, it switches to the customer search state; when the vehicle is in the customer search state and successfully finds a customer based on the probability model simulation, it switches to the customer transport state; when the vehicle is in the customer search state or customer transport state and its real-time battery level is lower than a preset charging threshold (e.g., 25%), it switches to the charging destination state; when the vehicle is in the charging destination state and its current location reaches the road network where the target charging station is located, it switches to the charging state; when the vehicle is in the charging state and its real-time battery level reaches a preset charging completion threshold (e.g., 95%), it switches to the customer search state; when the vehicle is in the customer transport state and the current trip is the last shift of the day or the current time exceeds the shift end time, it switches to the offline state. The system calls the state management function SLO_complex in each simulation step, iterates through all vehicles, and automatically executes state transitions according to the above rules, as shown in Table 2 below: Table 2: State Transition Table.

[0040]

[0041] This application defines six refined operational states: offline, initial trip, customer search, customer transport, heading to charging, and charging. It also sets clear transition conditions to comprehensively depict the behavioral logic of electric vehicles throughout the entire process from deployment, order acceptance, passenger transport, charging, to decommissioning. This structured state management enables the system to accurately track the real-time operational stage of each vehicle, avoiding decision-making confusion caused by ambiguous states. In particular, embedding charging trigger conditions into the two key operational states of customer search and customer transport ensures that vehicles can promptly switch to the heading to charging state based on their battery status, whether they are empty or on a mission, thus completing charging without interrupting core transportation services. Simultaneously, the state machine supports automatic deployment and decommissioning control at the start and end of shifts, achieving seamless integration of multiple shifts. This state machine model provides a clear environmental awareness foundation for reinforcement learning policy networks, enabling agents to make differentiated optimal decisions based on different states, improving the targeting and effectiveness of scheduling strategies.

[0042] In some embodiments, the step of determining whether a vehicle needs to be charged also incorporates charging anxiety value and time anxiety value as criteria: when the weighted sum of the vehicle's remaining battery power, charging anxiety value, and time anxiety value exceeds a preset threshold, a state switch to charging is triggered even if the battery power is not lower than the charging threshold; the charging anxiety value is calculated by weighting the vehicle's battery power, distance, and time factors; the time anxiety value is calculated by weighting the charging time and electricity price factors.

[0043] Specifically, in determining whether a vehicle needs charging, this application introduces charging anxiety and time anxiety values ​​as supplementary criteria to avoid premature or delayed charging issues that might arise from relying on a single battery level threshold. The calculation formula is as follows: ; in, This is a comprehensive evaluation value based on multiple factors. The current battery status is represented by the battery factor as (1-SOC), which is the normalized deficit of the current remaining battery power. The lower the battery level, the higher this value is. Charging anxiety level; Time anxiety value; , , For weighting coefficients. When Exceeding the threshold At that time, a charging decision is triggered: .

[0044] ; in, For charging anxiety level, Due to battery level factors, the lower the vehicle's battery level, the higher the anxiety level. The distance factor is represented by the ratio of the Euclidean distance of the vehicle to the nearest charging station to the preset maximum distance. The greater the distance, the higher the anxiety level. The time factor is represented as the ratio of the vehicle's actual working time to its expected working time; the longer the working time, the higher the anxiety level.

[0045] ; in, For time anxiety value, Due to charging time factor; To account for electricity price factors, the current electricity price is normalized.

[0046] The system calculates a multi-factor comprehensive evaluation value for each vehicle in real time at each simulation step and compares it with a threshold, thereby making a charging triggering decision that is more in line with actual operational needs.

[0047] This application introduces charging anxiety as a supplementary criterion in addition to a single battery level threshold judgment, effectively overcoming two typical problems caused by fixed battery level thresholds in traditional solutions: premature charging leading to wasted operating time and late charging causing breakdowns due to lack of power en route. Charging anxiety, by comprehensively quantifying battery level, distance, and time factors, more comprehensively reflects the vehicle's current actual charging urgency. By using charging and time anxiety values, the system is forced to proactively trigger a state switch to charging, thereby taking replenishment measures in advance before risks occur, achieving a decision-making upgrade from passive response to proactive prevention. This multi-factor weighted judgment mechanism makes charging trigger decisions more flexible and intelligent, adapting to the differentiated needs of different vehicles, different operating times, and different geographical locations. It significantly reduces the risk of operational interruptions due to untimely charging, while avoiding unnecessary frequent charging and optimizing overall operational efficiency.

[0048] In some embodiments, the training process of the reinforcement learning policy network specifically includes: defining a state space, an action space, and a reward function; the state space includes the vehicle's real-time battery level, location, current state, charging station occupancy rate, and electricity price information; the action space includes charging decision actions and charging station selection actions; the reward function includes operational efficiency rewards, charging efficiency rewards, cost control rewards, and service quality rewards, and the weighted sum of these rewards is used as the comprehensive reward value; wherein, the operational efficiency reward is calculated based on the number of passenger trips or mileage completed, the charging efficiency reward is calculated based on the rationality of charging timing and charging station utilization efficiency, the cost control reward is calculated based on a negative weighted average of operating costs and charging costs, and the service quality reward is calculated based on a negative weighted average of passenger waiting time and transportation time; the reinforcement learning policy network is iteratively trained with the goal of maximizing the comprehensive reward value to obtain the optimal scheduling policy network.

[0049] Specifically, the training process of a reinforcement learning policy network involves defining the state space, action space, and reward function.

[0050] The state space contains multi-dimensional information: the real-time battery level of each electric vehicle, its current location coordinates or the roadside markers, its current state type, the current occupancy rate of each charging station, and the current electricity price of each charging station.

[0051] The action space includes charging decision actions (go to charge or not go) and charging station selection actions (choose one from multiple candidate charging stations).

[0052] The reward function consists of a weighted sum of four parts to form a comprehensive reward value: operational efficiency reward, calculated based on the number of passenger trips or mileage completed within each simulation step, with higher rewards for more trips completed; charging efficiency reward, calculated by weighting the rationality of charging timing (lower penalty for charging when battery is too low, higher penalty for charging when battery is sufficient) and charging station utilization efficiency (higher reward for selecting an idle charging station); cost control reward, calculated by negatively weighting operating costs (empty mileage, time cost) and charging costs (electricity price × charging amount), with lower rewards for higher costs; and service quality reward, calculated by negatively weighting passenger waiting time and transportation time, with lower rewards for longer waiting or transportation times.

[0053] The system aims to maximize the overall reward value and uses the PPO algorithm in the RayRLlib framework to iteratively train the reinforcement learning policy network in the SUMO simulation model. Each training cycle contains multiple simulation steps. The agent outputs actions according to the current policy network. After the environment executes the actions, it returns the next state and the overall reward value. The interaction trajectory is stored in the experience replay buffer. The PPO algorithm samples data from the buffer to update the policy network parameters. The above process is repeated until the overall reward value converges, and the optimal scheduling policy network is obtained.

[0054] This application enables the agent to fully perceive the dynamic operating environment by explicitly defining a multi-dimensional state space including vehicle battery level, location, status, charging station occupancy rate, and electricity price. It also endows the agent with flexible scheduling and control capabilities by defining an action space including charging decisions and charging station selection. Furthermore, it guides the agent to pursue economic benefits while simultaneously considering service quality and resource utilization balance through a comprehensive reward function weighted by operational efficiency, charging efficiency, cost control, and service quality. Specifically, operational efficiency rewards encourage completing more transportation tasks, charging efficiency rewards guide the selection of reasonable charging times and low-load charging stations, cost control rewards suppress empty runs and high-price charging, and service quality rewards constrain passenger waiting and transportation time. Using the PPO algorithm to iteratively train the policy network under the guidance of this reward function, the agent can automatically learn the optimal charging scheduling strategy in complex environments, rather than relying on manually preset rules. This data-driven training method allows the scheduling strategy to adaptively evolve with environmental changes, enabling it to be migrated to different cities or operating scenarios without reprogramming, exhibiting strong generalization ability and continuous optimization potential.

[0055] In some embodiments, the target charging station is matched with the vehicle to be charged based on a multi-attribute utility function, taking into account distance, electricity price, and availability factors. The standardized utility values ​​of the distance, electricity price, and availability factors are weighted and summed to obtain the comprehensive utility value of the candidate charging station for the current vehicle. The charging station with the highest comprehensive utility value is selected as the target charging station.

[0056] Specifically, the charging station selection decision adopts a multi-attribute utility function: ; in, For vehicles Choose a charging station The overall utility value; These are the weighting coefficients for electricity price, distance, and availability, with default values ​​of 10.0, 3.0, and 2.0, respectively. This is a utility function used to convert raw attribute values ​​into standardized utility values; For charging stations For vehicles The charging price; For vehicles to charging station Spatial distance; For charging stations For vehicles Availability metrics.

[0057] Electricity price utility function A linear standardization method is used to convert charging prices into utility values ​​between 0 and 1, ensuring that low-priced charging stations receive higher utility scores.

[0058] ; in, This is the highest charging price in the system; The lowest charging price in the system; For charging stations For vehicles The charging price.

[0059] Distance utility function By employing an exponential decay approach, the law of diminishing marginal utility is realized as distance increases. That is, when the distance is 0, the utility value is 1, and as the distance increases, the utility value gradually approaches 0.

[0060] ; in, This is the distance attenuation parameter.

[0061] Availability utility function It directly reflects the load status of the charging station, encouraging the selection of charging stations with lower loads and helping to achieve load balancing. The availability utility value is 1 when the charging station is completely idle and 0 when it is fully loaded.

[0062] ; in, For charging stations The current number of items occupied; For charging stations The total number of charging stations.

[0063] This application, by simultaneously considering distance, electricity price, and availability factors, and standardizing each factor before weighted summation, can recommend the most cost-effective charging station for each vehicle waiting to be charged. The distance factor uses an exponential decay method, reflecting the law of diminishing marginal utility as distance increases, avoiding wasted time due to vehicles taking longer routes to charge. The electricity price factor uses a linear standardization method, encouraging the selection of lower-priced charging stations and directly reducing operating costs. The availability factor is calculated based on the real-time load of charging stations, guiding vehicles to choose idle or low-load charging stations, achieving a balanced allocation of charging resources and preventing some charging stations from being overcrowded while others are idle. Compared to single-factor decision-making, this method effectively balances cost, efficiency, and waiting time, improving overall charging resource utilization and reducing the average charging cost for the fleet.

[0064] In some embodiments, the vehicle's remaining battery status is updated in real time using the vehicle energy consumption model, specifically including: obtaining the vehicle's rolling resistance coefficient, vehicle mass, and gravitational acceleration; multiplying the rolling resistance coefficient, vehicle mass, and gravitational acceleration to obtain rolling resistance; obtaining air density, vehicle air resistance coefficient, vehicle frontal area, and vehicle speed relative to air; multiplying the air density, air resistance coefficient, and frontal area, and then multiplying by half the square of the vehicle speed relative to air to obtain air resistance; obtaining the vehicle mass and vehicle acceleration; multiplying the vehicle mass and acceleration to obtain acceleration resistance; summing the rolling resistance, air resistance, and acceleration resistance, multiplying by the vehicle speed, and dividing by the vehicle's propulsion efficiency to obtain the vehicle's propulsion power; and calculating the vehicle's battery consumption within each simulation step based on the propulsion power and simulation step length, and updating the vehicle's remaining battery status.

[0065] Specifically, this application updates the vehicle's remaining battery status in real time using a vehicle energy consumption model. This model is built upon vehicle dynamics principles, and the calculation formula is as follows: ; in, To increase power, To improve efficiency, For rolling resistance, For air resistance, To increase resistance, For vehicle speed.

[0066] Rolling resistance is ,in The rolling resistance coefficient, For vehicle weight, This is the acceleration due to gravity. Air resistance is... ,in, The air drag coefficient, air density, The vehicle's frontal area. The vehicle's wind speed. The acceleration drag is... , To accelerate the vehicle.

[0067] Based on the propulsion power and simulation step length Δt, the energy consumption ΔE = P_propulsion·Δt within each step is calculated. The energy consumption ΔE is then subtracted from the current remaining energy, and the vehicle's remaining energy status is updated. If the vehicle is charging, the energy is increased in reverse according to a preset charging power until the charging completion threshold is reached. This energy consumption model can accurately simulate energy changes under different driving conditions, providing a reliable data foundation for charging decisions.

[0068] This application, based on classic vehicle dynamics principles, calculates rolling resistance, air resistance, and acceleration resistance separately, and comprehensively considers key parameters such as vehicle mass, speed, acceleration, air density, frontal area, rolling resistance coefficient, air resistance coefficient, and propulsion efficiency. This allows for accurate simulation of instantaneous power demand and energy consumption under different driving conditions. This high-precision energy consumption estimation provides a reliable basis for updating remaining battery capacity, enabling the system to more accurately determine the vehicle's range and charging timing, avoiding misjudgments caused by energy consumption estimation errors.

[0069] Please refer to Figure 2 On the other hand, the present invention also provides an intelligent scheduling system for electric vehicle operation simulation based on reinforcement learning, comprising: an acquisition module 10, used to acquire road network data, community data, charging station data and vehicle data of a target area, unify the various data to the same plane projection coordinate system and establish spatial correlation; a simulation module 20, used to construct an electric vehicle operation simulation model based on the unified data, set a reinforcement learning strategy network in the electric vehicle operation simulation model, and introduce a vehicle energy consumption model and a vehicle state machine model; a training module 30, used to train the reinforcement learning strategy network in the electric vehicle operation simulation model to obtain an optimal scheduling strategy network; a judgment module 40, used to acquire the real-time status of the vehicle in each simulation step, call the vehicle state machine model, and combine the remaining power updated in real time by the vehicle energy consumption model to determine whether the vehicle needs to be charged; and a scheduling module 50, used to input the current status of the vehicle to be charged into the optimal scheduling strategy network when it is determined that the vehicle needs to be charged, match the target charging station for the vehicle, and output and execute the charging scheduling command.

[0070] On the other hand, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning.

[0071] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning.

[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0074] The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention's specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for intelligent scheduling of electric vehicle operation simulation based on reinforcement learning, characterized by the following steps: include: Acquire road network data, community data, charging station data, and vehicle data for the target area, unify all data into the same planar projection coordinate system, and establish spatial relationships. Based on the unified data, an electric vehicle operation simulation model is constructed. A reinforcement learning policy network is set in the electric vehicle operation simulation model, and a vehicle energy consumption model and a vehicle state machine model are introduced. The reinforcement learning policy network is trained in the electric vehicle operation simulation model to obtain the optimal scheduling policy network; In each simulation step, the real-time status of the vehicle is acquired, the vehicle state machine model is called, and the remaining power is updated in real time by the vehicle energy consumption model to determine whether the vehicle needs to be charged. When it is determined that a vehicle needs to be charged, the current state of the vehicle to be charged is input into the optimal scheduling strategy network, a target charging station is matched for the vehicle, and a charging scheduling instruction is output and executed.

2. The intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning according to claim 1, characterized in that, The steps of unifying all data into the same plane projection coordinate system and establishing spatial relationships specifically include: Convert the geographic coordinates of the community data and the charging station data into planar projected coordinates; Coordinate correction is performed using offset parameters obtained from the road network data; A spatial association is established between the community data, the charging station data, and the nearest road in the road network data after Euclidean distance is calculated and corrected.

3. The intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning according to claim 1, characterized in that, The vehicle state machine model defines the vehicle states as follows: offline state, initial trip state, customer search state, customer transport state, heading to charging state, and charging state. The vehicle state switching strategy includes: When the vehicle is offline, and the vehicle appears in the simulation environment or reaches the preset start time of the shift, switch to the initial trip state; When the vehicle is in the initial trip state and the current location reaches the road network road in the initial trip, switch to the customer search state; When the vehicle is in the customer search state and successfully finds a customer based on the probability model simulation, it switches to the customer transport state. When the vehicle is in customer search or customer transport mode and the real-time battery level is lower than the preset charging threshold, switch to charging mode. When the vehicle is on its way to a charging station and its current location reaches the road network where the target charging station is located, it switches to charging mode. When the vehicle is charging and the real-time battery level reaches the preset charging completion threshold, switch to the customer search mode. When a vehicle is in a customer transport status and the current trip is the last trip of the day or the current time has exceeded the end time of the trip, it will switch to offline status.

4. The intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning according to claim 3, characterized in that, In the step of determining whether a vehicle needs to be charged, charging anxiety value and time anxiety value are also introduced as the basis for judgment: when the weighted sum of the vehicle's remaining power, charging anxiety value and time anxiety value exceeds a preset threshold, a state switch to charging is triggered even if the power status is not lower than the charging threshold. The charging anxiety value is calculated by weighting factors such as vehicle battery level, distance, and time. The time anxiety value is calculated by weighting the charging time factor and the electricity price factor.

5. The intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning according to claim 1, characterized in that, The training process of the reinforcement learning policy network specifically includes: Define the state space, action space, and reward function; The state space includes the vehicle's real-time battery level, location, current status, charging station occupancy rate, and electricity price information; The action space includes charging decision actions and charging station selection actions; The reward function includes operational efficiency reward, charging efficiency reward, cost control reward, and service quality reward. The weighted sum of each reward is used as the comprehensive reward value. Among them, the operational efficiency reward is calculated based on the number of passenger trips or mileage completed, the charging efficiency reward is calculated based on the rationality of charging timing and the utilization efficiency of charging stations, the cost control reward is calculated based on the negative weighted sum of operating costs and charging costs, and the service quality reward is calculated based on the negative weighted sum of passenger waiting time and transportation time. The reinforcement learning policy network is iteratively trained with the goal of maximizing the overall reward value to obtain the optimal scheduling policy network.

6. The intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning according to claim 5, characterized in that, The target charging station is matched with the vehicle to be charged based on a multi-attribute utility function, taking into account factors such as distance, electricity price, and availability. The standardized utility values ​​of the distance factor, the electricity price factor, and the availability factor are weighted and summed to obtain the comprehensive utility value of the candidate charging station for the current vehicle. The charging station with the highest comprehensive utility value is selected as the target charging station.

7. The intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning according to claim 6, characterized in that, The vehicle's remaining battery status is updated in real time through the vehicle energy consumption model, specifically including: Obtain the rolling resistance coefficient, vehicle mass, and gravitational acceleration of the vehicle. Multiply the rolling resistance coefficient, vehicle mass, and gravitational acceleration together to obtain the rolling resistance. To obtain air resistance, multiply the air density, drag coefficient, frontal area, and speed relative to the air by the air density, drag coefficient, and frontal area, and then multiply by half the square of the speed relative to the air. Obtain the vehicle's mass and acceleration, and multiply the vehicle's mass and acceleration to obtain the acceleration drag. The vehicle's propulsion power is obtained by summing the rolling resistance, air resistance, and acceleration resistance, multiplying the sum by the vehicle speed, and dividing by the vehicle's propulsion efficiency. Based on the propulsion power and simulation step length, calculate the vehicle's power consumption within each simulation step length and update the vehicle's remaining power status.

8. A smart scheduling system for electric vehicle operation simulation based on reinforcement learning, characterized in that, include: The acquisition module is used to acquire road network data, community data, charging station data and vehicle data of the target area, unify the data into the same plane projection coordinate system and establish spatial relationships. The simulation module is used to build an electric vehicle operation simulation model based on the unified data. The electric vehicle operation simulation model is equipped with a reinforcement learning policy network and incorporates a vehicle energy consumption model and a vehicle state machine model. The training module is used to train the reinforcement learning policy network in the electric vehicle operation simulation model to obtain the optimal scheduling policy network. The judgment module is used to obtain the real-time status of the vehicle in each simulation step, call the vehicle state machine model, and combine the remaining power updated in real time by the vehicle energy consumption model to determine whether the vehicle needs to be charged. The scheduling module is used to input the current status of the vehicle to be charged into the optimal scheduling strategy network when it is determined that the vehicle needs to be charged, match the target charging station for the vehicle, and output and execute the charging scheduling command.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent scheduling method for electric vehicle operation simulation based on reinforcement learning as described in any one of claims 1 to 7.

Images