Electric vehicle scheduling method, device and equipment and storage medium

Abstract

The invention discloses an electric vehicle scheduling method, device and equipment and a storage medium, and relates to the technical field of vehicle scheduling. The method comprises the following steps: obtaining vehicle current state information of an electric vehicle and current state information of a passing battery swap station; the current state information of the vehicle and the current state information of the passing battery swap station are input into a preset reinforcement learning model, a vehicle target action is obtained, and the preset reinforcement learning model is obtained through vehicle sample state information, battery swap station sample state information, vehicle sample joint action and vehicle sample joint reward training; and the vehicles are scheduled based on the vehicle target actions, and the target actions of the vehicles are rapidly and accurately obtained through the trained preset reinforcement learning model, so that rapid scheduling of the electric vehicles is realized through the target actions of the vehicles, and the accuracy and efficiency of vehicle scheduling are improved.

Description

Technical Field

[0001] This application relates to the field of vehicle dispatching technology, and in particular to electric vehicle dispatching methods, devices, equipment and storage media. Background Technology

[0002] The emergence of battery swapping provides a new way to recharge electric vehicles, offering advantages such as shorter recharge times, improved grid load control, reduced battery failure rates, and extended battery life. By scheduling battery swapping for vehicles, the overall efficiency of vehicles in completing transportation tasks can be improved.

[0003] Existing vehicle battery swapping scheduling methods often employ mathematical modeling and solution. However, in specific application scenarios, the solution complexity is high, making it impossible to obtain the optimal scheduling strategy in a short time. Therefore, in practical application scenarios, once the scale of vehicle scheduling involved is large, it will be impossible to obtain the optimal scheduling strategy within an acceptable time, resulting in low vehicle scheduling efficiency.

[0004] Application content

[0005] The main objective of this application is to provide an electric vehicle scheduling method, apparatus, device, and storage medium, aiming to solve the technical problem of low vehicle scheduling efficiency in the prior art.

[0006] In a first aspect, embodiments of this application propose an electric vehicle scheduling method, the electric vehicle scheduling method comprising:

[0007] Obtain the current status information of the electric vehicle and the current status information of the battery swapping stations it passes through;

[0008] The current state information of the vehicle and the current state information of the battery swapping station along the route are input into a preset reinforcement learning model to obtain the target action of the vehicle. The preset reinforcement learning model is obtained by training the vehicle sample state information, the battery swapping station sample state information, the vehicle sample joint action, and the vehicle sample joint reward.

[0009] Vehicles are scheduled based on the target vehicle actions.

[0010] This embodiment proposes an electric vehicle scheduling method, including: acquiring the current vehicle state information and the current state information of the battery swapping stations along the route; inputting the current vehicle state information and the current state information of the battery swapping stations along the route into a preset reinforcement learning model to obtain the vehicle target action, wherein the preset reinforcement learning model is trained by vehicle sample state information, battery swapping station sample state information, vehicle sample joint actions, and vehicle sample joint rewards; scheduling vehicles based on the vehicle target action, thereby quickly and accurately obtaining the target action of each vehicle through the trained preset reinforcement learning model, and thus realizing the rapid scheduling of electric vehicles through the target actions of each vehicle, improving the accuracy and efficiency of vehicle scheduling.

[0011] In some embodiments, the step of inputting the current state information of the vehicle and the current state information of the battery swapping stations along the route into a preset reinforcement learning model to obtain the target action of the vehicle includes:

[0012] The current state information of the vehicle and the current state information of the battery swapping stations along the route are input into a preset reinforcement learning model for feature mapping to obtain the current joint state features.

[0013] Calculate the action value of each vehicle action based on the current joint state characteristics;

[0014] The target vehicle action is determined based on the action value.

[0015] In the technical solution of this application embodiment, feature mapping is performed by a preset reinforcement learning model to quickly characterize the features of vehicles and battery swapping stations, obtain the current joint state features, and calculate the action value of each action of the vehicle based on the current joint state features. The cumulative value of the action on the current state or subsequent states is considered, so as to accurately and quickly determine the target action of the vehicle and improve the scheduling effect.

[0016] In some embodiments, the step of inputting the current state information of the vehicle and the current state information of the battery swapping stations along the route into a preset reinforcement learning model for feature mapping to obtain the current joint state features includes:

[0017] The current state information of the vehicle and the current state information of the battery swapping stations along the route are input into a preset reinforcement learning model for structure extraction, so as to obtain the battery swapping station nodes, vehicle nodes, the edges between each battery swapping station along the route, and the edges between the vehicle and the battery swapping station.

[0018] Obtain the state characteristic function;

[0019] The current joint state features are obtained by performing feature mapping on the passing battery swapping station nodes, the vehicle nodes, the edges between each passing battery swapping station, and the edges between the vehicle and the passing battery swapping station using the state feature function.

[0020] In the technical solution of this application embodiment, the current status information of the vehicle and the current status information of the battery swapping station along the way are extracted as structured information, and the current status features of the vehicle and the battery swapping station along the way are extracted according to the status feature function, thereby improving the speed of status feature extraction.

[0021] In some embodiments, the step of performing feature mapping on the passing battery swapping station nodes, the vehicle nodes, the edges between each passing battery swapping station, and the edges between the vehicle and the passing battery swapping station using the state feature function to obtain the current joint state features includes:

[0022] Feature extraction is performed on the nodes of the battery swapping stations along the route to obtain a first feature vector representing the nodes of the battery swapping stations along the route.

[0023] Feature extraction is performed on the vehicle node to obtain a second feature vector representing the vehicle node;

[0024] Construct an adjacency matrix based on the passing battery swapping station nodes, the vehicle nodes, the edges between each passing battery swapping station, and the edges between the vehicle and the passing battery swapping station;

[0025] The current joint state features are obtained by calculating the first feature vector, the second feature vector, and the adjacency matrix using the state feature function.

[0026] In the technical solution of this application embodiment, by extracting features from the passing battery swapping station nodes and vehicle nodes, a first feature vector and a second feature vector are obtained respectively. The current joint state features can be quickly obtained by using the state feature function based on the constructed adjacency matrix, the first feature vector, and the second feature vector.

[0027] In some embodiments, the method further includes:

[0028] Obtain vehicle sample status information and battery swapping station sample status information;

[0029] A heterogeneous graph is constructed based on the vehicle sample status information and the battery swapping station sample status information. The structure of the heterogeneous graph includes one or more of the following: battery swapping station nodes, vehicle nodes, edges between each battery swapping station, and edges between vehicles and battery swapping stations.

[0030] An adjacency matrix is ​​constructed based on the heterogeneous graph, and features are extracted from the heterogeneous graph to obtain vehicle node feature vectors and battery swapping station node feature vectors.

[0031] Obtain the first mapping function of the graph neural network;

[0032] The state feature function of the graph neural network model is obtained by learning the state feature function through the first mapping function, the adjacency matrix, the vehicle node feature vector, and the battery swapping station node feature vector.

[0033] In the technical solution of this application embodiment, a structural relationship between the vehicle state and the battery swapping station state is established by pre-constructing a heterogeneous graph of the vehicle state and the battery swapping station state. The state feature function is learned by using the first mapping function of the set graph neural network and the extracted feature vector. Thus, when performing feature mapping, the learned state feature function can be used to quickly calculate the current joint state feature.

[0034] In some embodiments, the step of calculating the action value of each vehicle action based on the current joint state characteristics includes:

[0035] Obtain the vehicle joint reward under the current joint state characteristics;

[0036] Each vehicle action is obtained through the current joint state characteristics and the vehicle joint reward;

[0037] The action value of each action of the vehicle is calculated based on the current joint state characteristics and each action of the vehicle.

[0038] In the technical solution of this application embodiment, each action of the vehicle has a corresponding reward. By obtaining the joint reward of the vehicle under the current joint state features, each action of the vehicle is quickly determined by a preset reinforcement learning model, and the corresponding action value is accurately calculated based on each action of the vehicle.

[0039] In some embodiments, the step of calculating the action value of each vehicle action based on the current joint state characteristics and each vehicle action includes:

[0040] Obtain the action value function;

[0041] The action value function is used to calculate the action value of each action of the vehicle under the current joint state characteristics and each action of the vehicle.

[0042] In the technical solution of this application embodiment, the action value corresponding to each action of the vehicle under the current joint state characteristics is accurately evaluated by the action value function.

[0043] In some embodiments, the method further includes:

[0044] Obtain the second mapping function of the value neural network model;

[0045] Obtain the joint state features of the samples and the joint actions of the vehicle samples under the joint state features of the samples;

[0046] By learning the action value function through the second mapping function, the sample joint state features, and the vehicle sample joint actions, the action value function of the value neural network model is obtained.

[0047] In the technical solution of this application embodiment, the action value function can be learned in advance based on the second mapping function of the set value neural network model, the sample joint state features, and the vehicle sample joint actions, so that the action value corresponding to each subsequent vehicle action can be accurately evaluated based on the learned action value function.

[0048] In some embodiments, the step of determining the target vehicle action based on the action value includes:

[0049] The vehicle's actions are sorted based on their action value to obtain a list of vehicle actions.

[0050] Based on the action value, select a vehicle action from the vehicle action list that meets the preset action value to obtain the vehicle target action.

[0051] In the technical solution of this application embodiment, each vehicle action is sorted according to the magnitude of the action value of each action, thereby eliminating vehicle actions that do not meet the preset action value, obtaining the optimal vehicle actions, and improving the vehicle scheduling effect.

[0052] In some embodiments, the step of scheduling the vehicle based on the target vehicle action includes:

[0053] A scheduling strategy is generated based on the vehicle's target action;

[0054] Vehicles are scheduled using the aforementioned scheduling strategy.

[0055] In the technical solution of this application embodiment, a corresponding scheduling strategy can be generated based on the vehicle target action corresponding to the vehicle, thereby instructing the vehicle to be scheduled according to the scheduling strategy, thereby improving the scheduling effect.

[0056] In some embodiments, the method further includes:

[0057] The current status information of the vehicle and the current status information of the battery swapping station along the route are updated according to the target action of the vehicle to obtain updated vehicle status information and updated status information of the battery swapping station along the route, and the process returns to the step of obtaining the current status information of the electric vehicle and the current status information of the battery swapping station along the route.

[0058] In the technical solution of this application embodiment, the current status information of the vehicle and the status information of the battery swapping stations along the way can be updated according to the target action of the vehicle. In this way, the updated status information is used to continuously update the target action of the vehicle, thereby depicting the scheduling process of each vehicle from a global perspective and improving the scheduling effect.

[0059] Secondly, embodiments of this application also propose an electric vehicle dispatching device, the electric vehicle dispatching device comprising:

[0060] The acquisition module is used to acquire the current status information of the electric vehicle and the current status information of the battery swapping stations it passes through;

[0061] The input module is used to input the current state information of the vehicle and the current state information of the battery swapping station along the route into a preset reinforcement learning model to obtain the target action of the vehicle. The preset reinforcement learning model is obtained by training the vehicle sample state information, the battery swapping station sample state information, the vehicle sample joint action, and the vehicle sample joint reward.

[0062] The scheduling module is used to schedule vehicles based on the target vehicle actions.

[0063] Thirdly, embodiments of this application also propose an electric vehicle scheduling device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the electric vehicle scheduling method described above.

[0064] Fourthly, embodiments of this application also propose a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the electric vehicle scheduling method described above.

[0065] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of the electric vehicle scheduling method described above. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating an embodiment of the electric vehicle scheduling method proposed in this application.

[0069] Figure 2 This is a schematic diagram showing the locations of electric vehicles, battery swapping stations, and warehouses in one embodiment of the electric vehicle scheduling method proposed in this application.

[0070] Figure 3 This is a schematic diagram of the overall process of electric vehicle scheduling in one embodiment of the electric vehicle scheduling method proposed in this application;

[0071] Figure 4 This is another flowchart illustrating an embodiment of the electric vehicle scheduling method proposed in this application.

[0072] Figure 5 This is a schematic diagram of the specific structure of the heterogeneous graph in one embodiment of the electric vehicle scheduling method proposed in this application;

[0073] Figure 6 This is a schematic diagram illustrating the modeling and training process of a preset reinforcement learning model in one embodiment of the electric vehicle scheduling method proposed in this application.

[0074] Figure 7 This is a block diagram of the module structure of the electric vehicle dispatching device according to an embodiment of this application;

[0075] Figure 8 This is a schematic diagram of the equipment structure of the hardware operating environment involved in the electric vehicle scheduling method in this application embodiment.

[0076] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0077] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0078] To better understand the technical solution of this application, the embodiments of the technical solution of this application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solution of this application more clearly, and are therefore only examples, and cannot be used to limit the protection scope of this application.

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0080] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0081] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0082] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0083] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0084] In the description of the embodiments of this application, the technical terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0085] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0086] Currently, existing electric vehicle scheduling strategies employ mathematical modeling and problem-solving methods. The modeled problem is an NP-hard problem, which has high complexity and cannot be solved in polynomial time. In actual vehicle scheduling scenarios, if the problem scale is large, it is impossible to obtain the optimal strategy within the set time or the user's acceptable time, thus affecting the efficiency of vehicle scheduling.

[0087] To address the aforementioned deficiencies, the inventive concept of this application is as follows:

[0088] A deep reinforcement learning-based electric vehicle scheduling method is proposed. This method aims to ensure scheduling accuracy while promptly handling unexpected events during the scheduling process by training a pre-set reinforcement learning model offline and then using the pre-set reinforcement learning model for online inference.

[0089] To achieve this objective, this embodiment first acquires the current status information of the vehicle and the current status information of the battery swapping stations it passes through. This information is then input into a preset reinforcement learning model to determine the vehicle's target action. The preset reinforcement learning model can determine the corresponding vehicle target action based on the vehicle's current status and the current status of the battery swapping stations it passes through. This allows for vehicle scheduling based on the target action, precise control of the scheduling process for each vehicle, and responses to different emergencies based on the actions of each vehicle. This enables timely updates to the vehicle scheduling, ensuring scheduling accuracy while promptly handling emergencies and efficiently solving the battery swapping scheduling problem.

[0090] In this embodiment, for ease of description, the electric vehicle dispatching equipment will be used as the execution subject in the following description.

[0091] Based on the above considerations, in order to solve the problem of low vehicle dispatching efficiency, after in-depth research, an electric vehicle dispatching method, device, equipment and storage medium are proposed. It is applicable to various scenarios of dispatching and swapping batteries during the operation of different types of vehicles. For example, in the specific application scenario of swapping batteries for trucks or passenger buses performing transportation tasks, or in the specific application scenario of swapping batteries sequentially for each electric vehicle in a fleet of electric vehicles traveling on a main line.

[0092] To illustrate the technical solution of this application, specific embodiments are described below.

[0093] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or electric vehicle dispatching device capable of performing the above functions. The following description uses an electric vehicle dispatching device as an example to illustrate this embodiment and the subsequent embodiments.

[0094] In some embodiments of this application, reference is made to Figure 1 This invention proposes an electric vehicle scheduling method, which may include:

[0095] Step S10: Obtain the current status information of the electric vehicle and the current status information of the battery swapping stations it passes through.

[0096] like Figure 2 As shown, Figure 2 This diagram illustrates the locations of electric vehicles, battery swapping stations, and warehouses along a main road. Several warehouses, battery swapping stations, and electric vehicles are present. Each battery swapping station has several charging bays and batteries for battery swapping. Each electric vehicle is assigned several tasks by the fleet. A task refers to a vehicle departing from warehouse A at a certain time and ultimately arriving at warehouse B. During task execution, electric vehicles need to stop at battery swapping stations to replenish their energy and ensure the smooth completion of the task. This embodiment optimizes vehicle battery swapping scheduling by scheduling vehicles to ensure that each vehicle reaches the optimal battery swapping station, thereby achieving optimization goals such as maximizing the revenue of all battery swapping stations or all vehicles along the main road, or minimizing vehicle queuing times.

[0097] It should be noted that the vehicle's current status information while in motion includes, but is not limited to, the vehicle's task information, current location, current speed, current battery level, and estimated arrival time at the battery swapping station. This current status information can be obtained in real time through sensors installed on the vehicle and is the vehicle's latest status information.

[0098] The current status information of the battery swapping stations along the route includes, but is not limited to, the location of the battery swapping stations, the occupancy status of the charging compartments within the stations, battery inventory, queue length, battery swapping demand, number of swaps, number of fully charged batteries, charging power, battery swapping operation time, time-of-use pricing, current SOC of each battery, number of vehicles currently swapping batteries, and battery swapping efficiency. This information can be acquired in real time through vehicle sensors, positioning systems, and the management system within the battery swapping stations and transmitted to the dispatch center.

[0099] The battery swapping stations along the route are those that a vehicle may pass through while traveling on a certain main road. If the vehicle runs out of power during its journey, it will need to go to the nearest battery swapping station to swap or charge its battery. Therefore, information on all battery swapping stations along the vehicle's route can be obtained in advance, and battery swapping stations that the vehicle will not pass through can be filtered out based on the vehicle's transportation task, thus obtaining information on the battery swapping stations along the route.

[0100] Step S20: Input the current state information of the vehicle and the current state information of the battery swapping stations along the way into the preset reinforcement learning model to obtain the target action of the vehicle. The preset reinforcement learning model is obtained by training the vehicle sample state information, the battery swapping station sample state information, the vehicle sample joint action, and the vehicle sample joint reward.

[0101] It should be noted that the preset reinforcement learning model is obtained through offline training based on vehicle sample state information, battery swapping station sample state information, vehicle sample joint actions, and vehicle sample joint rewards.

[0102] It is understandable that vehicle sample status information can be collected in advance, showing different statuses of the vehicle at different times. Similarly, battery swapping station sample status information can be collected in advance, showing different statuses of the battery swapping station at different times.

[0103] The vehicle sample joint actions can be obtained by collecting vehicle joint actions in advance. The vehicle joint actions are which battery swapping station each vehicle goes to at each time. By collecting the actions of each vehicle going to the corresponding battery swapping station, the vehicle joint actions can be obtained by summarizing them.

[0104] The joint reward for vehicle samples can be set in advance. The joint reward for vehicles is the reward obtained by each vehicle for taking the target action in the corresponding state, such as the result of queuing reduction or battery swapping revenue improvement. Specific vehicle rewards can be set according to the optimization goals of each vehicle. For example, if the vehicle optimization goal is to minimize queuing time, the vehicle reward can be set as the queuing time reduction obtained by taking the corresponding action.

[0105] Understandably, a simulation model can be built in advance to simulate the state updates of vehicles and battery swapping stations. A reinforcement learning model can then be built based on the specific battery swapping scenario. This model continuously updates the states of the vehicles and battery swapping stations through the simulation, obtaining updated joint state information. The reinforcement learning model is then continuously trained using vehicle sample joint actions, vehicle sample joint rewards, and the updated joint state information, resulting in a pre-defined reinforcement learning model. In practical implementation, the initial reinforcement learning model can be trained through continuous interaction with the simulation model, thus obtaining the pre-defined reinforcement learning model.

[0106] In this embodiment, the current status information of the vehicle and the current status information of the battery swapping stations along the route can be input into a preset reinforcement learning model. The preset reinforcement learning model can then directly provide vehicle scheduling actions, i.e., vehicle target actions, online based on actual data.

[0107] In one embodiment, the reinforcement learning model can also be trained directly using vehicle sample state information, battery swapping station sample state information, vehicle sample joint actions, and vehicle sample scheduling strategies to generate a preset reinforcement learning model. Then, during vehicle scheduling, the current state information of the vehicle and the current state information of the battery swapping stations along the way are directly input into the preset reinforcement learning model to directly output the target scheduling strategy.

[0108] It should be noted that the preset reinforcement learning model is obtained by repeatedly interacting and training the simulation model with the initial reinforcement learning model that is being trained.

[0109] Step S30: Dispatch vehicles based on their target actions.

[0110] In practice, after obtaining the target vehicle action, each vehicle can be scheduled accordingly based on the target vehicle action. For example, the vehicle can be scheduled to go to the corresponding battery swapping station based on the target vehicle action.

[0111] It should be noted that if a vehicle fails to proceed to the corresponding battery swapping station according to the target action, the status of both the vehicle and the battery swapping station can be updated, thereby generating a new target action for the vehicle. This will promptly resolve any unforeseen events where vehicles fail to be dispatched according to the target action and improve the real-time performance of dispatching.

[0112] like Figure 3 As shown, Figure 3 This is a schematic diagram of the overall process of electric vehicle scheduling in this embodiment, divided into an offline training process and an online inference process. The offline training process may include building a battery swapping scheduling simulation model. The role of the battery swapping scheduling simulation model is to simulate the state updates of vehicles and battery swapping stations, and to build an initial reinforcement learning model based on different battery swapping scenarios or a specific battery swapping scenario. The reinforcement learning model is trained through the simulation model, and after training, a preset reinforcement learning model is obtained. The online inference process may include acquiring real data of vehicles in the battery swapping scenario and inputting the real data into the preset reinforcement learning model, thereby quickly outputting a battery swapping scheduling strategy based on the preset reinforcement learning model, improving the accuracy and efficiency of scheduling.

[0113] In some embodiments, the preset reinforcement learning model processes the current state information and the current state information of the passing battery swapping stations, including feature extraction and action value calculation, referring to... Figure 4 Step S20 includes:

[0114] Step S201: Input the current state information of the vehicle and the current state information of the battery swapping stations along the route into the preset reinforcement learning model for feature mapping to obtain the current joint state features.

[0115] It should be noted that by inputting the current state information of the vehicle and the current state information of the battery swapping stations along the way into the preset reinforcement learning model, the preset reinforcement learning model can be used to perform feature mapping to obtain the current state features of the vehicle and the current state features of the battery swapping stations along the way, and the current joint state features can be obtained based on the current state features of the vehicle and the current state features of the battery swapping stations along the way.

[0116] It should be noted that the current joint state characteristics represent the state characteristics of the vehicle and the state characteristics of the battery swapping station at the current moment.

[0117] In practical implementation, when using a pre-set reinforcement learning model for feature mapping, the current state information of the vehicle and the current state information of the battery swapping stations along the route can be analyzed first to identify important or necessary features, such as the vehicle's current battery level, the vehicle's estimated driving range, the queuing time at the battery swapping stations along the route, and the battery inventory status at the battery swapping stations along the route.

[0118] The pre-defined reinforcement learning model can include a feature mapping function to integrate these different pieces of information into a joint state feature representing the current situation. This joint state feature reflects the current status of the vehicle and the battery swapping stations it passes, providing a basis for subsequent decisions, such as whether to go to a battery swapping station and which station to choose.

[0119] Step S202: Calculate the action value of each action of the vehicle based on the current joint state characteristics.

[0120] It should be noted that the action value of each vehicle action is calculated based on the current joint state characteristics.

[0121] When calculating action value, a vehicle joint reward can be set under the current joint state characteristics, so that the action value of each action taken by the vehicle can be calculated based on the vehicle joint reward under the current joint state characteristics.

[0122] It should be noted that the action value of each action is an estimate of the expected reward or value generated by taking each action under the current joint state characteristics. It can be used to measure the merits of taking different actions under the current joint state characteristics, thereby guiding the vehicle to choose the optimal action when making decisions.

[0123] In practice, the value of an action can be calculated using the action value function, or other functions for calculating action value can be used. This embodiment does not impose any restrictions on this.

[0124] Step S203: Determine the target vehicle action based on the action value.

[0125] Understandably, the value of each action can be calculated or filtered to select the optimal vehicle action, i.e., the target vehicle action.

[0126] In the technical solution of this application embodiment, feature mapping is performed by a preset reinforcement learning model to quickly characterize the features of vehicles and battery swapping stations, obtain the current joint state features, and calculate the action value of each action of the vehicle based on the current joint state features. The cumulative value of the action on the current state or subsequent states is considered, so as to accurately and quickly determine the target action of the vehicle and improve the scheduling effect.

[0127] In some embodiments, the feature mapping function may be a state feature function; therefore, step S201 may include:

[0128] The current status information of the vehicle and the current status information of the battery swapping stations along the route are input into the preset reinforcement learning model for structure extraction, which yields the nodes of the battery swapping stations along the route, the vehicle nodes, the edges between each battery swapping station along the route, and the edges between the vehicle and the battery swapping station.

[0129] Understandably, structural extraction involves constructing a heterogeneous graph from the vehicle's current state information and the current state information of the battery swapping stations it passes through, which includes two attributes: nodes and edges.

[0130] In practice, there are two types of nodes: battery swapping station nodes and vehicle nodes. Therefore, the structure can be extracted by a preset reinforcement learning model to obtain the battery swapping station nodes and vehicle nodes that pass through. The attributes of the battery swapping station nodes may include charging rate, battery swapping station SOC information, queue number, etc., while the attributes of the vehicle nodes may include vehicle speed, vehicle SOC, power consumption, etc.

[0131] In practical implementation, there are two types of edges: edges between battery swapping stations and edges between electric vehicles and battery swapping stations. Therefore, a pre-set reinforcement learning model can be used for structure extraction to obtain the edges between each passing battery swapping station and the edges between vehicles and battery swapping stations. The edges between each passing battery swapping station are adjacent stations and can assist in the transmission of state information between battery swapping stations. The edges between vehicles and battery swapping stations are between vehicles and reachable stations and can describe the transmission of state information between vehicles and stations. Based on the structure of vehicles and passing battery swapping stations, the characteristics of vehicles and passing battery swapping stations can be more easily characterized.

[0132] Obtain the state characteristic function;

[0133] It should be noted that the state feature function is a pre-learned function used to extract features from the state.

[0134] In specific implementations, the state feature function can be a multinomial feature function, a nonlinear feature function, a dictionary feature function, a convolutional feature function, etc., and this embodiment does not impose any restrictions on it.

[0135] In some embodiments, the state feature function can be learned in advance to improve the efficiency of subsequent feature mapping. The method further includes:

[0136] Obtain vehicle sample status information and battery swapping station sample status information;

[0137] It should be noted that the vehicle sample status information is the historical status information of vehicles in different states, and the battery swapping station sample status information is the historical status information of battery swapping stations in different states.

[0138] A heterogeneous graph is constructed based on vehicle sample status information and battery swapping station sample status information. The structure of the heterogeneous graph includes one or more of the following: battery swapping station nodes, vehicle nodes, edges between each battery swapping station, and edges between vehicles and battery swapping stations.

[0139] In practical implementation, a heterogeneous graph can be constructed based on vehicle sample status information and battery swapping station sample status information. The specific structure of the heterogeneous graph is as follows: Figure 5 It includes battery swapping station (BS) nodes and electric vehicle (EV) nodes, as well as edges between battery swapping stations and edges between vehicles and battery swapping stations.

[0140] In practice, the heterogeneous graphs constructed can be used to quickly characterize the features of vehicles and battery swapping stations, thereby improving the efficiency of feature extraction.

[0141] An adjacency matrix is ​​constructed based on the heterogeneous graph, and features are extracted from the heterogeneous graph to obtain the feature vectors of vehicle nodes and battery swapping station nodes.

[0142] In practical implementation, an adjacency matrix can be constructed based on the heterogeneous graph. This matrix represents the connection relationships between different nodes in the graph. The adjacency matrix is ​​typically represented by a square matrix, where the rows and columns correspond to the nodes in the graph, and the elements represent the connections between nodes—that is, the edges between battery swapping stations and the edges between vehicles and battery swapping stations. The adjacency matrix can be used to represent the graph structure, calculate graph attributes (such as degree and path), and perform graph computation, graph search, and graph analysis. Using the adjacency matrix, the connection relationships between nodes can be easily found, enabling various graph algorithms and related tasks.

[0143] In practice, feature extraction models or algorithms can be used to extract features from heterogeneous graphs, thereby obtaining the vehicle node feature vector V and the battery swapping station node feature vector U.

[0144] Obtain the first mapping function of the graph neural network;

[0145] In practical implementation, the pre-set reinforcement learning model structure may include a graph neural network model and a value neural network model. The graph neural network model is used to extract the current state features of the vehicle and the battery swapping station. Before constructing the graph neural network model, the graph neural network can be trained. The graph neural network model includes state feature functions, which are learned in advance to construct the graph neural network model.

[0146] Understandably, the first mapping function of a graph neural network can be pre-set.

[0147] The state feature function of the graph neural network model is obtained by learning the state feature function through the first mapping function, the adjacency matrix, the vehicle node feature vector, and the battery swapping station node feature vector.

[0148] In practical implementation, the state feature function can be learned through the first mapping function, adjacency matrix, vehicle node feature vector, and battery swapping station node feature vector, thereby obtaining the state feature function of the graph neural network model. The process of learning the state feature function is as follows: Equation 1:

[0149] S=φ GCN (U,V,G)(Equation 1)

[0150] In Equation 1 above, S represents the joint state characteristic, and φ GCN Let U be the first mapping function of the graph neural network, U be the feature vector of the battery swapping station node, V be the feature vector of the vehicle node, and G be the adjacency matrix.

[0151] By pre-constructing heterogeneous graphs of vehicle and battery swapping station states, a structural relationship between the two states is established. The state feature function is learned using the first mapping function of the graph neural network and the extracted feature vectors. This allows for the rapid calculation of the current joint state features using the learned state feature function during feature mapping.

[0152] The current joint state features are obtained by performing feature mapping on the nodes passing through the battery swapping station, vehicle nodes, edges between each passing battery swapping station, and edges between vehicles and passing battery swapping stations using the state feature function.

[0153] It should be understood that the state features of vehicles and the state features of passing battery swapping stations can be extracted by performing feature mapping on the nodes passing through the battery swapping stations, vehicle nodes, the edges between each passing battery swapping station, and the edges between vehicles and battery swapping stations through the state feature function, thus obtaining the current joint state features.

[0154] In the technical solution of this application embodiment, the current status information of the vehicle and the current status information of the battery swapping station along the way are extracted as structured information, and the current status features of the vehicle and the battery swapping station along the way are extracted according to the status feature function, thereby improving the speed of status feature extraction.

[0155] In some embodiments, step S201 may include:

[0156] Feature extraction is performed on the nodes passing through the battery swapping station to obtain the first feature vector representing the nodes passing through the battery swapping station.

[0157] In practical implementation, feature extraction models or algorithms can be used to extract features from the nodes passing through the battery swapping stations, thereby obtaining the first feature vector representing the nodes. When extracting features from the nodes, the location of the nodes, queuing information, and inventory information of the battery swapping stations can be extracted.

[0158] Feature extraction is performed on vehicle nodes to obtain a second feature vector representing the vehicle nodes;

[0159] In practice, features of vehicle nodes can be extracted using feature extraction models or algorithms to obtain a second feature vector representing the vehicle node. When extracting features from vehicle nodes, information such as the vehicle node's location, battery level, and speed can be extracted.

[0160] Construct an adjacency matrix based on the nodes of the battery swapping stations along the route, the vehicle nodes, the edges between each battery swapping station along the route, and the edges between the vehicle and the battery swapping stations along the route.

[0161] It should be noted that since the rows and columns of the adjacency matrix correspond to the nodes of the passing battery swapping stations and the vehicle nodes, respectively, and the elements in the matrix represent the connection between the nodes, that is, the edges between each battery swapping station and the edges between the vehicle and the battery swapping station, the adjacency matrix can be quickly constructed based on the nodes of the passing battery swapping stations, the vehicle nodes, the edges between each passing battery swapping station, and the edges between the vehicle and the passing battery swapping station.

[0162] The current joint state features are obtained by calculating the first eigenvector, the second eigenvector, and the adjacency matrix using the state feature function.

[0163] It should be noted that the first eigenvector, the second eigenvector, and the adjacency matrix can be used as state variables. The state feature function processes these state variables to obtain a new state feature S, which is the current joint state feature.

[0164] In the technical solution of this application embodiment, by extracting features from the passing battery swapping station nodes and vehicle nodes, a first feature vector and a second feature vector are obtained respectively. The current joint state features can be quickly obtained by using the state feature function based on the constructed adjacency matrix, the first feature vector, and the second feature vector.

[0165] In some embodiments, the action value of each action of the vehicle can be evaluated based on the current joint state characteristics, then step S202 may include:

[0166] Obtain the vehicle joint reward under the current joint state characteristics;

[0167] It should be noted that vehicle joint rewards are the sum of rewards obtained by vehicles for taking actions under the current joint state characteristics. For example, vehicle joint rewards may be to shorten queuing time or increase battery swapping revenue.

[0168] Each action of the vehicle is obtained by using the current joint state characteristics and the vehicle joint reward;

[0169] In this embodiment, each action of the vehicle can be obtained through the current joint state features and the vehicle joint reward. Since the preset reinforcement learning model has been trained in advance to establish the relationship between the joint reward, the joint state and the action, each corresponding action of the vehicle can be obtained based on the current joint state features and the vehicle joint reward.

[0170] like Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the modeling and training process of a pre-defined reinforcement learning model. The environment corresponds to the simulation model, used to update the states of the vehicle and the battery swapping station. The agent represents the electric vehicle, and the joint state S... t This represents the status information of the vehicle and the battery swapping station at time t, and action a. i The reward r is given for choosing which battery swapping station vehicle i goes to. i Represents the current state S t Below, vehicle i takes action a i The rewards received.

[0171] For t = 1, 2, 3, ..., T, all vehicles acquire the joint state S through the environment. t Each vehicle, based on the joint state and joint reward {r1, r2, ... r... n} Decision actions {a1, a2, ... a n The environment (simulation model) is based on the joint actions {a1, a2, ... a...}. n Update the joint state S t+1 .

[0172] Calculate the action value of each action of the vehicle based on the current joint state characteristics and each action of the vehicle.

[0173] It should be noted that the action value of each action of the vehicle can be calculated based on the current joint state characteristics and each action of the vehicle. For example, the action value of each action of the vehicle can be calculated through the action value function.

[0174] In some embodiments, the step of calculating the action value of each vehicle action based on the current joint state characteristics and each vehicle action includes:

[0175] Obtain the action value function;

[0176] The action value function is used to calculate the action value of each action of the vehicle under the current joint state characteristics and each action of the vehicle.

[0177] It should be noted that the action value function can be learned in advance, for example, by learning the action value function through another neural network model, obtaining the action value function, and then constructing a value neural network model.

[0178] In practice, the action value function characterizes the relationship between joint state features, vehicle actions, and action values. Therefore, the action value of each action of the vehicle under the current joint state features can be obtained by calculating the current joint state features and each vehicle action through the action value function.

[0179] The action value of each action can be calculated using the action value function in the value neural network model for each possible action a. i The action value Q(s, a) can be accurately evaluated through the action value function for each action of the vehicle under the current joint state characteristics.

[0180] In the technical solution of this application embodiment, each action of the vehicle has a corresponding reward. By obtaining the joint reward of the vehicle under the current joint state features, each action of the vehicle is quickly determined by a preset reinforcement learning model, and the corresponding action value is accurately calculated based on each action of the vehicle.

[0181] In some embodiments, the action value function of the value neural network model can be learned in advance to improve the efficiency of action value calculation. The method further includes:

[0182] Obtain the second mapping function of the value neural network model;

[0183] It should be noted that the second mapping function is used to construct the relationship between joint state features, joint vehicle actions, and action value functions. The second mapping function can be learned or set in advance.

[0184] Obtain the joint state features of the samples and the joint actions of the vehicle samples under the joint state features;

[0185] In this embodiment, the sample joint state features are the features of vehicles and battery swapping stations under different states collected in advance, and the vehicle sample joint actions are the summary of different actions of vehicles under the sample joint state features.

[0186] The action value function of the value neural network model is obtained by learning the action value function through the second mapping function, the joint state features of the samples, and the joint actions of the vehicle samples.

[0187] It is understandable that the action value function of the value neural network model can be obtained by learning the action value function through the second mapping function, the joint state features of the samples, and the joint actions of the vehicle samples. The learning process is as follows: Equation 2:

[0188] Q(S,A)=φ DRL (S,A)(Equation 2)

[0189] In Equation 2 above, Q(S,A) is the action value function, φ DRL Let S be the joint state feature and A be the joint vehicle action. The value neural network can be an MLP, CNN, or other similar neural network. The action value function can be learned through various methods, such as Q-learning, Deep Q-Network (DQN), or other reinforcement learning algorithms. The convergence of the reinforcement learning model is guaranteed by combining multi-agent reinforcement learning techniques with the message passing mechanism of the graph neural network model.

[0190] In the technical solution of this application embodiment, the action value function can be learned in advance based on the second mapping function of the set value neural network model, the sample joint state features, and the vehicle sample joint actions, so that the action value corresponding to each subsequent vehicle action can be accurately evaluated based on the learned action value function.

[0191] In some embodiments, the target vehicle action can be determined based on the vehicle's action value, then step S203 may include:

[0192] The vehicle's actions are sorted based on their action value to obtain a list of vehicle actions.

[0193] Based on the action value, select the vehicle action that meets the preset action value from the vehicle action list to obtain the vehicle target action.

[0194] It should be noted that each action of the vehicle can be sorted according to the action value of each action. For example, the actions of the vehicle include action 1: vehicle 1 goes to battery swap station A to swap batteries, action 2: vehicle 1 goes to battery swap station C to swap batteries, and action 3: vehicle 1 goes to battery swap station D to swap batteries. The action value of action 1 is 0.8, the action value of action 2 is 0.7, and the action value of action 3 is 0.95. The actions can be sorted from largest to smallest or smallest to largest to obtain the list of vehicle actions.

[0195] In practice, each vehicle action can be ranked based on its value, thus determining the order of priority for each action. This ranking provides a clearer understanding of the performance of each action, offering a basis for subsequent scheduling decisions. The evaluation of action value depends not only on the value of the current state but also on the cumulative value of that action to subsequent states.

[0196] It should be understood that vehicle actions that meet a preset action value can be selected from the vehicle action list based on the action value. The preset action value is the maximum action value. The optimal action can be obtained by using an ε-greedy strategy. The process of obtaining the target vehicle action using the greedy strategy is as follows: Equation 3:

[0197] π * =argmax a Q(s,a) (Equation 3)

[0198] Using Equation 3 above, the action with the highest action value can be selected, where a∈A represents the vehicle action and s∈S represents the joint state feature. A greedy strategy is used to obtain the optimal action under different states. An empty action sequence can be set to store the optimal action sequence. For the current joint state s, all possible actions a are traversed, the action value Q of each action is calculated, and the action with the highest value is selected as the vehicle target action. The vehicle target action is added to the action sequence, and the state transition is performed according to the selected action, updating the current joint state. This can accurately characterize the correlation between the state updates of each vehicle and the battery swapping station in the entire trunk line.

[0199] It should be noted that different types of pre-defined reinforcement learning models achieve the above objectives in different ways. For example, if a value-based reinforcement learning model (such as Q-Learning) is used, a value model needs to be trained to evaluate the value of different actions, and then the appropriate action is selected based on the value. If a policy gradient-based reinforcement learning model is used, we directly train a model to output the current (optimal) policy. For example, we can use vehicle state information, the state information of the battery swapping stations along the route, and the scheduling policy to train a reinforcement learning model representing the scheduling policy, and directly obtain the optimal scheduling policy from the reinforcement learning model based on real-time data.

[0200] In the technical solution of this application embodiment, each vehicle action is sorted according to the magnitude of the action value of each action, thereby eliminating vehicle actions that do not meet the preset action value, obtaining the optimal vehicle actions, and improving the vehicle scheduling effect.

[0201] In some embodiments, the step of scheduling vehicles based on a target vehicle action includes:

[0202] Generate scheduling strategies based on vehicle target actions;

[0203] Vehicles are dispatched using scheduling strategies.

[0204] It should be noted that a scheduling strategy can be generated based on the vehicle's target action. Since the vehicle's target action is for vehicle i to go to the target battery swapping station for charging and swapping, a corresponding scheduling strategy can be generated.

[0205] In practice, the scheduling strategy may include the charging and swapping time, the specific battery swapping station, and the specific vehicles to be charged or swapped. The vehicles can be scheduled through the scheduling strategy. For example, at time t, vehicles 1 and 3 can be arranged to go to battery swapping station A for battery swapping.

[0206] In the technical solution of this application embodiment, a corresponding scheduling strategy can be generated based on the vehicle target action corresponding to the vehicle, thereby instructing the vehicle to be scheduled according to the scheduling strategy, thereby improving the scheduling effect.

[0207] In some embodiments, after obtaining the vehicle's target action, the vehicle's target action will affect the subsequent vehicle scheduling process; therefore, the method further includes:

[0208] The current status information of the vehicle and the current status information of the battery swapping stations along the route are updated according to the target action of the vehicle to obtain the updated status information of the vehicle and the updated status information of the battery swapping stations along the route, and the steps of obtaining the current status information of the electric vehicle and the current status information of the battery swapping stations along the route are returned.

[0209] It should be noted that after receiving the vehicle's target action, the vehicle will be dispatched according to the target action. However, some unforeseen events may occur, such as the battery swapping station being dispatched having no inventory or the driver not swapping batteries according to the vehicle's target action. In such cases, it is necessary to handle the unforeseen events in a timely manner. The current status information of the vehicle and the current status information of the battery swapping stations along the route will be updated according to the vehicle's target action. The vehicle's target action will then be re-determined based on the updated status information. This can automatically and promptly respond to unforeseen events during dispatching, thereby improving dispatching efficiency.

[0210] In practical applications, the vehicle will perform action a in state s and then update the (Q(s,a)) function based on the feedback from the environment (new state and possible reward) to improve future action selection.

[0211] In the technical solution of this application embodiment, the current status information of the vehicle and the status information of the battery swapping stations along the way can be updated according to the target action of the vehicle. In this way, the updated status information is used to continuously update the target action of the vehicle, thereby depicting the scheduling process of each vehicle from a global perspective and improving the scheduling effect.

[0212] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the electric vehicle scheduling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0213] This application also provides an electric vehicle dispatching device, please refer to... Figure 7 , Figure 7 This paper illustrates a structural diagram of an embodiment of the electric vehicle dispatching device provided in this application. The electric vehicle dispatching device includes:

[0214] The acquisition module 10 is used to acquire the current status information of the electric vehicle and the current status information of the battery swapping stations it passes through.

[0215] The input module 20 is used to input the current state information of the vehicle and the current state information of the battery swapping stations it passes into a preset reinforcement learning model to obtain the target action of the vehicle. The preset reinforcement learning model is obtained by training the vehicle sample state information, the battery swapping station sample state information, the vehicle sample joint action, and the vehicle sample joint reward.

[0216] The scheduling module 30 is used to schedule vehicles based on the target actions of the vehicles.

[0217] This embodiment acquires the current state information of electric vehicles and the current state information of passing battery swapping stations; it inputs the current state information of the vehicles and the current state information of passing battery swapping stations into a preset reinforcement learning model to obtain the target actions of the vehicles. The preset reinforcement learning model is trained using vehicle sample state information, battery swapping station sample state information, vehicle sample joint actions, and vehicle sample joint rewards; and it schedules vehicles based on the target actions of the vehicles. By quickly and accurately obtaining the target actions of each vehicle through the trained preset reinforcement learning model, it achieves rapid scheduling of electric vehicles through the target actions of each vehicle, thereby improving the accuracy and efficiency of vehicle scheduling.

[0218] In one embodiment, the input module 20 is further configured to input the current state information of the vehicle and the current state information of the battery swapping stations it passes through into a preset reinforcement learning model for feature mapping to obtain the current joint state features; calculate the action value of each action of the vehicle based on the current joint state features; and determine the target action of the vehicle based on the action value.

[0219] In one embodiment, the input module 20 is further configured to input the current state information of the vehicle and the current state information of the battery swapping stations along the route into a preset reinforcement learning model for structure extraction, thereby obtaining the battery swapping station nodes along the route, the vehicle nodes, the edges between each battery swapping station along the route, and the edges between the vehicle and the battery swapping station; obtain the state feature function; and perform feature mapping on the battery swapping station nodes along the route, the vehicle nodes, the edges between each battery swapping station along the route, and the edges between the vehicle and the battery swapping station through the state feature function to obtain the current joint state feature.

[0220] In one embodiment, the input module 20 is further configured to extract features from the nodes passing through the battery swapping stations to obtain a first feature vector representing the nodes passing through the battery swapping stations; extract features from the vehicle nodes to obtain a second feature vector representing the vehicle nodes; construct an adjacency matrix based on the nodes passing through the battery swapping stations, the vehicle nodes, the edges between each passing battery swapping station, and the edges between the vehicle and the passing battery swapping station; and calculate the first feature vector, the second feature vector, and the adjacency matrix using a state feature function to obtain the current joint state features.

[0221] In one embodiment, the input module 20 is further configured to acquire vehicle sample state information and battery swapping station sample state information; construct a heterogeneous graph based on the vehicle sample state information and battery swapping station sample state information, wherein the structure of the heterogeneous graph includes one or more of the following: battery swapping station nodes, vehicle nodes, edges between each battery swapping station, and edges between vehicles and battery swapping stations; construct an adjacency matrix based on the heterogeneous graph, and extract features from the heterogeneous graph to obtain vehicle node feature vectors and battery swapping station node feature vectors; acquire the first mapping function of the graph neural network; and learn the state feature function of the graph neural network model by using the first mapping function, the adjacency matrix, the vehicle node feature vectors, and the battery swapping station node feature vectors.

[0222] In one embodiment, the input module 20 is further configured to obtain the vehicle joint reward under the current joint state features; obtain each action of the vehicle through the current joint state features and the vehicle joint reward; and calculate the action value of each action of the vehicle based on the current joint state features and each action of the vehicle.

[0223] In one embodiment, the input module 20 is further configured to obtain an action value function;

[0224] The action value function is used to calculate the action value of each action of the vehicle under the current joint state characteristics and each action of the vehicle.

[0225] In one embodiment, the input module 20 is further configured to obtain a second mapping function of the value neural network model;

[0226] Obtain the joint state features of the samples and the joint actions of the vehicle samples under the joint state features;

[0227] The action value function of the value neural network model is obtained by learning the action value function through the second mapping function, the joint state features of the samples, and the joint actions of the vehicle samples.

[0228] In one embodiment, the input module 20 is further configured to sort each action of the vehicle based on the magnitude of the action value of each action of the vehicle, thereby obtaining a list of vehicle actions;

[0229] Based on the action value, select the vehicle action that meets the preset action value from the vehicle action list to obtain the vehicle target action.

[0230] In one embodiment, the scheduling module 30 is further configured to generate a scheduling strategy based on the vehicle's target action;

[0231] Vehicles are dispatched using scheduling strategies.

[0232] In one embodiment, the electric vehicle scheduling device further includes: an update module; the update module is used to update the current status information of the vehicle and the current status information of the battery swapping stations along the route according to the target action of the vehicle, to obtain updated vehicle status information and updated status information of the battery swapping stations along the route, and to return to the step of obtaining the current status information of the electric vehicle and the current status information of the battery swapping stations along the route.

[0233] This application provides an electric vehicle scheduling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the electric vehicle scheduling method in the above embodiment 1.

[0234] The following is for reference. Figure 8The diagram illustrates a structural schematic suitable for implementing the electric vehicle dispatching device in the embodiments of this application. The electric vehicle dispatching device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electric vehicle dispatching device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0235] like Figure 8 As shown, the electric vehicle dispatching device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the electric vehicle dispatching device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the electric vehicle dispatching equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows electric vehicle dispatching equipment with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0236] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0237] The electric vehicle dispatching device provided in this application, employing the electric vehicle dispatching method in the above embodiments, can solve the technical problem of low vehicle dispatching efficiency. Compared with the prior art, the beneficial effects of the electric vehicle dispatching device provided in this application are the same as those of the electric vehicle dispatching method provided in the above embodiments, and other technical features in this electric vehicle dispatching device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0238] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0239] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0240] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the electric vehicle scheduling method in the above embodiments.

[0241] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0242] The aforementioned computer-readable storage medium may be included in the electric vehicle dispatching equipment; or it may exist independently and not be installed in the electric vehicle dispatching equipment.

[0243] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the electric vehicle scheduling device, the electric vehicle scheduling device: acquires the current vehicle status information and the current status information of the battery swapping stations it passes through; inputs the current vehicle status information and the current status information of the battery swapping stations it passes through into a preset reinforcement learning model to obtain the vehicle target action, wherein the preset reinforcement learning model is obtained by training with vehicle sample status information, battery swapping station sample status information, vehicle sample joint actions, and vehicle sample joint rewards; and schedules the vehicle based on the vehicle target action.

[0244] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0245] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0246] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0247] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described electric vehicle scheduling method, thereby solving the technical problem of low vehicle scheduling efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the electric vehicle scheduling method provided in the above embodiments, and will not be repeated here.

[0248] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the electric vehicle scheduling method described above.

[0249] The computer program product provided in this application can solve the technical problem of low vehicle dispatching efficiency. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the electric vehicle dispatching method provided in the above embodiments, and will not be repeated here.

[0250] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An electric vehicle scheduling method, characterized in that, The electric vehicle scheduling method includes: Obtain the current status information of the electric vehicle and the current status information of the battery swapping stations it passes through; The current state information of the vehicle and the current state information of the battery swapping station along the route are input into a preset reinforcement learning model to obtain the target action of the vehicle. The preset reinforcement learning model is obtained by training the vehicle sample state information, the battery swapping station sample state information, the vehicle sample joint action, and the vehicle sample joint reward. Vehicles are scheduled based on the target vehicle actions.

2. The method as described in claim 1, characterized in that, The step of inputting the current state information of the vehicle and the current state information of the battery swapping stations along the route into a preset reinforcement learning model to obtain the target action of the vehicle includes: The current state information of the vehicle and the current state information of the battery swapping stations along the route are input into a preset reinforcement learning model for feature mapping to obtain the current joint state features. Calculate the action value of each vehicle action based on the current joint state characteristics; The target vehicle action is determined based on the action value.

3. The method as described in claim 2, characterized in that, The step of inputting the current state information of the vehicle and the current state information of the battery swapping stations along the route into a preset reinforcement learning model for feature mapping to obtain the current joint state features includes: The current state information of the vehicle and the current state information of the battery swapping stations along the route are input into a preset reinforcement learning model for structure extraction, so as to obtain the battery swapping station nodes, vehicle nodes, the edges between each battery swapping station along the route, and the edges between the vehicle and the battery swapping station. Obtain the state characteristic function; The current joint state features are obtained by performing feature mapping on the passing battery swapping station nodes, the vehicle nodes, the edges between each passing battery swapping station, and the edges between the vehicle and the passing battery swapping station using the state feature function.

4. The method as described in claim 3, characterized in that, The step of performing feature mapping on the passing battery swapping station nodes, the vehicle nodes, the edges between each passing battery swapping station, and the edges between the vehicle and the passing battery swapping station using the state feature function to obtain the current joint state features includes: Feature extraction is performed on the nodes of the battery swapping stations along the route to obtain a first feature vector representing the nodes of the battery swapping stations along the route. Feature extraction is performed on the vehicle node to obtain a second feature vector representing the vehicle node; Construct an adjacency matrix based on the passing battery swapping station nodes, the vehicle nodes, the edges between each passing battery swapping station, and the edges between the vehicle and the passing battery swapping station; The current joint state features are obtained by calculating the first feature vector, the second feature vector, and the adjacency matrix using the state feature function.

5. The method as described in claim 3, characterized in that, The method further includes: Obtain vehicle sample status information and battery swapping station sample status information; A heterogeneous graph is constructed based on the vehicle sample status information and the battery swapping station sample status information. The structure of the heterogeneous graph includes one or more of the following: battery swapping station nodes, vehicle nodes, edges between each battery swapping station, and edges between vehicles and battery swapping stations. An adjacency matrix is ​​constructed based on the heterogeneous graph, and features are extracted from the heterogeneous graph to obtain vehicle node feature vectors and battery swapping station node feature vectors. Obtain the first mapping function of the graph neural network; The state feature function of the graph neural network model is obtained by learning the state feature function through the first mapping function, the adjacency matrix, the vehicle node feature vector, and the battery swapping station node feature vector.

6. The method as described in claim 2, characterized in that, The step of calculating the action value of each vehicle action based on the current joint state characteristics includes: Obtain the vehicle joint reward under the current joint state characteristics; Each vehicle action is obtained through the current joint state characteristics and the vehicle joint reward; The action value of each action of the vehicle is calculated based on the current joint state characteristics and each action of the vehicle.

7. The method as described in claim 6, characterized in that, The step of calculating the action value of each vehicle action based on the current joint state characteristics and each vehicle action includes: Obtain the action value function; The action value function is used to calculate the action value of each action of the vehicle under the current joint state characteristics and each action of the vehicle.

8. The method as described in claim 7, characterized in that, The method further includes: Obtain the second mapping function of the value neural network model; Obtain the joint state features of the samples and the joint actions of the vehicle samples under the joint state features of the samples; By learning the action value function through the second mapping function, the sample joint state features, and the vehicle sample joint actions, the action value function of the value neural network model is obtained.

9. The method as described in claim 2, characterized in that, The step of determining the vehicle target action based on the action value includes: The vehicle's actions are sorted based on their action value to obtain a list of vehicle actions. Based on the action value, select a vehicle action from the vehicle action list that meets the preset action value to obtain the vehicle target action.

10. The method according to any one of claims 1 to 9, characterized in that, The step of scheduling vehicles based on the target vehicle action includes: A scheduling strategy is generated based on the vehicle's target action; Vehicles are scheduled using the aforementioned scheduling strategy.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: The current status information of the vehicle and the current status information of the battery swapping station along the route are updated according to the target action of the vehicle to obtain updated vehicle status information and updated status information of the battery swapping station along the route, and the process returns to the step of obtaining the current status information of the electric vehicle and the current status information of the battery swapping station along the route.

12. An electric vehicle dispatching device, characterized in that, The electric vehicle dispatching device includes: The acquisition module is used to acquire the current status information of the electric vehicle and the current status information of the battery swapping stations it passes through; The input module is used to input the current state information of the vehicle and the current state information of the battery swapping station along the route into a preset reinforcement learning model to obtain the target action of the vehicle. The preset reinforcement learning model is obtained by training the vehicle sample state information, the battery swapping station sample state information, the vehicle sample joint action, and the vehicle sample joint reward. The scheduling module is used to schedule vehicles based on the target vehicle actions.

13. An electric vehicle dispatching device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the electric vehicle scheduling method as described in any one of claims 1 to 11.

14. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the electric vehicle scheduling method as described in any one of claims 1 to 11.

Images