Deep reinforcement learning method and device for electric taxi dispatching based on fusion of double representation models, terminal and storage medium

Abstract

The application relates to the technical field of traffic scheduling. The application discloses a deep reinforcement learning electric taxi scheduling method and device fusing a double-characterization model, a terminal and a storage medium, which can improve the prediction accuracy of the model, improve the scheduling accuracy of the electric taxi and improve the reliability of the model prediction. The method comprises the following steps: acquiring a road network enhanced state and a historical behavior characterization of an electric taxi at a current time; splicing the road network enhanced state and the historical behavior characterization to form an enhanced state vector; inputting the enhanced state vector into a scheduling model for prediction processing and outputting an estimated Q value; generating a scheduling instruction based on the estimated Q value and sending the scheduling instruction to a terminal of the electric taxi.

Description

Technical Field

[0001] This application relates to the field of traffic dispatching technology. More specifically, this application relates to a deep reinforcement learning method, device, terminal, and storage medium for dispatching electric taxis that integrates a dual-representation model. Background Technology

[0002] Existing electric taxi dispatching methods based on deep reinforcement learning models, while superior to traditional rule-based or heuristic methods in handling dynamic environments, still suffer from significant limitations in state-space representation. This leads to inaccurate and unreliable model predictions (or decisions), resulting in imprecise dispatching of electric taxis. Specifically, in environmental state representation, existing methods typically locate vehicle positions using only discrete identifiers, without integrating road network data. This prevents the model from perceiving the complex differences in the road network environment, affecting the accuracy of model predictions and consequently leading to imprecise electric taxi dispatching. In behavioral state representation, existing methods rely solely on the current dispatching behavior of electric taxis for prediction, resulting in unreliable prediction results.

[0003] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0004] The purpose of this application is to provide a deep reinforcement learning method, device, terminal, and storage medium for dispatching electric taxis that integrates a dual-representation model. This method can improve the accuracy of model predictions, the precision of electric taxi dispatching, and the reliability of model predictions. This application is mainly achieved through the following technical solutions: A first aspect of this application provides a deep reinforcement learning-based electric taxi dispatching method that integrates a dual-representation model, comprising: Obtain the current road network enhancement status and historical behavior representation of electric taxis; The enhanced state of the road network and the historical behavior representation are concatenated to form an enhanced state vector; The enhanced state vector is input into the scheduling model for prediction processing, and the estimated Q value is output. A dispatch instruction is generated based on the estimated Q value, and the dispatch instruction is sent to the terminal of the electric taxi.

[0005] According to one embodiment of this application, the step of obtaining the road network enhancement status includes: Obtain the original state of the electric taxi and parse the original state to obtain the location identifier; The location identifier and the global road network node embedding table are indexed and matched to obtain a node dense vector. The enhanced road network state is generated based on the original state and the node density vector.

[0006] According to one embodiment of this application, the step of obtaining the global road network node embedding table includes: Obtain the city road network map corresponding to the electric taxi; The graph representation learning algorithm of the trained graph attention network is used to perform low-dimensional dense vector transformation on each node in the urban road network map to obtain the target low-dimensional dense vector corresponding to each node. The global road network node embedding table is constructed by using all target low-dimensional dense vectors.

[0007] According to one embodiment of this application, the step of generating the enhanced road network state based on the original state and the node density vector includes: The node density vector is inserted after the road segment number or intersection node number in the original state to form the enhanced road network state.

[0008] According to one embodiment of this application, the dispatching instruction is a passenger-carrying instruction, a waiting-for-passengers instruction, a charging instruction, or an empty-running cruise instruction.

[0009] According to one embodiment of this application, the step of obtaining the historical behavior representation includes: Extract feature sequences from multiple time steps from the cache; A temporal encoder is used to perform behavioral representation extraction processing on the feature sequence to obtain the historical behavioral representation.

[0010] According to one embodiment of this application, after the steps of generating a scheduling instruction based on the estimated Q value and sending the scheduling instruction to the terminal of the electric taxi, the deep reinforcement learning electric taxi scheduling method fusing the dual representation model further includes: The scheduling instructions are processed using one-hot encoding to obtain action codes; The action code and the enhanced state vector are concatenated into a comprehensive feature; The comprehensive features are stored in the cache.

[0011] A second aspect of this application provides a deep reinforcement learning electric taxi dispatching device that integrates a dual-representation model, comprising: The acquisition module is used to acquire the current road network enhancement status and historical behavior representation of electric taxis; The splicing module is used to splice the road network enhancement state and the historical behavior representation to form an enhancement state vector; The prediction module is used to input the enhanced state vector into the scheduling model for prediction processing and output the estimated Q value; The dispatch instruction generation module is used to generate dispatch instructions based on the estimated Q value and send the dispatch instructions to the terminal of the electric taxi.

[0012] A third aspect of this application provides a terminal device, including a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the steps of the deep reinforcement learning electric taxi scheduling method with fused dual representation models provided in the first aspect of this application.

[0013] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the steps of the deep reinforcement learning electric taxi dispatching method with fused dual representation models provided in the first aspect of this application.

[0014] The beneficial effects of the embodiments of this application include: This application embodiment acquires the road network enhancement state and historical behavior representation of electric taxis at the current moment; concatenates the road network enhancement state and the historical behavior representation to form an enhancement state vector; inputs the enhancement state vector into a scheduling model for prediction processing, outputting an estimated Q value; generates a scheduling instruction based on the estimated Q value, and sends the scheduling instruction to the terminal of the electric taxi. Compared with the prior art, this application embodiment introduces road network enhancement state (i.e., road network data) into the environmental state representation and historical behavior representation, i.e., the historical scheduling behavior of electric taxis, into the behavior state representation. This allows the scheduling model to perceive the complex differences in the road network environment, thereby improving the accuracy of model prediction and thus improving the scheduling precision of electric taxis; furthermore, the use of the historical scheduling behavior of electric taxis can improve the reliability of the scheduling model's prediction. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 The flowcharts for some embodiments of the deep reinforcement learning electric taxi dispatching method with fusion dual representation model of this application are shown below. Figure 2 This is a block diagram illustrating the principle of the deep reinforcement learning electric taxi dispatching device based on the fusion of dual representation models in some embodiments of this application. Detailed Implementation

[0017] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0018] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0019] The terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0020] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or apparatus.

[0021] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items.

[0022] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0023] refer to Figure 1 The diagram shown is a flowchart of a deep reinforcement learning electric taxi scheduling method that integrates a dual-representation model, provided in the first aspect of an embodiment of this application. Figure 1The deep reinforcement learning electric taxi scheduling method that integrates dual representation models includes the following steps S1, S2, S3 and S4.

[0024] S1. Obtain the current road network enhancement status and historical behavior representation of electric taxis.

[0025] Furthermore, the step of obtaining the road network enhancement state includes: obtaining the original state of the electric taxi, parsing the original state to obtain a location identifier; performing index matching processing on the location identifier and the global road network node embedding table to obtain a node dense vector; and generating the road network enhancement state based on the original state and the node dense vector.

[0026] The original state includes the current geographical location information of the electric taxi. The current geographical location information is either the road segment number where the electric taxi is located at the current moment, or the intersection node number closest to the electric taxi. In other embodiments, the original state may also include the remaining battery power and order revenue of the electric taxi, or other content, which can be specifically set by those skilled in the art according to actual needs.

[0027] Furthermore, the step of generating the enhanced road network state based on the original state and the node density vector includes: inserting the node density vector after the road segment number or intersection node number of the original state to form the enhanced road network state.

[0028] Further, the step of obtaining the global road network node embedding table includes: obtaining the city road network map corresponding to the electric taxi; performing low-dimensional dense vector transformation processing on each node in the city road network map using a trained graph attention network graph representation learning algorithm to obtain the target low-dimensional dense vector corresponding to each node; and constructing the global road network node embedding table from all the target low-dimensional dense vectors. Further, the step of obtaining the city road network map corresponding to the electric taxi includes: downloading road network data of the city where the electric taxi is located from an open map platform; performing graph transformation processing on the road network data to obtain the city road network map corresponding to the electric taxi.

[0029] The open map platform can be OSM (OpenStreetMap). In other embodiments, the open map platform can also be other platforms, which can be set by those skilled in the art according to actual needs.

[0030] The step of performing graph transformation processing on the road network data to obtain the urban road network map corresponding to the electric taxi can be implemented using the OSMnx (OpenStreetMap Network eXtension) tool. OSMnx is a Python toolkit for downloading, modeling, analyzing, and visualizing urban street networks and other geospatial data from OpenStreetMap.

[0031] The urban road network map consists of a set of nodes and a set of edges. The urban road network map can be represented as follows: ,in, It is the set of nodes, which corresponds to all intersections or road segments in the city; The set of edges corresponds to the connection relationships between nodes in the set of nodes. Each edge in the set has attributes including road segment length, speed limit, and number of lanes. In other embodiments, the attributes of each edge in the set may also include other content, which can be set by those skilled in the art according to actual needs.

[0032] In other implementations, other graph neural network architectures such as GCN (Graph Convolution Neural), GAT (Graph Attention Networks), or GraphSAGE (Graph Sample and Aggregate) can be used to replace the graph representation learning algorithm.

[0033] Furthermore, the step of using a trained graph attention network graph representation learning algorithm to perform low-dimensional dense vector transformation processing on each node in the urban road network graph to obtain the target low-dimensional dense vector corresponding to each node includes: obtaining the neighbor set of a first target node, where the first target node is any node in the node set of the urban road network graph; and using multiple graph attention layers of the graph representation learning algorithm to perform low-dimensional dense vector transformation processing on the first target node to obtain the target low-dimensional dense vector corresponding to the first target node. This embodiment of the application, through a multi-layer iterative calculation method, enables nodes to capture higher-order topological neighborhood information through recursive aggregation. For example, when the plurality of graph attention layers include a first graph attention layer, a second graph attention layer, and a third graph attention layer, the step of using the plurality of graph attention layers of the graph representation learning algorithm to perform low-dimensional dense vector transformation processing on the first target node to obtain the target low-dimensional dense vector corresponding to the first target node includes: using the first graph attention layer to perform low-dimensional dense vector transformation processing on the first target node to obtain a first target node feature corresponding to the first target node; using the second graph attention layer to perform low-dimensional dense vector transformation processing on the first target node feature to obtain a second target node feature corresponding to the first target node; using the third graph attention layer to perform low-dimensional dense vector transformation processing on the second target node feature to obtain a third target node feature corresponding to the first target node; and using the third target node feature as the target low-dimensional dense vector corresponding to the first target node.

[0034] In other embodiments, the number of graph attention layers among the plurality of graph attention layers can be set by those skilled in the art according to actual needs.

[0035] Further, the step of using the first graph attention layer to perform low-dimensional dense vector transformation on the first target node to obtain the first target node features includes: performing correlation calculation on the initial feature vector corresponding to the first target node and the initial feature vector corresponding to the second target node to obtain the original correlation coefficients corresponding to the first target node and the second target node, where the second target node is any node in the neighbor set; normalizing the original correlation coefficients to obtain the attention weights corresponding to the first target node and the second target node; and calculating the attention weights and the initial feature vector corresponding to the second target node to obtain the first target node features. Further, the calculation formula for the step of performing correlation calculation on the initial feature vector corresponding to the first target node and the initial feature vector corresponding to the second target node to obtain the original correlation coefficients corresponding to the first target node and the second target node is as follows: ; in, It is the original correlation coefficient between the first target node and the second target node, where the first target node is the first node in the node set of the urban road network map. The first node is the set of nodes, and the second target node is the neighbor set. The One node; It is the first activation function. The full English name is Leaky Rectified Linear Unit. These are learnable in-layer vectors; Represents transposition; It is a learnable weight matrix; It is the initial feature vector corresponding to the first target node; It is the initial feature vector corresponding to the second target node; This indicates a vector concatenation operation.

[0036] Furthermore, the step of obtaining the initial feature vector corresponding to the first target node includes: encoding the original attributes of the first target node into the initial feature vector corresponding to the first target node.

[0037] The original attributes of the first target node include geographic coordinates. In other embodiments, the original attributes of the first target node may also include other content, which can be set by those skilled in the art according to actual needs.

[0038] Furthermore, the steps for obtaining the initial feature vector corresponding to the second target node are the same as those for obtaining the initial feature vector corresponding to the first target node, and will not be repeated here.

[0039] Furthermore, the formula for calculating the attention weights corresponding to the first target node and the second target node by normalizing the original correlation coefficients is as follows: ; in, These are the attention weights corresponding to the first target node and the second target node; It is an exponential function; It is the first target node and the neighbor set. The Middle The original correlation coefficients corresponding to each node; The neighbor set The Each node.

[0040] In other implementations, the second preset algorithm can be understood as the Softmax function.

[0041] Furthermore, the calculation formula for the step of calculating and processing the attention weights and the initial feature vector corresponding to the second target node to obtain the first target node feature corresponding to the first target node is as follows: ; in, It is the target node feature corresponding to the first target node; It is the second activation function.

[0042] Furthermore, the implementation process of the step "using the second graph attention layer to perform low-dimensional dense vector transformation on the features of the first target node to obtain the second target node features corresponding to the first target node" is the same as the implementation process of the step "using the first graph attention layer to perform low-dimensional dense vector transformation on the first target node to obtain the first target node features corresponding to the first target node". The only difference is that "the first target node features corresponding to the first target node" is replaced with "the initial feature vector corresponding to the first target node" in the implementation process.

[0043] Furthermore, the implementation process of the step "using the third graph attention layer to perform low-dimensional dense vector transformation on the features of the second target node to obtain the third target node features corresponding to the first target node" is the same as the implementation process of the step "using the first graph attention layer to perform low-dimensional dense vector transformation on the first target node to obtain the first target node features corresponding to the first target node". The only difference is that "the second target node features corresponding to the first target node" is used instead of "the initial feature vector corresponding to the first target node" in the implementation process.

[0044] Furthermore, the step of obtaining the neighbor set of the first target node includes: establishing an adjacency table of the first target node based on the edge set of the city road network map; and generating a neighbor set corresponding to the first target node based on the adjacency table of the first target node.

[0045] Furthermore, the step of obtaining the historical behavior representation includes: extracting feature sequences from multiple time steps in the cache; and using a time encoder to perform behavior representation extraction processing on the feature sequences to obtain the historical behavior representation.

[0046] Furthermore, the step of extracting feature sequences from multiple time steps from the cache specifically involves retrieving the feature sequences of the K most recent time steps from the cache. The value of K can be set by those skilled in the art according to actual needs.

[0047] Furthermore, the step of using a temporal encoder to extract behavioral representations from the feature sequence to obtain the historical behavioral representations includes: the temporal encoder iteratively processes the feature sequence step by step, captures the temporal dependencies of the feature sequence, and outputs the hidden state at each time step; all hidden states are used as the historical behavioral representations.

[0048] In other implementations, the timing encoder can be replaced by a model such as a Transformer, an attention mechanism, or a spatiotemporal graph neural network.

[0049] The historical behavior representation can be the historical dispatching behavior of the electric taxi.

[0050] S2. The enhanced state of the road network and the historical behavior representation are concatenated to form an enhanced state vector.

[0051] S3. Input the enhanced state vector into the scheduling model for prediction processing and output the estimated Q value.

[0052] The scheduling model adopts The greedy strategy selects the optimal action (i.e., selects the optimal predicted Q value).

[0053] The scheduling model is a pre-trained dual-depth Q-network. In other embodiments, the scheduling model can also be a pre-trained reinforcement learning algorithm such as PPO (Proximal Policy Optimization) or SAC (Soft Actor-Critic), which can be set by those skilled in the art according to actual needs.

[0054] S4. Generate a dispatch instruction based on the estimated Q value, and send the dispatch instruction to the terminal of the electric taxi.

[0055] The dispatching instructions are passenger-carrying instructions, stationary waiting instructions, charging instructions, or empty-running cruise instructions. The action corresponding to the passenger-carrying instruction is to carry passengers; the action corresponding to the stationary waiting instructions is to wait for passengers in place; the action corresponding to the charging instructions is to charge; and the action corresponding to the empty-running cruise instructions is to cruise empty.

[0056] The terminal of the electric taxi can execute the dispatch instructions to achieve the purposes of picking up passengers, waiting for passengers in place, charging, or cruising empty. The terminal of the electric taxi can also collect its own current geographical location information, remaining battery power, order revenue, and other raw status data.

[0057] This application's embodiments introduce enhanced road network status (i.e., road network data) into the environmental status representation and historical behavior representation, i.e., the historical dispatch behavior of electric taxis, into the behavioral status representation. This enables the dispatch model to perceive the complex differences in the road network environment, thereby improving the accuracy of model predictions and thus improving the dispatch accuracy of electric taxis. Furthermore, the use of the historical dispatch behavior of electric taxis can improve the reliability of the dispatch model's predictions.

[0058] In some implementations, after generating a scheduling instruction based on the estimated Q value and sending the scheduling instruction to the terminal of the electric taxi, the deep reinforcement learning electric taxi scheduling method that integrates a dual representation model further includes: performing one-hot encoding on the scheduling instruction to obtain an action code; concatenating the action code and the enhanced state vector into a comprehensive feature; and storing the comprehensive feature in a cache.

[0059] In some implementations, the deep reinforcement learning electric taxi scheduling method that integrates dual representation models further includes a training step of a graph representation learning algorithm for a trained graph attention network.

[0060] The goal of the training steps of the graph representation learning algorithm for the trained graph attention network is to maximize the topological similarity and semantic consistency between nodes.

[0061] Furthermore, the training steps of the graph representation learning algorithm for the trained graph attention network include: obtaining a first training set; obtaining the neighbor set of each first training node in the first training set; performing low-dimensional dense vector transformation on each first training node using multiple initial graph attention layers of the graph representation learning algorithm for the initial graph attention network to obtain a low-dimensional dense training vector corresponding to each first training node; constructing a first loss function based on the low-dimensional dense training vector corresponding to each first training node; and adjusting the parameters of the multiple initial graph attention layers based on the first loss function to form the graph representation learning algorithm for the trained graph attention network.

[0062] Furthermore, the formula for calculating the first loss function is as follows: ; in, It is the first loss function; It is the first training set The low-dimensional dense training vector of the first training node (This can be understood as the transpose of the embedding vector); the first training set of the first... The first training node can be understood as the central node. ; yes Real neighbor nodes Low-dimensional dense training vectors; Forming positive sample pairs; This indicates the distribution from the noise distribution (i.e., the negative sampling probability distribution). Sampled from ) The mathematical expectation; It is a negative sample node Low-dimensional dense training vectors; Through negative sampling probability distribution Randomly selected non-neighbor nodes; Forming negative sample pairs; It represents the total number of negative samples.

[0063] This application embodiment minimizes the first loss function, causing node vectors with physical connections in the urban road network map to move closer to each other, while randomly sampled unconnected node vectors move further apart.

[0064] In some implementations, embodiments of this application may also utilize a multi-head feature coding layer to nonlinearly map elements with different attributes such as time, electricity consumption, and revenue in the enhanced road network state to a high-dimensional feature space, thereby generating the enhanced state vector.

[0065] In some implementations, the scheduling model is trained using a primitive dual-depth Q-network. The primitive dual-depth Q-network includes a main network, a target network, and a hierarchical experience replay pool. In this embodiment, the primitive dual-depth Q-network is trained by constructing a multi-objective collaborative reward function and utilizing a hierarchical experience replay mechanism.

[0066] In some implementations, to balance immediate revenue and long-term operational benefits, a first reward is designed for passenger-carrying scenarios. The formula for calculating the first reward is as follows: ; in, It is the first reward; It is the first empirical adjustment parameter; This is the revenue from the current orders of the electric taxi.

[0067] In some implementations, the three actions of empty cruising, charging, and waiting for passengers are all combined with the unit electricity cost of the electric taxi and the fixed standby energy consumption or charging amount to design a second reward. The calculation formula for the second reward is as follows: ; in, It is the second reward; It is the second empirical adjustment parameter; This refers to the unit electricity cost of the electric taxi; It refers to the fixed standby power consumption or charging amount.

[0068] It should be understood that the rewards (which can be understood as reward functions) used in this application can be adjusted according to actual operational needs by incorporating node popularity or multi-objective optimization ideas.

[0069] In some implementations, the original is based on a dual-depth Q-network to start from time 1. Until the end time The objective is to achieve a cumulative discount return, and the formula for calculating the cumulative discount return is as follows: ; in, This is the cumulative reward for the aforementioned discount; It is a discount factor. ; In time step The instant reward value obtained.

[0070] In some implementations, the training process of the scheduling model can also be implemented in the following manner: Obtain the first network augmentation training state and the first historical behavior training representation at the current moment, and concatenate the first network augmentation training state and the first historical behavior training representation to form the first augmentation state vector corresponding to the current moment; obtain the second network augmentation training state and the second historical behavior training representation at the next moment, and concatenate the second network augmentation training state and the second historical behavior training representation to form the second augmentation state vector corresponding to the next moment; combine the first augmentation state vector, the second augmentation state vector, the scheduling training action executed at the current moment, and the immediate reward. Complete experience tuples are constructed and stored in the hierarchical experience replay pool.

[0071] When the experience replay pool reaches the sampling condition, batches of experience data are randomly selected for learning. First, the optimal action is selected in the action space through the original main network based on a dual-depth Q-network. The original target network based on a dual-depth Q-network is then used to evaluate the predicted Q-value of the optimal action, and the target Q-value is calculated by combining it with the immediate reward. The formula for calculating the target Q-value is: ; in, It is the target Q value; It is a discount factor; It is the estimated Q value output by the target network; It is the second enhanced state vector; It is the estimated Q value output by the main network; These are the parameters of the main network; It is the main network in Next, all possible actions From the calculated Q values, select the action corresponding to the largest Q value; These are the parameters of the target network.

[0072] Next, a second loss function is constructed based on the target Q value; and the parameters of the original dual-depth Q-network are periodically updated based on the second loss function to form the scheduling model.

[0073] The formula for calculating the second loss function is: ; in, It is the second loss function; It is the expectation operator; The main network pair and The current estimate; It is the first enhanced state vector; Is The actual actions to be performed.

[0074] Furthermore, the dispatching training actions include carrying passengers, waiting for passengers in place, charging, or cruising empty.

[0075] In some implementations, the steps for obtaining the first road network augmentation training state and the second road network augmentation training state are the same as the steps for obtaining the target low-dimensional dense vector corresponding to each node in the urban road network map.

[0076] In some implementations, the steps for obtaining the first historical behavior training representation and the second historical behavior training representation are the same as the steps for obtaining the historical behavior representation.

[0077] In some implementations, embodiments of this application periodically update the parameters of the original dual-depth Q-network by minimizing the second loss function until the network converges.

[0078] In some implementations, the experience playback strategy used in this application may be a priority experience playback strategy or a time-series experience playback strategy.

[0079] refer to Figure 2 The diagram shown is a principle block diagram of a deep reinforcement learning electric taxi dispatching device that integrates a dual-representation model, provided in the second aspect of an embodiment of this application. Figure 2 The deep reinforcement learning electric taxi dispatching device 100 that integrates dual representation models includes: The acquisition module 101 is used to acquire the road network enhancement status and historical behavior representation of electric taxis at the current moment; The splicing module 102 is used to splice the road network enhancement state and the historical behavior representation to form an enhancement state vector; Prediction module 103 is used to input the enhanced state vector into the scheduling model for prediction processing and output the estimated Q value; The dispatch instruction generation module 104 is used to generate dispatch instructions based on the estimated Q value and send the dispatch instructions to the terminal of the electric taxi.

[0080] A third aspect of this application provides a terminal device, which includes a processor and a memory. The memory stores a computer program, and the processor calls and runs the computer program stored in the memory to execute the steps of the deep reinforcement learning electric taxi dispatching method with fused dual representation models provided in the first aspect of this application.

[0081] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the steps of the deep reinforcement learning electric taxi dispatching method with fused dual representation models provided in the first aspect of this application.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] The technical features of the above embodiments can be combined without changing the basic principles of this application. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the patent protection scope of this application should be determined by the appended claims.

Claims

1. A deep reinforcement learning-based electric taxi scheduling method integrating a dual-representation model, characterized in that, include: Obtain the current road network enhancement status and historical behavior representation of electric taxis; The enhanced state of the road network and the historical behavior representation are concatenated to form an enhanced state vector; The enhanced state vector is input into the scheduling model for prediction processing, and the estimated Q value is output. A dispatch instruction is generated based on the estimated Q value, and the dispatch instruction is sent to the terminal of the electric taxi; The steps for obtaining the enhanced road network state include: obtaining the original state of the electric taxi and parsing the original state to obtain a location identifier; performing index matching processing on the location identifier and the global road network node embedding table to obtain a node dense vector; and generating the enhanced road network state based on the original state and the node dense vector. The steps for obtaining the global road network node embedding table include: obtaining the urban road network map corresponding to the electric taxi; using a trained graph attention network graph representation learning algorithm to perform low-dimensional dense vector transformation on each node in the urban road network map to obtain the target low-dimensional dense vector corresponding to each node; and constructing the global road network node embedding table from all the target low-dimensional dense vectors.

2. The deep reinforcement learning electric taxi scheduling method based on a dual-representation model according to claim 1, characterized in that, The steps for generating the enhanced road network state based on the original state and the node density vector include: The node density vector is inserted after the road segment number or intersection node number in the original state to form the enhanced road network state.

3. The deep reinforcement learning electric taxi scheduling method based on a dual-representation model according to claim 1, characterized in that, The dispatching instructions can be passenger-carrying instructions, in-place passenger-waiting instructions, charging instructions, or empty-running cruise instructions.

4. The deep reinforcement learning electric taxi scheduling method based on the fusion of dual representation models according to claim 1, characterized in that, The steps for obtaining the historical behavior representation include: Extract feature sequences from multiple time steps from the cache; A temporal encoder is used to perform behavioral representation extraction processing on the feature sequence to obtain the historical behavioral representation.

5. The deep reinforcement learning electric taxi scheduling method based on the fusion of dual representation models according to claim 1, characterized in that, After generating a scheduling instruction based on the estimated Q-value and sending the scheduling instruction to the terminal of the electric taxi, the deep reinforcement learning electric taxi scheduling method that integrates a dual representation model further includes: The scheduling instructions are processed using one-hot encoding to obtain action codes; The action code and the enhanced state vector are concatenated into a comprehensive feature; The comprehensive features are stored in the cache.

6. A deep reinforcement learning-based electric taxi dispatching device integrating a dual-representation model, characterized in that, include: The acquisition module is used to acquire the current road network enhancement status and historical behavior representation of electric taxis; The splicing module is used to splice the road network enhancement state and the historical behavior representation to form an enhancement state vector; The prediction module is used to input the enhanced state vector into the scheduling model for prediction processing and output the estimated Q value; The dispatch instruction generation module is used to generate dispatch instructions based on the estimated Q value and send the dispatch instructions to the terminal of the electric taxi. The deep reinforcement learning electric taxi dispatching device that integrates dual representation models is also used to obtain the original state of the electric taxi, and to parse the original state to obtain a location identifier; to perform index matching processing on the location identifier and the global road network node embedding table to obtain a node dense vector; and to generate the road network enhanced state based on the original state and the node dense vector. The deep reinforcement learning electric taxi dispatching device that integrates dual representation models is also used to obtain the urban road network map corresponding to the electric taxi; the graph representation learning algorithm of the trained graph attention network is used to perform low-dimensional dense vector transformation on each node in the urban road network map to obtain the target low-dimensional dense vector corresponding to each node; and all the target low-dimensional dense vectors are used to form the global road network node embedding table.

7. A terminal device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory, to perform the steps of the deep reinforcement learning electric taxi dispatching method with fused dual representation models as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the steps of the deep reinforcement learning electric taxi dispatching method based on the fusion dual representation model as described in any one of claims 1 to 5.

Images