Dynamic carpooling scheduling method for unmanned electric taxis

Abstract

The invention discloses a dynamic carpooling scheduling method for unmanned electric taxis. The method comprises the following steps: firstly, acquiring data and cleaning the data; mapping the data in each hexagonal grid; establishing a travel demand feature matrix and a spatial dependency matrix; constructing a space-time diagram convolutional neural network based on an attention mechanism, and obtaining a travel demand prediction result; setting constraint conditions matched with the car orders in the dynamic car-pooling scene; based on the travel demand prediction result and the vehicle order matching constraint condition, constructing a vehicle order matching decision model based on distributed deep reinforcement learning; and designing a Double DQN solving algorithm, carrying out real-time order matching and idle vehicle scheduling based on a result, notifying the matched vehicle to complete a corresponding order, and notifying the unmatched idle vehicle to carry out regional scheduling or charging energy supplementation. According to the method, in dynamic carpooling of the multi-passenger bicycle, the endurance process constraint of the electric automobile and the charging energy complementation requirement of the bicycle are considered, the actual operation requirement is better met, and the quality is improved.

Description

Technical Field

[0001] The present invention relates to unmanned driving technology, and in particular to a dynamic carpooling scheduling method for unmanned electric taxis. Background Art

[0002] With the rapid development of the sharing economy, autonomous driving technology and the electric vehicle industry, driverless electric shared travel is expected to become an important part of the future urban transportation system. Compared with other shared travel modes, the dynamic carpooling mode of multiple passengers per vehicle has outstanding advantages in reducing passenger travel costs, improving vehicle utilization and alleviating traffic pressure, and has development potential. However, the existing shared electric vehicle fleet vehicle-order matching decision methods still have shortcomings such as short-sighted strategies, high computational complexity and poor algorithm adaptability. Therefore, it is necessary to develop a dynamic carpooling scheduling method that considers the characteristics of unmanned electric shared travel operation scenarios, integrates passenger travel demand prediction and fleet scheduling tasks, and aims to maximize the long-term benefits of fleet operations.

[0003] The search found:

[0004] The Chinese patent application with application number 202410994378.1, "Intercity Dynamic Carpooling Scheduling Optimization Method and Device Based on Two-stage Scheduling", constructs a two-stage scheduling model by obtaining carpooling order information and vehicle information. In the first stage, online vehicle scheduling and offline vehicle scheduling are performed based on the insertion algorithm. In the second stage, the vehicle orders of all unsent vehicles are matched based on the large neighborhood search algorithm to reduce the total operating time and maximize the total profit. Disadvantages: Two-stage scheduling requires decision-making and adjustments at multiple time points, which may introduce uncertainty and volatility, and the system is not scalable enough for large-scale intercity carpooling demand scenarios.

[0005] The Chinese patent application numbered 202311306118.2, "A real-time cross-platform carpooling route planning method", proposes an improved dynamic programming insertion method to plan new routes for workers on different crowdsourcing platforms, which can achieve the optimal allocation of tasks on multiple different platforms and the optimization of worker user routes. Disadvantages: The vehicle range constraint is not considered, and the vehicle refueling demand is not included in the setting of the operation management scenario, which affects the actual scheduling effect.

[0006] The Chinese patent application numbered 202310811799.1, "A point-to-point carpooling dynamic scheduling method and device for online car-hailing", uses cluster analysis to form point-to-point combined orders with close departure and destination points, and achieves the optimal matching solution of the vehicle order by setting the planning cycle, multi-objective function and constraint condition set to effectively balance the interests of operators and passengers. Disadvantages: It does not consider the future user's riding demand, and it is impossible to schedule and deploy idle vehicles in advance. In addition, it does not consider the long-term rewards of vehicle order matching, and the vehicle order matching decision is short-sighted.

[0007] The Chinese patent application "A Vehicle Dispatch Management System Based on Heuristic Algorithm" with application number 202211738830.5 proposes a vehicle dispatch management system including a data acquisition module, a dispatch optimization calculation module and a vehicle dispatch module, which can select the taboo search algorithm, simulated annealing algorithm or ant colony search algorithm to obtain the optimal solution for vehicle dispatch based on the comparison result of the total amount of vehicle data and the preset threshold. Disadvantages: Very sensitive to the initial condition setting, difficult algorithm tuning, and insufficient scalability.

[0008] The Chinese patent application with application number 202111433060.9, "A method and system for real-time order matching and idle vehicle scheduling for online ride-hailing", proposes a method and system for evaluating the vehicle value function based on a value iteration algorithm based on dynamic programming, and then performing real-time order matching and idle vehicle scheduling, which achieves the effective use of idle vehicles and maximizes long-term social welfare. Disadvantages: The idle vehicle regional scheduling is not carried out in advance based on the user travel demand prediction results, which affects the efficiency of vehicle order matching.

[0009] The Chinese patent application with application number 202110477323.X, "A method and system for matching online ride-hailing orders based on vehicle conditions", matches order information with the vehicle's navigation route to obtain candidate vehicles, and establishes a driving prediction model based on navigation route environmental information such as weather, road surface, and traffic to obtain tire safety parameters, and then compares the tire condition of the candidate vehicle with the tire safety parameters to determine whether the candidate vehicle meets the vehicle order matching requirements. Disadvantages: Additional road condition information is required, including road surface flatness, number of road surface cracks, etc., which is difficult to obtain accurately and timely in actual large-scale deployment.

[0010] The Chinese patent application numbered 202110203068.X, "Vehicle Scheduling Method Based on Dynamic Carpooling and Related Products", proposes a method and product that calculates the correlation between multiple vehicles and the target user's carpooling needs and the carpooling fees of each vehicle selected by the target user based on the user's carpooling information and vehicle information, so as to improve the accuracy of pushing target vehicles to the target user. Disadvantages: Only the current order demand is considered, and long-term rewards are not considered, and the vehicle scheduling decision is relatively short-sighted.

[0011] In summary, the main deficiencies of the prior art are:

[0012] 1. Existing methods do not fully consider the spatiotemporal characteristics of user travel demand, and do not make vehicle dispatch deployment in advance based on travel demand forecast results, which affects the efficiency of vehicle-order matching;

[0013] 2. Some vehicle-order matching decision methods do not consider the long-term benefits after vehicle-order matching, and the vehicle carpooling scheduling decision is relatively short-sighted;

[0014] 3. Some methods do not take into account the vehicle's range constraints and the vehicle's energy replenishment needs in actual operating scenarios, resulting in a poor user experience of actual carpooling services. Summary of the invention

[0015] Purpose of the invention: In order to achieve efficient decision-making in matching unmanned electric shared travel vehicles, the present invention provides a dynamic carpooling scheduling method for unmanned electric taxis.

[0016] Technical solution: On one hand, the present invention provides a dynamic carpooling scheduling method for driverless electric taxis, comprising the following steps:

[0017] (1) Obtain the user's historical travel orders, carpooling orders, and vehicle status and location information data;

[0018] (2) Clean the data of users’ historical travel orders;

[0019] (3) Divide the operating area into regular hexagonal grids and map the cleaned data into each hexagonal grid to reflect the differences in the spatiotemporal distribution of users' carpooling travel needs;

[0020] (4) Based on the data cleaning and travel demand spatiotemporal distribution results in step (3), a travel demand characteristic matrix and a spatial dependence matrix are established to quantify the travel time characteristics and spatial characteristics of passengers, respectively;

[0021] (5) Construct a spatiotemporal graph convolutional neural network based on the attention mechanism to obtain travel demand prediction results;

[0022] (6) Setting constraints for matching rides in dynamic carpooling scenarios, including capacity constraints, distance constraints, time constraints, mileage constraints, and sequence constraints;

[0023] (7) Based on the travel demand prediction results in step (5) and the vehicle-order matching constraints in step (6), a vehicle-order matching decision model based on distributed deep reinforcement learning is constructed;

[0024] (8) Design a Double DQN solution algorithm, use the Double DQN network to dynamically generate optimization strategies, define the vehicle's optimal action value function as the maximum expected achievable reward, and use the action that maximizes the neural network output value for each vehicle. The learning process starts from scratch and uses the Epsilon-Greedy method to select actions using a greedy scheme. Under this strategy, the agent selects the action that results in the highest Q value with probability 1-ε, otherwise it selects a random action. For the nth vehicle, after selecting an action, it is given a reward r. t , n, update the Q value with learning rate a;

[0025] (9) Based on the model solution results in step (8), real-time order matching and idle vehicle scheduling are performed, matching vehicles are notified to complete corresponding orders, and unmatched idle vehicles are notified to perform regional scheduling or charging and energy replenishment.

[0026] Furthermore, the data in step (1) includes the starting point and end point of the passenger trip, the number of passengers and the departure time, the remaining battery power and driving range of the vehicle, the vehicle location, and the number of passengers that the vehicle can carry.

[0027] Furthermore, the data cleaning in step (2) includes: converting longitude and latitude into WGS84 coordinate system, converting timestamp into date format, deleting data outside the study area, and clearing duplicate orders.

[0028] Furthermore, the step (3) comprises:

[0029] 3.1 Use the H3 spatial index system to divide the operation area into spatial grids, and use the H3 hexagonal grid with a resolution of 8 as the minimum action unit;

[0030] 3.2 Through the H3 spatial index system, the carpooling demand, available passenger vehicle supply, and supply-demand difference values ​​cleaned in step (2) are mapped to each hexagonal grid to reflect the differences in the spatiotemporal distribution of users' carpooling demand.

[0031] Furthermore, the step (4) comprises:

[0032] 4.1 In the spatial dimension, the Granger causality test is used to characterize the spatial dependence characteristics of passenger travel between regions, and a spatial dependence matrix A is established. The elements in the matrix only contain 0 and 1, representing whether there is a dependence on travel demand between two regions: if there is no dependence, it is recorded as 0; if there is a dependence, it is recorded as 1, as follows,

[0033]

[0034] In the formula, A adj; i and j are the divided areas; Lj is the arrival demand time series of area j; Li is the travel demand series of area i;

[0035] 4.2 In the time dimension, passenger travel demand data has a natural periodicity. In addition, the travel demand near the prediction time will also affect the prediction of the target time series. The travel demand data of the previous moment of the target prediction time, the travel demand data of the corresponding moment of the previous day, and the travel demand data of the corresponding moment of the previous week are extracted to construct the travel demand feature matrix X, and then construct a time series segment containing adjacent time, daily cycle and weekly cycle.

[0036] Furthermore, the step (5) comprises:

[0037] 5.1 Construct a spatiotemporal graph convolutional neural network based on the attention mechanism, input the time series fragments in step 4, together with the spatial dependency matrix, into the corresponding dependency analysis component to generate the normalized spatial attention matrix E′ and temporal attention matrix S′;

[0038] 5.2 Temporal Attention Mechanism: First, a time dimension convolution module is constructed to operate along the time axis of the data to extract important features in the time series, thereby capturing the dynamic patterns and correlations of passenger travel demand over time. Through the one-dimensional convolution operation in the time dimension, the model updates the passenger travel demand in real time along the time axis according to the size of the convolution kernel, capturing the correlation between each moment, as follows:

[0039]

[0040] In the formula, ReLU is the activation function, Φ is the convolution and parameter in the time dimension,

[0041] Then, we construct a time dimension attention module. Through the self-attention mechanism, the model can automatically identify and extract the key dependency features in the travel demand data, and dynamically adjust the correlation strength of travel demand in the time dimension according to the specific situation of the input data, as follows:

[0042]

[0043] Where V i 、b i , Z1, Z2, and Z3 are parameters learned from the input data; E is the time correlation matrix automatically learned by the attention module of the time dimension according to the input data at the current moment; S′ is the time attention matrix obtained by normalizing the E matrix using the Softmax function; S m，n is the correlation between nodes m and n,

[0044] Furthermore, by dynamically adjusting the input data, its features can be extracted more deeply, and the normalized time attention matrix Applied to the current input data, we get Using the adjusted data, the features for the next stage can be further extracted;

[0045] 5.3 Spatial Attention Mechanism: First, we build a spatial dimension graph convolution module, use spectral graph theory to interpret the data as a graph structure, use convolution technology to process these graph data, and extract spatial features from the graph information through spatial dimension ChebNet graph convolution analysis, thereby revealing the correlation of passengers' travel needs at the spatial level. The details are as follows:

[0046]

[0047] In the formula, is the graph signal matrix feature of the passenger travel data at the t-th time in the M-th area; is the parameter to be learned; is the normalized Laplace matrix,

[0048] Then, we construct a spatial dimension attention module and use the self-attention mechanism to capture the dynamic changes of inter-region dependencies in the spatial dimension and automatically adjust the correlation between passenger travel demand data to adapt to these changes, as follows:

[0049]

[0050] In the formula, is the input data of the k-th spatiotemporal module; C k-1 The number of channels for input data to the k-th spatiotemporal module; T k-1 V is the time dimension length of the input data of the k-th spatiotemporal module; j 、b j , Y1, Y2, and Y3 are parameter sets learned through the attention mechanism; E is the spatial attention matrix learned based on the current input data; E m，n is the strength of the interaction between nodes m and n; E′ is the normalized spatial attention matrix generated by standardizing E using the Softmax function. Each element value in the matrix represents the weight coefficient between regions.

[0051] Furthermore, the spatial attention matrix E′ and the matrix of spatial dependency in the Chebyshev polynomial in the spatial graph convolution are dynamically adjusted to achieve a reasonable distribution of attention between regions by influencing the weights;

[0052] 5.4 By stacking spatiotemporal modules and applying a residual learning strategy for training, the outputs of the proximity dependency, daily cycle, and weekly cycle components are adjusted using fully connected layers to match the dimensions of the prediction target, and then these outputs are fused to obtain the final travel demand forecast.

[0053] Furthermore, the step (6) comprises:

[0054] 6.1 The mathematical expression of capacity constraint is as follows,

[0055] N r ≤C max -n c

[0056] Where N r is the number of passengers included in the new order r, which must not exceed the current remaining available capacity of the vehicle; n c is the number of passengers on board; C max is the total number of seats in the vehicle, which is set to 4;

[0057] 6.2 The mathematical expression of the order acceptance distance constraint is as follows:

[0058] d r，n (r loc , O r )≤ω

[0059] t r，n (n loc , O r )≤t q -t x

[0060] In the formula, O r is the pickup location of the new order r; n loc is the current position of vehicle n; d r，n (n loc , O r ) is the vehicle n from the current position n loc Go to the order pick-up location O r The order receiving distance; t r，n (n loc , O r ) is the order acceptance time required according to the current route; t x is the order response time; t q is the maximum time a user can wait, set to 10 minutes; ω is 3 km,

[0061] 6.3 The mathematical expression of the detour distance constraint is as follows:

[0062]

[0063] Where t is the time that has passed since the passenger trip l requested the ride; is the new estimated travel time due to detour or adding a new carpooling passenger to the passenger trip starting from time t to the end; is the travel time if the vehicle only serves passenger trip l; ρ is a constant, set to 0.5,

[0064] 6.4 The mathematical expression of the cruising range constraint is as follows:

[0065] r′ n -l r ≤range0

[0066] In the formula, range0 is the minimum cruising range; l r is the updated driving route mileage after order r is added, which is set to 100 km; r′ n is the cruising range of the current vehicle n,

[0067] 6.5 The mathematical expression of the order constraint is as follows,

[0068]

[0069] In the formula, Pick-up location for passengers O r order in the set of vehicle destinations; Passenger drop-off point D r The order in the vehicle's destination set.

[0070] Furthermore, the step (7) comprises:

[0071] 7.1 Model the real-time order matching and vehicle dispatching process as a Markov decision process, and define the states, actions, rewards, and state transition probabilities involved in the Markov decision process;

[0072] Define a triple to capture the system state update at time t: (X t , V t：t+T , D t：t+T ), and combine the triplet state variables in a vector s t Among them, X t ={x t，1 , x t，2 , ..., x t,N} is the current state of all N vehicles at time t, V t：t+T is the number of vehicles in each area in the next T time periods, D t：t+T Forecast future travel demand for each region from time t to time t+T;

[0073] Define the action a of vehicle nt,n It consists of two parts: the first part is that if the vehicle is partially full, it decides whether to serve existing passengers or new passengers; the second part is that if there are no passengers on the vehicle or it decides to serve new passengers, it needs to decide the area that the vehicle should go to at time slot t, using u t，n，i If the vehicle decides to provide service to a new user and decides to go to area i, then u t，n，i =1, otherwise 0; if vehicle n has no vacancies, it cannot serve any other passengers;

[0074] Define the reward r of vehicle n at time slot t t，n as follows,

[0075]

[0076] In the formula, β i is the weight of each part in the reward expression; b t，n The number of orders served by vehicle n at time t; c t，n is the time it takes for vehicle n to take a detour to pick up an additional passenger if there are available seats; δ t，n，l The additional time that vehicle n spends due to carpooling compared to directly serving passenger trip l; e t，n is the vehicle occupancy status, which is set to 1 if the idle vehicle n is occupied, and 0 if a vehicle with passengers accepts a new customer;

[0077] Define the state transition probability matrix as P(s t+1 |s t , a t，n ), represents the probabilistic dependency between action and reward functions. This patent uses a neural network to determine the transition probability and reward distribution, and is used to find the state action value Q value;

[0078] 7.2 Designing decision variables and objective functions for multi-agents,

[0079] At each time step t, the model algorithm obtains the state s of the environment t and the reward r given by the environment t , based on the state of the environment and the reward, the model takes action a i To guide the idle vehicles to different locations to maximize the expected future long-term rewards, the agent has two decision variables in this process. The first decision variable is At time t, the agent decides to dispatch vehicle n to area j, then Otherwise, it is 0. The second decision variable is ζ n，tIf the vehicle n is not full, the agent needs to decide whether the vehicle can provide services to the new customer in time period t. If it decides to provide services to the customer, then n，t =1, otherwise ζ n， t=0, thus, the objective function of the multi-agent at time t is as follows:

[0080]

[0081] In the formula, β1, β2, β3, and β4 are weight factors for each target; the negative sign indicates that the final goal is actually to minimize the terms in the brackets; diff t (D) to provide for the vehicle supply gap; is the total scheduling time; Δ t is the total additional travel time; e t The total number of vehicles serviced.

[0082] Furthermore, the updating process of step (8) is as follows:

[0083]

[0084] Since the system space is large, a neural network is needed to maintain the system space. When updating these values, the loss function L is used. i (θ i ) to calculate the difference between the predicted Q value and the target Q value. The loss function expression is as follows:

[0085]

[0086] In the formula, θ i and are the weights of the training neural network and the target neural network, respectively.

[0087] Compared with the prior art, the present invention has the following beneficial effects:

[0088] 1. The present invention focuses on driverless electric taxis. In a dynamic carpooling scenario with multiple passengers, the present invention takes into account the electric vehicle's range constraints and vehicle charging and energy replenishment needs, which is more in line with actual operational requirements and can effectively improve service quality.

[0089] 2. The present invention establishes a spatiotemporal graph convolutional neural network based on the attention mechanism, which can capture the spatiotemporal characteristics of passenger travel demand better than the baseline model, thereby realizing high-precision rolling prediction of passenger travel demand in multiple regions, which can effectively improve vehicle operation efficiency and reduce resource waste.

[0090] 3. This invention improves the Double DQN algorithm, considers the convergence characteristics of the loss function in the training process, and establishes a dynamic adjustment mechanism for the learning rate and greed coefficient, so that it changes nonlinearly according to the training process. This improvement improves the training stability, speeds up the convergence speed in complex dynamic scenarios, and improves the efficiency of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 is a flow chart of the method of the present invention;

[0092] Figure 2 Schematic diagram of the structure of the spatiotemporal graph convolutional neural network based on the attention mechanism, including: (a) the structure of the spatiotemporal graph convolutional neural network based on the attention mechanism, (b) the spatiotemporal module structure, (c) the temporal attention module, and (d) the spatial attention module;

[0093] Figure 3 An improved Double DQN algorithm is proposed for the present invention;

[0094] Figure 4 Pseudocode for the Double DQN algorithm. DETAILED DESCRIPTION

[0095] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0096] Figure 1 The following is a dynamic carpooling scheduling method for driverless electric taxis in this embodiment. Figure 1 A detailed description of the steps involved:

[0097] Step 1: Obtain the user's historical travel orders, the user's real-time carpooling orders, and the vehicle's real-time status and location information, including but not limited to: the passenger's travel starting point and end point, the number of travelers and departure time, the vehicle's remaining battery power and mileage, the vehicle's location, the number of passengers the vehicle can carry, etc.;

[0098] Step 2: Clean the data of the user's historical travel orders, including but not limited to: converting longitude and latitude into the WGS84 coordinate system, converting timestamps into date format, deleting data outside the study area, clearing duplicate orders, etc.

[0099] Step 3: Divide the operating area into regular hexagonal grids and map the cleaned data into each hexagonal grid. The specific process is as follows:

[0100] 3.1 Use the H3 spatial index system to divide the operation area into spatial grids, and use the H3 hexagonal grid with a resolution of 8 as the minimum action unit.

[0101] 3.2 Through the H3 spatial index system, the carpooling demand, the supply of vehicles that can pick up passengers, the supply-demand difference values, etc. cleaned in step 2 are mapped to each hexagonal grid respectively to reflect the differences in the spatiotemporal distribution of users' carpooling demand.

[0102] Step 4: Based on the data cleaning and travel demand spatiotemporal distribution results in step 3, establish the travel demand characteristic matrix and spatial dependence matrix to quantify the passenger travel time characteristics and spatial characteristics respectively. The specific process is as follows:

[0103] 4.1 In the spatial dimension, passenger travel demands in different regions will affect each other and have spatial dependence. Granger causality test is a hypothesis testing method used to detect whether a set of variables is affected by the lag of another set of variables. If such an influence exists between two variables, there is a Granger causal relationship. The Granger causality test is used to characterize the spatial dependence characteristics of passenger travel between regions, and a spatial dependence matrix A is established. The elements in the matrix only contain 0 and 1, representing whether there is a dependence on travel demand between the two regions: if there is no dependence, it is recorded as 0; if there is a dependence, it is recorded as 1. The details are as follows,

[0104]

[0105] In the formula, A adj ; i and j are the divided areas; Lj is the arrival demand time series of area j; Li is the travel demand series of area i.

[0106] 4.2 In the time dimension, passenger travel demand data has natural periodicity, such as daily periodicity and weekly periodicity. In addition, the travel demand near the prediction time will also affect the prediction of the target time series. Therefore, the travel demand data of the previous moment of the target prediction time, the travel demand data of the corresponding moment of the previous day, and the travel demand data of the corresponding moment of the previous week are extracted to construct the travel demand feature matrix X, and then the time series fragments containing the adjacent time, daily period and weekly period are constructed;

[0107] Step 5: Construct a spatiotemporal graph convolutional neural network based on the attention mechanism, input the travel demand feature matrix X and the spatial dependency matrix A in step 4, and obtain the travel demand prediction results. The specific process is as follows:

[0108] 5.1 Construct a spatiotemporal graph convolutional neural network based on the attention mechanism, input the time series fragments in step 4, together with the spatial dependency matrix, into the corresponding dependency analysis component to generate the normalized spatial attention matrix E′ and temporal attention matrix S′;

[0109] 5.2 The temporal attention mechanism in this patent is as follows Figure 2(c) As shown. First, a time dimension convolution module is constructed to operate along the time axis of the data to extract important features in the time series, thereby capturing the dynamic patterns and correlations of passenger travel demand over time. Through the one-dimensional convolution operation in the time dimension, the model updates the passenger travel demand in real time along the time axis according to the size of the convolution kernel, capturing the correlation between each moment. This time dimension analysis can increase the depth of the fully connected layer, ensure that the output matches the prediction target dimension, and optimize the accuracy and efficiency of demand forecasting. Specifically,

[0110]

[0111] Where ReLU is the activation function and Φ is the convolution and parameter in the time dimension.

[0112] Then, we build a time dimension attention module. Through the self-attention mechanism, the model can automatically identify and extract the key dependency features in the travel demand data, and dynamically adjust the correlation strength of travel demand in the time dimension according to the specific situation of the input data. The details are as follows:

[0113]

[0114]

[0115] Where V i 、b i , Z1, Z2, and Z3 are parameters learned from input data; E is the temporal correlation matrix automatically learned by the time dimension attention module based on the input data at the current moment. S′ is the temporal attention matrix obtained by normalizing the E matrix using the Softmax function; S m，n is the correlation degree between nodes m and n.

[0116] Furthermore, by dynamically adjusting the input data, its features can be extracted more deeply. Normalized temporal attention matrix Applied to the current input data, we get Using the adjusted data, the features for the next stage can be further extracted;

[0117] 5.3 The spatial attention mechanism in this patent is as follows Figure 2 (d) First, we build a spatial dimension graph convolution module, use spectral graph theory to interpret the data as a graph structure, and use convolution technology to process these graph data. Through the ChebNet graph convolution analysis of the spatial dimension, we extract spatial features from the graph information, thereby revealing the correlation of passenger travel needs at the spatial level. The details are as follows:

[0118]

[0119] In the formula, is the graph signal matrix feature of the passenger travel data at the t-th time in the M-th area; is the parameter to be learned; is the normalized Laplacian matrix.

[0120] Then, a spatial dimension attention module is constructed, and the self-attention mechanism is adopted to capture the dynamic changes of inter-region dependencies in the spatial dimension, and automatically adjust the correlation between passenger travel demand data to adapt to these changes.

[0121] The details are as follows:

[0122]

[0123] In the formula, is the input data of the k-th spatiotemporal module; C k-1 The number of channels for input data to the k-th spatiotemporal module; T k-1 V is the time dimension length of the input data of the k-th spatiotemporal module; j 、b j , Y1, Y2, and Y3 are parameter sets learned through the attention mechanism; E is the spatial attention matrix learned based on the current input data; E m，n is the strength of the interaction between nodes m and n; E′ is the normalized spatial attention matrix generated by standardizing E using the Softmax function, and each element value in the matrix represents the weight coefficient between regions.

[0124] Furthermore, the spatial attention matrix E′ and the matrix of spatial dependency in the Chebyshev polynomial in the spatial graph convolution are dynamically adjusted to achieve a reasonable distribution of attention between regions by influencing the weights;

[0125] 5.4 The dynamic spatiotemporal features extracted in steps 5.2 and 5.3 are trained by stacking spatiotemporal modules and applying a residual learning strategy. The outputs of the proximity dependency, daily cycle, and weekly cycle components are adjusted using a fully connected layer to match the dimensions of the prediction target, and then these outputs are fused to obtain the final travel demand forecast.

[0126] Step 6: Set the constraints for matching the order in the dynamic carpooling scenario, including but not limited to: capacity constraints, distance constraints, time constraints, mileage constraints, and sequence constraints, as follows:

[0127] 6.1 During operation, the vehicle must provide enough seats and strictly control the number of passengers not exceeding the total number of seats. Therefore, for any new order, the number of passengers included in the order must not exceed the current remaining available capacity of the vehicle. The mathematical expression of the capacity constraint is as follows:

[0128] N r ≤C max -n c

[0129] Where N r is the number of passengers included in the new order r, which must not exceed the current remaining available capacity of the vehicle; n c is the number of passengers on board; C max is the total number of seats in the vehicle, which is set to 4;

[0130] 6.2 Pickup distance constraint By controlling the distance between the current position of the vehicle and the pick-up point, the order pickup distance is guaranteed not to exceed the maximum order pickup distance. At the same time, the total waiting time of the passenger must not exceed the maximum waiting time. If the order pickup distance is too large, the solution space will be too large and the order pickup efficiency will be reduced. The mathematical expression of the order pickup distance constraint is as follows:

[0131] d r，n (n loc , O r )≤ω

[0132] t r，n (n loc , O r )≤t q -t x

[0133] In the formula, O r is the pickup location of the new order r; n loc is the current position of vehicle n; d r，n (n loc , O r ) is the vehicle n from the current position n loc Go to the order pick-up location O r The order receiving distance; t r，n (n loc , O r ) is the order acceptance time required according to the current route; t x is the order response time; t q is the maximum time a user can wait, set to 10 minutes; ω is 3 km.

[0134] 6.3 The detour time constraint means that the additional driving time of passengers due to carpooling shall not exceed several times the time of passengers traveling alone, thereby reducing the detour time in the process of carpooling, improving passenger satisfaction and improving the efficiency of carpooling. The larger the detour time constraint, the higher the possibility of successful carpooling. The mathematical expression of the detour distance constraint is as follows,

[0135]

[0136] Where t is the time that has passed since the passenger trip l requested the ride; is the new estimated travel time due to detour or adding a new carpooling passenger to the passenger trip starting from time t to the end; is the travel time if the vehicle only serves passenger trip l; ρ is a constant, set to 0.5.

[0137] 6.4 The range constraint reflects the electrification characteristics of driverless taxis. The range constraint can ensure that the vehicle can complete the received order tasks, and the range cannot be too low at the end of the entire journey, so as to reserve power for the vehicle in a low-power state to travel to the charging facility. Assuming that the vehicle needs to be recharged every 12 hours of work, considering the actual charging situation, the recharge time is set to 2 hours. The mathematical expression of the range constraint is as follows,

[0138] r′ n -l r ≤range o

[0139] In the formula, range0 is the minimum cruising range; l r is the updated driving route mileage after order r is added, which is set to 100 km; r′ n is the cruising range of the current vehicle n.

[0140] 6.5 The order constraint ensures that the starting point of any order should be before the end point in the new driving route. It is mainly used in the process of assigning new orders to vehicles and forming a new driving path. The mathematical expression of the order constraint is as follows:

[0141]

[0142] In the formula, Pick-up location for passengers O r order in the set of vehicle destinations; Passenger drop-off point D r The order in the vehicle's destination set.

[0143] Step 7: Based on the travel demand prediction results in step 5 and the vehicle-order matching constraints in step 6, a vehicle-order matching decision model based on distributed deep reinforcement learning is constructed. The specific process is as follows:

[0144] 7.1 The real-time order matching and vehicle scheduling process is modeled as a Markov decision process, and the states, actions, rewards, and state transition probabilities involved in the Markov decision process are defined.

[0145] Define state variables to map the state of the environment, which will affect the reward feedback obtained by different actions. This patent defines a triple to capture the system state update at time t: (X t , V t：t+T , D t：t+T ), and combine the triplet state variables in a vector s t Among them, X t ={x t,1 , x t，2 , ..., x t,N} is the current state of all N vehicles at time t, V t：t+T is the number of vehicles in each area in the next T time periods, D t：t+T Forecast future travel demand for each region from time t to time t+T.

[0146] Define the action a of vehicle n t,n It consists of two parts: the first part is if the vehicle is partially full, it decides whether to serve existing passengers or new passengers. The second part is if there are no passengers on the vehicle or it decides to serve new passengers, it needs to decide the area the vehicle should go to at time slot t, using u t，n，i If the vehicle decides to provide service to a new user and decides to go to area i, then u t，n，i =1, otherwise 0. If vehicle n has no empty slots, it cannot serve any other passengers. If the vehicle only serves existing passengers, it may not drop off passengers in the order they boarded. In order to minimize the driving distance and cover all destinations quickly, the vehicle will choose the best route to reach each destination more quickly.

[0147] If a vehicle chooses to serve only the existing passengers in the vehicle, it has no effect on the demand imbalance, dispatch time, detour time, and number of vehicles used, so its reward is 0. Therefore, this study focuses on the decision of whether to make a carpool in the vehicle-order matching decision, and turns its attention to the scenario where the vehicle decides to serve a new passenger and is willing to detour at time t. Therefore, the reward r of vehicle n at time slot t is defined as t , n is as follows,

[0148]

[0149] In the formula, β i is the weight of each part in the reward expression; b t，n The number of orders served by vehicle n at time t; c t，n is the time it takes for vehicle n to take a detour to pick up an additional passenger if there are available seats; δ t，n，l The additional time that vehicle n spends due to carpooling compared to directly serving passenger trip l; et，n is the vehicle occupancy status, which is set to 1 if the idle vehicle n is occupied, and is 0 if a vehicle with passengers accepts a new customer.

[0150] Define the state transition probability matrix as P(s t+1 |s t , a t，n ), which represents the probabilistic dependency between action and reward functions. This patent uses a neural network to determine the transition probability and reward distribution, and is used to find the state action value Q value. At time t, the agent observes the environment state s consisting of vehicle states, future demands, and future available vehicles. t , according to the reward r t Take action a t After selecting an action, the environment matches the vehicle and the order according to the action, and updates the vehicle status X in time after the matching is completed. t .

[0151] 7.2 Design decision variables and objective functions for multi-agents.

[0152] At each time step t, the model algorithm obtains the state s of the environment t and the reward r given by the environment t Based on the state of the environment and the reward, the model takes action a i To guide the idle vehicles to different locations to maximize the expected future long-term rewards. In this process, the agent has two decision variables. The first decision variable is At time t, the agent decides to dispatch vehicle n to area j, then Otherwise, it is 0. The second decision variable is ζ n，t , if the vehicle n is not full, the agent needs to decide whether the vehicle can provide service to the new customer in time period t. If it decides to provide service to the customer, then ζ n，t =1, otherwise ζ n，t = 0. Therefore, the objective function of the multi-agent at time t is as follows:

[0153]

[0154]

[0155] In the formula, β1, β2, β3, and β4 are weight factors for each target; the negative sign indicates that the final goal is actually to minimize the terms in the brackets; diff t (D) to provide for the vehicle supply gap; is the total scheduling time; Δ t is the total additional travel time; e t The total number of vehicles serviced.

[0156] Step 8: Design the Double DQN solution algorithm. The pseudo code of the improved Double DQN algorithm is as follows: Figure 3 As shown in the figure, the detailed steps of the training phase are shown. First, the experience pool D is initialized, the Q network parameters θ are trained, and the target Q network parameters At the beginning of each subsequent training, the simulator is initialized, the scheduling command is executed at each time step, and the entire environment state Ω is updated t After that, the agent obtains the corresponding reward according to the action based on the environment information. Then, the following loop operation is performed on all available vehicles. First, the state vector Ω of the vehicle is obtained. t，n , then record the state at the last moment, the action at the last moment, the reward obtained, the state vector of vehicle n at this moment, and the scheduling time c required to reach the new destination t，n , and store it in the tuple (Ω t-1，n , a t-1，n， r t，n ,Ω t，n , c t，n ). Then, tuples are selected from the experience pool D as samples for training, and the network is continuously updated, while the available vehicle set A is updated. t After obtaining the set of available vehicles, the Epsilon-Greedy method is used to select actions for all available vehicles. A random action is selected with a probability of ε, and the action with the largest action value in the training neural network is selected with a probability of 1-ε. After selecting the action, the vehicle is transferred to the new destination according to the selected action, and the new vehicle state is obtained. For the nth vehicle, after selecting the action, according to the reward r t，n , update the Q value with learning rate a, the update process is as follows:

[0157]

[0158] Since the system space is large, a neural network is needed to maintain the system space. When updating these values, the loss function L is used i (θ i ) to calculate the difference between the predicted Q value and the target Q value. The loss function expression is as follows:

[0159]

[0160] In the formula, θ i and are the weights of the training neural network and the target neural network, respectively.

[0161] This patent uses experience replacement and two independently trained neural networks to estimate action values, and dynamically adjusts the learning rate a and the greedy coefficient ε as the training process and the loss function change during the training process, making them dynamically variable parameters.

[0162] In the initial stage of training, a warm-up phase is set up to prevent the instability of the model at the beginning of training due to the use of a larger learning rate. In this phase, the learning rate gradually increases from a smaller value to the predetermined initial learning rate. After the warm-up period, it switches to the normal training phase. At the beginning of this phase, the learning rate is higher, which is conducive to the intelligent agent's exploration and learning of the environment, so that the model can converge faster. After that, to ensure the stability of training, the learning rate is gradually reduced. In order to make the learning rate change smoother, this patent selects exponential preheating and exponential decay in the warm-up period. The learning rate formulas for the exponential preheating period and the exponential decay period are as follows:

[0163]

[0164] Where α(t) is the learning rate at training step t; s is the initial value of the learning rate; α max is the maximum learning rate at the end of the warm-up period; α min is the minimum learning rate; T w is the total number of steps in the warm-up period.

[0165] On this basis, in order to improve the generalization ability and adaptability of the model, this patent uses a loss feedback mechanism to dynamically adjust the learning rate according to the rate of change of the loss. It can change the learning rate in time when fluctuations or instability occur during the training process to prevent the model from overfitting at the local optimal solution. The principle of the learning rate loss feedback mechanism is: if the loss of the model continues to decrease, it indicates that the current parameter update direction is correct, and the learning rate can be appropriately reduced for fine-tuning; if the loss suddenly increases, it is necessary to increase the learning rate to jump out of the local minimum. Therefore, on the basis of the basic learning rate scheduling α(t), α′(t) can be dynamically adjusted according to the change of the loss value, and its expression is as follows:

[0166] α′ (t) =α(t)·f(ΔL i (θ i ))

[0167] Where, ΔL i (θ i ) is the change of the loss function, which is the weighted average of the changes of the loss function in the last 10 steps; f(·) is the dynamic adjustment function, which is defined as follows:

[0168]

[0169] In the formula, k and k′ are positive adjustment coefficients used to adjust the speed of the learning rate. If the loss value decreases, then f(ΔL i (θ i)) will cause the learning rate to be slightly decreased; if the loss increases, the learning rate will be increased to try to correct the direction or escape from the local optimum.

[0170] Step 9: Based on the model solution results in step 8, real-time order matching and idle vehicle scheduling are performed, matching vehicles are notified to complete corresponding orders, and unmatched idle vehicles are notified to perform regional scheduling or charging and energy replenishment.

[0171] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A dynamic carpooling scheduling method for driverless electric taxis, characterized in that: The steps include: (1) Obtain the user's historical travel orders, carpooling orders, and vehicle status and location information data; (2) Clean the data of users’ historical travel orders; (3) Divide the operating area into regular hexagonal grids and map the cleaned data into each hexagonal grid to reflect the differences in the spatiotemporal distribution of users' carpooling travel needs; (4) Based on the data cleaning and travel demand spatiotemporal distribution results in step (3), a travel demand characteristic matrix and a spatial dependence matrix are established to quantify the travel time characteristics and spatial characteristics of passengers, respectively; (5) Construct a spatiotemporal graph convolutional neural network based on the attention mechanism to obtain travel demand prediction results; (6) Setting constraints for matching rides in dynamic carpooling scenarios, including capacity constraints, distance constraints, time constraints, mileage constraints, and sequence constraints; (7) Based on the travel demand prediction results in step (5) and the vehicle-order matching constraints in step (6), a vehicle-order matching decision model based on distributed deep reinforcement learning is constructed; (8) Design a Double DQN solution algorithm, use the Double DQN network to dynamically generate optimization strategies, define the vehicle's optimal action value function as the maximum expected achievable reward, and use the action that maximizes the neural network output value for each vehicle. The learning process starts from scratch and uses the Epsilon-Greedy method to select actions using a greedy scheme. Under this strategy, the agent selects the action that results in the highest Q value with probability 1-ε, otherwise it selects a random action. For the nth vehicle, after selecting an action, it is given a reward r. t，n , update the Q value with learning rate a; (9) Based on the model solution results in step (8), real-time order matching and idle vehicle scheduling are performed, matching vehicles are notified to complete corresponding orders, and unmatched idle vehicles are notified to perform regional scheduling or charging and energy replenishment.

2. The dynamic carpooling scheduling method for driverless electric taxis according to claim 1 is characterized in that: The data in step (1) include the starting point and end point of the passenger trip, the number of passengers and the departure time, the remaining battery power and driving range of the vehicle, the vehicle location, and the number of passengers that the vehicle can carry.

3. The dynamic carpooling scheduling method for driverless electric taxis according to claim 2 is characterized in that: The data cleaning in step (2) includes: converting longitude and latitude into the WGS84 coordinate system, converting timestamps into date format, deleting data outside the study area, and clearing duplicate orders.

4. The dynamic carpooling scheduling method for driverless electric taxis according to claim 3 is characterized in that: The step (3) comprises: 3.1 Use the H3 spatial index system to divide the operation area into spatial grids, and use the H3 hexagonal grid with a resolution of 8 as the minimum action unit; 3.2 Through the H3 spatial index system, the carpooling demand, available passenger vehicle supply, and supply-demand difference values ​​cleaned in step (2) are mapped to each hexagonal grid to reflect the differences in the spatiotemporal distribution of users' carpooling demand.

5. The dynamic carpooling scheduling method for driverless electric taxis according to claim 3 is characterized in that: The step (4) comprises: 4.1 In the spatial dimension, the Granger causality test is used to characterize the spatial dependence characteristics of passenger travel between regions, and a spatial dependence matrix A is established. The elements in the matrix only contain 0 and 1, representing whether there is a dependence on travel demand between two regions: if there is no dependence, it is recorded as 0; if there is a dependence, it is recorded as 1, as follows, In the formula, A adj ; i and j are the divided areas; Lj is the arrival demand time series of area j; Li is the travel demand series of area l; 4.2 In the time dimension, passenger travel demand data has a natural periodicity. In addition, the travel demand near the prediction time will also affect the prediction of the target time series. The travel demand data of the previous moment of the target prediction time, the travel demand data of the corresponding moment of the previous day, and the travel demand data of the corresponding moment of the previous week are extracted to construct the travel demand feature matrix X, and then construct a time series segment containing adjacent time, daily cycle and weekly cycle.

6. The dynamic carpooling scheduling method for driverless electric taxis according to claim 5 is characterized in that: The step (5) comprises: 5.1 Construct a spatiotemporal graph convolutional neural network based on the attention mechanism, input the time series fragments in step 4, together with the spatial dependency matrix, into the corresponding dependency analysis component to generate the normalized spatial attention matrix E′ and temporal attention matrix S′; 5.2 Temporal Attention Mechanism: First, a time dimension convolution module is constructed to operate along the time axis of the data to extract important features in the time series, thereby capturing the dynamic patterns and correlations of passenger travel demand over time. Through the one-dimensional convolution operation in the time dimension, the model updates the passenger travel demand in real time along the time axis according to the size of the convolution kernel, capturing the correlation between each moment, as follows: In the formula, ReLU is the activation function, Φ is the convolution and parameter in the time dimension, Then, we construct a time dimension attention module. Through the self-attention mechanism, the model can automatically identify and extract the key dependency features in the travel demand data, and dynamically adjust the correlation strength of travel demand in the time dimension according to the specific situation of the input data, as follows: Where V i 、b i , Z1, Z2, and Z3 are parameters learned from the input data; E is the time correlation matrix automatically learned by the attention module of the time dimension according to the input data at the current moment; S′ is the time attention matrix obtained by normalizing the E matrix using the Softmax function; S m，n is the correlation between nodes m and n, Furthermore, by dynamically adjusting the input data, its features can be extracted more deeply, and the normalized time attention matrix Applied to the current input data, we get Using the adjusted data, the features for the next stage can be further extracted; 5.3 Spatial Attention Mechanism: First, we build a spatial dimension graph convolution module, use spectral graph theory to interpret the data as a graph structure, use convolution technology to process these graph data, and extract spatial features from the graph information through spatial dimension ChebNet graph convolution analysis, thereby revealing the correlation of passengers' travel needs at the spatial level. The details are as follows: In the formula, is the graph signal matrix feature of the passenger travel data at the t-th time in the M-th area; is the parameter to be learned; is the normalized Laplace matrix, Then, we construct a spatial dimension attention module and use the self-attention mechanism to capture the dynamic changes of inter-region dependencies in the spatial dimension and automatically adjust the correlation between passenger travel demand data to adapt to these changes, as follows: In the formula, is the input data of the k-th spatiotemporal module; C k-1 The number of channels for input data to the k-th spatiotemporal module; T k-1 V is the time dimension length of the input data of the k-th spatiotemporal module; j 、b j , Y1, Y2, and Y3 are parameter sets learned through the attention mechanism; E is the spatial attention matrix learned based on the current input data; E m，n is the strength of the interaction between nodes m and n; E′ is the normalized spatial attention matrix generated by normalizing E using the Softmax function. Each element value in the matrix represents the weight coefficient between regions. Furthermore, the spatial attention matrix E′ and the matrix of spatial dependency in the Chebyshev polynomial in the spatial graph convolution are dynamically adjusted to achieve a reasonable distribution of attention between regions by influencing the weights; 5.4 By stacking spatiotemporal modules and applying a residual learning strategy for training, the outputs of the proximity dependency, daily cycle, and weekly cycle components are adjusted using fully connected layers to match the dimensions of the prediction target, and then these outputs are fused to obtain the final travel demand forecast.

7. The dynamic carpooling scheduling method for driverless electric taxis according to claim 6 is characterized in that: Step (6) include: 6.1 The mathematical expression of capacity constraint is as follows, N r ≤C max -n c Where N r is the number of passengers included in the new order r, which must not exceed the current remaining available capacity of the vehicle; n c is the number of passengers on board; C max is the total number of seats in the vehicle, set to 4; 6.2 The mathematical expression of the order acceptance distance constraint is as follows: d r，n (n loc ,O r )≤ω t r，n (n loc ,O r )≤t q -t x In the formula, O r is the pickup location of the new order r; n loc is the current position of vehicle n; d r,n (n loc , O r ) is the vehicle n from the current position n loc Go to the order pick-up location O r The order receiving distance; t r，n (n loc , O r ) is the order acceptance time required according to the current route; t x Order response time; t q is the maximum time a user can wait, set to 10 minutes; ω is 3 km, 6.3 The mathematical expression of the detour distance constraint is as follows: Where t′ is the time that has passed since the passenger trip l requested a ride; is the new estimated travel time due to detour or adding a new carpooling passenger to the passenger trip starting from time t to the end; is the travel time if the vehicle only serves passenger trip l; ρ is a constant, set to 0.5, 6.4 The mathematical expression of the cruising range constraint is as follows: r′ n -l r ≤range0 In the formula, range0 is the minimum cruising range; l r is the updated driving route mileage after order r is added, which is set to 100 km; r′ n is the cruising range of the current vehicle n, 6.5 The mathematical expression of the order constraint is as follows, In the formula, Pick-up location for passengers O r order in the set of vehicle destinations; Passenger drop-off point D r The order in the vehicle's destination set.

8. The dynamic carpooling scheduling method for driverless electric taxis according to claim 7 is characterized in that: The step (7) comprises: 7.1 Model the real-time order matching and vehicle dispatching process as a Markov decision process, and define the states, actions, rewards, and state transition probabilities involved in the Markov decision process; Define a triple to capture the system state update at time t: (X t , V t：t+T , D t：t+T ), and combine the triplet state variables in a vector s t Among them, X t ={x t，1 , x t,2 , ..., x t,N } is the current state of all N vehicles at time t, V t：t+T is the number of vehicles in each area in the next T time periods, D t :t +T Forecast future travel demand for each region from time t to time t+T; Define the action a of vehicle n t，n It consists of two parts: the first part is that if the vehicle is partially full, it decides whether to serve existing passengers or new passengers; the second part is that if there are no passengers on the vehicle or it decides to serve new passengers, it needs to decide the area that the vehicle should go to at time slot t, using u t，n，i If the vehicle decides to provide service to a new user and decides to go to area i, then u t，n，i =1, otherwise 0; if vehicle n has no vacancies, it cannot serve any other passengers; Define the reward r of vehicle n at time slot t t，n as follows, In the formula, β i is the weight of each part in the reward expression; b t，n The number of orders served by vehicle n at time t; c t，n is the time it takes for vehicle n to detour to pick up additional passengers if there are available seats; δ t，n，l The additional time that vehicle n spends due to carpooling compared to directly serving passenger trip l; e t，n is the vehicle occupancy status, which is set to 1 if the idle vehicle n is occupied, and 0 if a vehicle with passengers accepts a new customer; Define the state transition probability matrix as P(s t+1 |s t , a t，n ), represents the probabilistic dependency between action and reward functions. This patent uses a neural network to determine the transition probability and reward distribution, and is used to find the state action value Q value; 7.2 Designing decision variables and objective functions for multi-agents, At each time step t, the model algorithm obtains the state s of the environment t and the reward r given by the environment t , based on the state of the environment and the reward, the model takes action a t To guide the idle vehicles to different locations to maximize the expected future long-term rewards, the agent has two decision variables in this process. The first decision variable is At time t, the agent decides to dispatch vehicle n to area j, then Otherwise, it is 0. The second decision variable is ζ n，t If the vehicle n is not full, the agent needs to decide whether the vehicle can provide services to the new customer in time period t. If it decides to provide services to the customer, then n，t =1, otherwise ζ n，t =0, thus, the objective function of the multi-agent at time t is as follows: In the formula, β1, β2, β3, and β4 are weight factors for each target; the negative sign indicates that the final goal is actually to minimize the terms in the brackets; diff t (D) to provide for the vehicle supply gap; is the total scheduling time; Δ t is the total additional travel time; e t The total number of vehicles serviced.

9. The dynamic carpooling dispatching method for driverless electric taxis according to claim 8 is characterized in that: The updating process of step (8) is as follows: Since the system space is large, a neural network is needed to maintain the system space. When updating these values, the loss function L is used. i (θ i ) to calculate the difference between the predicted Q value and the target Q value. The loss function expression is as follows: In the formula, θ i and are the weights of the training neural network and the target neural network, respectively.

Images