Vehicle path planning method and device based on reinforcement learning

Abstract

The application provides a vehicle path planning method and device based on reinforcement learning, and relates to the technical field of data processing. The method comprises the following steps: inputting a state vector at a t time step into a first model to obtain a first expected return value corresponding to a first sample action; selecting a target first sample action to update a state vector at a t+1 time step; inputting the state vector at the t+1 time step into a second model to obtain a plurality of second sample actions and corresponding second expected return values; determining a target expected return value based on the second expected return values, determining a training loss based on the target expected return value and the first expected return value corresponding to the target first sample action, updating parameters of the first model based on the training loss; performing soft updating on parameters of the second model based on the parameters of the first model after a plurality of time steps; and obtaining a vehicle path planning result based on output data of the first model after training. The application can improve the robustness of vehicle path planning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a vehicle path planning method and device based on reinforcement learning. BACKGROUND

[0002] Reinforcement learning is a sequential decision-making method based on the interaction between an agent and an environment. The agent learns a strategy through continuous exploration and feedback to maximize its long-term cumulative reward. Reinforcement learning has been applied to solve vehicle path planning problems.

[0003] In actual application scenarios, vehicle path planning problems are often complex and changeable. However, in existing vehicle path planning methods based on traditional reinforcement learning, there is often a problem of strategy bias caused by overestimation of expected return value (Q value). Moreover, there is a lack of training stability guarantee mechanism in high-dimensional state space, which easily leads to strategy fluctuation and convergence difficulty. The robustness of existing vehicle path planning methods is poor. SUMMARY

[0004] The present application provides a vehicle path planning method and device based on reinforcement learning to solve the problem of poor robustness of existing vehicle path planning methods and improve the robustness of vehicle path planning methods.

[0005] The present application provides a vehicle path planning method based on reinforcement learning, comprising:

[0006] Input a state vector at time step t into a first model to obtain first expected return values corresponding to a plurality of first sample actions output by the first model. Each first sample action corresponds to a call to a path planning operator, and the state vector includes a sequence of operators that have been called.

[0007] Select one action as a target first sample action from the plurality of first sample actions, update the state vector based on the target first sample action, and obtain the state vector at time step t+1.

[0008] Input the state vector at time step t+1 into a second model to obtain a plurality of second sample actions output by the second model and second expected return values corresponding to each second sample action.

[0009] Determine a target expected return value based on the second expected return values, determine a training loss based on the target expected return value and the first expected return value corresponding to the target first sample action, and update parameters of the first model based on the training loss.

[0010] After a plurality of time steps, soft update parameters of the second model based on the parameters of the first model.

[0011] Based on the output data of the first model after training, a vehicle path planning result is obtained.

[0012] According to the vehicle path planning method based on reinforcement learning provided by the application, each path planning operator corresponds to a path adjustment operation, and each path planning operator is associated with a parameter k, which is used to control the size of the path adjustment operation corresponding to the path planning operator.

[0013] According to the vehicle path planning method based on reinforcement learning provided by the application, selecting one action as a target first sample action from the plurality of first sample actions comprises:

[0014] Based on a first formula, the exploration rate is determined, and the first formula is: Wherein, represents the exploration rate corresponding to the t time step, represents the exploration rate reference value, and T is an adjustment parameter, and the size of T is negatively related to the size of the time step;

[0015] Based on the exploration rate, the target first sample action is selected from the plurality of first sample actions.

[0016] According to the vehicle path planning method based on reinforcement learning provided by the application, the target expected return value is determined based on the second expected return value, comprising:

[0017] The second sample action corresponding to the maximum second expected return value is obtained as a target second sample action;

[0018] Based on the effect value of the path planning scheme corresponding to the plurality of second sample actions, the reward value of the t time step is determined;

[0019] Based on the reward value of the t time step, the target expected return value is determined.

[0020] According to the vehicle path planning method based on reinforcement learning provided by the application, the reward value of the t time step is determined based on the effect value of the path planning scheme corresponding to the plurality of second sample actions, comprising:

[0021] Obtaining the maximum value and the average value of the effect improvement value of the path planning scheme corresponding to the plurality of second sample actions,

[0022] Obtaining a reward change trend value, the reward change trend value reflecting the change trend of the reward value of the latest N time steps;

[0023] Based on the maximum value, the average value and the reward change trend value, the reward value of the t time step is determined.

[0024] According to the vehicle path planning method based on reinforcement learning provided by the application, the first model is a multi-layer fully connected neural network, and each hidden layer in the first model has a Dropout mechanism.

[0025] The application further provides a vehicle path planning device based on reinforcement learning, comprising:

[0026] The first expected return obtaining module is configured to input a state vector at a t time step into the first model, and obtain first expected return values corresponding to a plurality of first sample actions output by the first model, each of the first sample actions corresponding to calling a path planning operator, and the state vector comprising a sequence of operators that have been called.

[0027] The action selection module is configured to select one action as a target first sample action from the plurality of first sample actions, update the state vector based on the target first sample action, and obtain the state vector at a t+1 time step.

[0028] The second expected return obtaining module is configured to input the state vector at the t+1 time step into the second model, and obtain a plurality of second sample actions output by the second model and second expected return values corresponding to each of the second sample actions.

[0029] The target expected return determining module is configured to determine a target expected return value based on the second expected return values, determine a training loss based on the target expected return value and the first expected return value corresponding to the target first sample action, and update parameters of the first model based on the training loss.

[0030] The soft updating module is configured to perform soft updating on parameters of the second model based on the parameters of the first model after a plurality of time steps.

[0031] The result output module is configured to obtain a vehicle path planning result based on output data of the first model after training.

[0032] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the vehicle path planning method based on reinforcement learning according to any one of the above when executing the computer program.

[0033] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executable on a processor to implement the vehicle path planning method based on reinforcement learning according to any one of the above.

[0034] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the vehicle path planning method based on reinforcement learning according to any one of the above.

[0035] The application provides a vehicle path planning method and device based on reinforcement learning, which decouples action selection and expected return value evaluation by introducing a double-network architecture of a first model and a second model, can effectively inhibit the overestimation of expected return values in traditional reinforcement learning, thereby improving the stability and convergence quality of vehicle path planning strategy learning, and ensuring robustness in a high-dimensional state space. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0037] Figure 1 is a flowchart of the vehicle path planning method based on reinforcement learning provided by the application.

[0038] Figure 2 is a schematic diagram of path planning operator types and influences in the vehicle path planning method based on reinforcement learning provided by the application.

[0039] Figure 3 is a model training flowchart in the vehicle path planning method based on reinforcement learning provided by the application.

[0040] Figure 4 is an effect diagram of the vehicle path planning method based on reinforcement learning provided by the application Figure 1 .

[0041] Figure 5 is an effect diagram of the vehicle path planning method based on reinforcement learning provided by the application Figure 2 .

[0042] Figure 6 is an effect diagram of the vehicle path planning method based on reinforcement learning provided by the application Figure 3 .

[0043] Figure 7 is a structural diagram of the vehicle path planning device based on reinforcement learning provided by the application.

[0044] Figure 8 is a structural diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0047] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0048] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0049] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0050] The following is combined Figures 1-6 This invention describes a vehicle path planning method based on reinforcement learning. For example... Figure 1 As shown, the reinforcement learning-based vehicle path planning method includes the following steps:

[0051] S110. Input the state vector at time step t into the first model, obtain the first expected revenue value corresponding to multiple first sample actions output by the first model, and call a path planning operator for each first sample action. The state vector includes the sequence of operators that have been called.

[0052] S120, select one of the plurality of first sample actions as a target first sample action, update the state vector based on the target first sample action to obtain a state vector at a t+1 time step;

[0053] S130, input the state vector at the t+1 time step into the second model to obtain a plurality of second sample actions output by the second model and a second expected return value corresponding to each second sample action;

[0054] S140, determine a target expected return value based on the second expected return value, and determine a training loss based on the target expected return value and a first expected return value corresponding to the target first sample action, and update parameters of the first model based on the training loss;

[0055] S150, after a plurality of time steps, soft update parameters of the second model based on the parameters of the first model;

[0056] S160, obtain a vehicle path planning result based on output data of the first model after training is completed.

[0057] The vehicle path planning method based on reinforcement learning provided in the application can effectively inhibit the overestimation of expected return value in traditional reinforcement learning by introducing a dual network architecture of the first model and the second model to decouple action selection and expected return value evaluation, thereby improving the stability and convergence quality of vehicle path planning strategy learning and ensuring robustness in a high-dimensional state space.

[0058] In the vehicle path planning method based on reinforcement learning provided in the application, each path planning operator corresponds to a path adjustment operation, and different targeted operators are designed for vehicle path planning to realize learning of complex and diverse vehicle path planning strategies.

[0059] More specifically, the vehicle path planning method based on reinforcement learning provided in the application is for vehicle path planning for performing a delivery task, and includes multiple roles: a customer, a distribution center, and a vehicle. The vehicle needs to deliver goods from the distribution center to the customer. Before training the first model and the second model, the vehicle path planning method based on reinforcement learning provided in the application first performs data preparation and environment state definition, and can use an existing standard VRPTW (vehicle routing problem with time windows) benchmark data set (such as Solomon, Gehring-Homberger data set, etc.), which covers multiple customer distribution scenarios such as C, R, and RC. Each problem instance contains customer coordinates, service time length, demand quantity, vehicle capacity, etc. For each instance, the current solution state can be saved, including: customer composition of each path, remaining capacity of each vehicle, current path length, cost of each path, and other information.

[0060] In the method provided in the application, the vehicle routing problem with time windows (VRPTW) is constructed with multiple high-adaptability heuristic operators, which are designed for three levels of intra-path, inter-path and overall structure optimization, and can effectively deal with key problems such as time window constraints, capacity constraints and path scheduling. In the method provided in the application, the action space is composed of two path planning operators: operator operation within the distribution center and operator operation between distribution centers. Specifically, the operator operation within the distribution center refers to the vehicle path adjustment operation of the delivery task of the customers assigned to the same distribution center, and the operator operation between distribution centers refers to the vehicle path adjustment operation of the delivery task of the customers corresponding to different distribution centers. The path planning mode constructed for processing between distribution centers can be expanded from the operation level of the path to the level of the distribution center, and can be adapted to more complex multi-depot vehicle routing problems (Multi-Depot Vehicle Routing Problem, MDVRP).

[0061] As shown in Figure 2 , the operator operation within the distribution center includes the following:

[0062] 1. xchg_in: intra-path exchange of consecutive k customers;

[0063] 2. ins_in: intra-path removal and reinsertion of consecutive k customers;

[0064] 3. xchg_bw: inter-path exchange of consecutive k customers;

[0065] 4. ins_bw: inter-path removal and reinsertion of consecutive k customers;

[0066] 5. 2opt : merging and restructuring of two paths;

[0067] 6. ruin_recreate: destroy and rebuild path structure.

[0068] The operator operation between distribution centers includes the following:

[0069] 1. reassign: reassign k consecutive customers to different distribution centers;

[0070] 2. xchg_out: inter-path exchange of consecutive k customers between different distribution centers;

[0071] 3. ins_out: inter-path removal and reinsertion of consecutive k customers between different distribution centers;

[0072] 4. mer_spl: merging of two paths of different distribution centers and re-splitting;

[0073] 5. no_op: do nothing.

[0074] The specific path adjustment operations corresponding to each path planning operator are described as follows.

[0075] xchg_in (in-path exchange): exchange k consecutive customers in a single path, mainly used to optimize local access order, compress service time span, and relieve time window conflicts.

[0076] ins_in (in-path insertion): remove k consecutive customers from a path and reinsert them at new positions, used to adjust customer order, eliminate local overload or waiting time, and improve path feasibility and compactness.

[0077] xchg_bw (between-path exchange): select k consecutive customers from each of two paths and exchange their positions, coordinate the load and time window tightness between paths, and enhance global balance.

[0078] ins_bw (between-path insertion): remove k consecutive customers from one path and insert them into another path, suitable for resource imbalance or single path overload situations, and assist in reasonable transfer of tasks between paths.

[0079] 2opt : merge two paths into one and then reassign customers and build paths, improve path structure quality through overall rearrangement, often used to escape local optimum.

[0080] ruin_recreate (ruin_recreate): destroy part of the path structure from the current solution and then rebuild it, suitable for global search stage, reshape low-quality paths, and improve solution space diversity.

[0081] reassign (cross-center reassignment): select k consecutive customers from one distribution center and reassign them to other distribution centers, and then reconstruct the path according to the feasibility of the receiving center. This operation helps to break the regional distribution rigidity, relieve local resource pressure, and improve overall distribution flexibility and adaptability.

[0082] xchg_out (cross-center exchange): select k consecutive customers from two different distribution centers and exchange them, used to optimize cross-regional resource utilization efficiency, coordinate task load between centers, and improve global service quality.

[0083] ins_out (inserting out of center): removing continuous k customers in a certain distribution center path and inserting them into the path of other centers, suitable for scenarios of partial distribution center resource bottleneck or uneven customer geographical distribution, effectively improving the overall path feasibility and distribution efficiency.

[0084] mer_spl (merge-split): extracting one path from each of two distribution centers, merging them and then dividing them into two new paths, and respectively belonging to the original distribution center. This operation restructures the customer distribution structure through path recombination, enhances local rearrangement capability, helps to break the disadvantages brought by the original path structure, and improves the overall solution quality.

[0085] no_op (null operation): keeping the current solution unchanged without any path adjustment. Used for strategy control to keep the current solution stable, when the estimated income is low or the disturbance is not conducive to convergence, to avoid invalid operations to improve the robustness of the strategy.

[0086] Further, in the method provided by the application, for each path planning operator, there is a parameter k associated with the path planning operator, which is used to control the size of the path adjustment operation corresponding to the path planning operator. Each operator introduces a parameter k to control the operation size (such as the number of exchanged customers), and by constructing a combination of [operator x k], a complete action space is obtained. For example, xchg_in (k=2) represents exchanging 2 consecutive customers in the path, and ins_bw (k=3) represents inserting 3 customers between paths. Such combinations enable the strategy to flexibly switch between large disturbances and small adjustments.

[0087] By controlling the size of the path adjustment operation corresponding to the operator, the path time window and vehicle capacity constraints are explicitly considered in the operator design, ensuring that the action has structural feasibility in state updating. Moreover, in the method provided by the application, the above-mentioned operators are not only covered by the internal and external multi-level operations of the distribution center and the path, but also have a k value parameter to control the operation size, which can adjust the disturbance intensity according to the task state, and has good adjustability and adaptability. The application not only supports intelligent selection of operator categories, but also first introduces the internal parameters (such as k value) of the operator into the automatic adjustment system, combined with the simulated annealing mechanism to realize dynamic control of search intensity, achieving a better balance between global exploration and local optimization. By jointly encoding the path planning operator and the adjustable parameter k value, a structured action space is constructed, each action not only corresponds to an operation type, but also corresponds to a disturbance size, so that the agent can learn "what operation" and "how much operation" in the process of strategy learning.

[0088] In the method provided in the application, according to the instance size of the solved standard set, the fixed network length is designed, the first model and the second model are policy networks, which will select the next operator according to the selection of the historical operator and the reward brought by the operation. The first model is the main network, and the second model is the target network. The network structures of the first model and the second model are consistent. The main body is a three-layer fully connected neural network, the number of nodes in each layer is determined according to the size of the vehicle path planning problem, the activation function adopts ReLU, the Dropout mechanism is added to each hidden layer to suppress overfitting, and the output is consistent with the size of the action space. The expected return value corresponding to each operator combination. The main network and the target network are decoupled for action selection and evaluation, which effectively alleviates the problem of overestimation of Q value and improves the stability of policy convergence. Through the end-to-end joint learning mechanism, the overall framework realizes the cooperative optimization of search strategy and operator parameters, significantly improves the adaptability and self-evolution ability of the algorithm, reduces the dependence on artificial experience, and is more suitable for complex and variable actual application scenarios.

[0089] In the method provided in the application, in the training and testing process of the reinforcement learning model, the current path state is first represented as a vector form input. According to the size of the problem, a sufficient state space is defined for convergence to a local feasible solution. The state space mainly focuses on the selection of historical actions, and the selection strategy of the action is learned, while the remaining capacity, path distance, service time window and other conditions are uniformly processed in the operation of the operator. When training the first model and the second model based on reinforcement learning, after each action execution, the state transition is recorded and stored in the experience replay buffer.

[0090] Among them, (state at time t): the state at time t. It represents an observation of the environment, which in the application represents a sequence of operators that have been called.

[0091] (action at time t): the action selected by the agent under the state . In the application scenario, it refers to the operator that can be selected.

[0092] (reward at time t): the feedback given by the environment to the agent after executing the action . It is usually a scalar value, which represents the “goodness” of the current action. In the application scenario, it refers to whether the optimization effect of the solution has improved.

[0093] (reward at next state): the feedback given by the environment to the agent after executing the action After, the environment moves to a new state. This embodies the dynamic nature of the system, i.e. actions cause changes in states.

[0094] During training, as shown in FIG. 1, a small batch of data is periodically sampled from the experience pool, the first model outputs the expected return value at the current time step, the second model outputs the target expected return value, and the model is updated based on the outputs of the two models to realize the training of the model. Figure 3

[0095] In the method provided in the application, a neural network architecture capable of learning historical operator sequence information is constructed, a three-layer fully connected network structure is designed, and a ReLU activation function and a Dropout suppression mechanism are introduced, thereby enhancing the stability of policy convergence, and significantly improving the expression ability of problem structure and the policy generalization performance.

[0096] Specifically, after the first model receives the input state vector, a Q value vector with the same dimension as the action space is output, and each dimension represents the expected return value of a certain perturbation operator (such as xchg_in(k=2)) at the current state. That is, the first model outputs a plurality of first sample actions and a plurality of first expected return values corresponding to the plurality of first sample actions respectively. Based on the output of the first model, an action is selected as a target first sample action and the action is executed on the current path solution to generate a new path solution, and the current state vector is updated to obtain a state vector at the t+1 time step.

[0097] Specifically, the action can be selected as the target first sample action by using a greedy strategy or an ε-greedy strategy. In one possible implementation, the action is selected as the target first sample action from the plurality of first sample actions, including:

[0098] Based on the first formula, the exploration rate is determined, and the first formula is: wherein, represents the exploration rate corresponding to the t time step, represents an exploration rate reference value, and T is an adjustment parameter, and the size of T is negatively related to the size of the time step;

[0099] The target first sample action is selected from the plurality of first sample actions based on the exploration rate.

[0100] ​In this implementation, the initial action exploration is performed by adopting the epsilon-greedy strategy, and the simulated annealing mechanism is introduced to adjust the exploration intensity. In the initial training stage, a higher temperature T is set to encourage the use of large-scale perturbation (larger operator size k) to promote global search. As the training round progresses, the temperature gradually decreases, making the perturbation tend to be local fine-tuning (smaller k). The mechanism of adjusting the exploration rate epsilon and the temperature T through the annealing function combines the simulated annealing idea, maintains a high temperature (high perturbation) in the initial training stage to promote solution space exploration, and gradually reduces the perturbation amplitude in the later stage to guide the strategy to converge to fine search, realizing the linkage of search intensity and strategy evolution stage, while improving the stability of the strategy and maintaining a moderate exploration ability.

[0101] The method provided in the application embeds the operator category and the parameter (such as the exchange number k value) in the action space together, and designs a temperature control mechanism by means of the simulated annealing idea, realizes the control strategy of gradually converging the parameter value (the perturbation scale of the problem) with the training stage, and thus enhances the adaptability and search efficiency of the algorithm in different optimization stages.

[0102] After determining the target first sample action and obtaining the state vector of the t+1 time step based on the target first sample action, the state vector of the t+1 time step is input into the second model, and the second model, like the first model, outputs a Q value vector with the same dimension as the action space, each dimension representing the expected return value of a certain perturbation operator (such as xchg_in(k=2)) under the current state (the state vector of the t+1 time step). The expected return value output by the second model is called the second expected return value.

[0103] The target expected return value is determined based on the second expected return value, specifically including:

[0104] The second sample action corresponding to the maximum second expected return value is obtained as the target second sample action;

[0105] The reward value of the t time step is determined based on the effect values of the path planning schemes corresponding to the plurality of second sample actions;

[0106] The target expected return value is determined based on the reward value of the t time step.

[0107] The calculation formula of the target expected return value can be expressed as:

[0108] ;

[0109] Wherein, is the target expected return value, r is the reward value of the t time step, is a discount factor (discount factor), between 0 and 1, representing the importance of future rewards, a state vector at time step t+1, represents a first model prediction of an expected return value of performing a target second sample action a at a state represents a Q value evaluated using a second model at a state

[0110] The reward value at time step t is determined based on the effect values of the path planning schemes corresponding to the plurality of second sample actions, including:

[0111] Obtain the maximum value and the average value of the effect improvement values of the path planning schemes corresponding to the plurality of second sample actions,

[0112] Obtain a reward change trend value, which reflects the change trend of the reward values of the latest N time steps;

[0113] Determine the reward value at time step t based on the maximum value, the average value, and the reward change trend value.

[0114] The effect value of the path planning scheme reflects the performance of the path planning scheme on a preset index (such as path length, transportation cost, delivery time, etc.). Specifically, in order to balance short-term improvement and long-term convergence effect, the reward function is designed as follows:

[0115] ;

[0116] wherein, and respectively represent the maximum and average improvement values of all solution actions in the current step; T is a reward change trend value calculated based on the last 50 training times; w=0.3 if the historical data is sufficient, otherwise 0. All reward components are standardized to the range of [-1, 1] to enhance the stability of training. This reward function encourages robust long-term convergence and avoids strategies falling into short-sighted optimization, and is particularly suitable for the later stage of training.

[0117] Based on the difference between the first expected return value corresponding to the target first sample action and the target expected return value, a training loss is determined, and the parameters of the first model are updated based on the training loss. Specifically, the training loss can be represented by the formula:

[0118] ;

[0119] wherein, L is the training loss, is the target expected return value, represents the first expected return value corresponding to the target first sample action output by the first model based on the state vector at time t.

[0120] ​​Based on the training loss, the parameters of the first model are updated by back propagation, and the parameters of the first model are soft updated to the second model every fixed step, keeping the stability of the second model.

[0121] The training process will continue until a certain custom stop condition is met, such as reaching the maximum number of runs or the quality of the solution converges. The final output includes: the optimal solution currently found (the path with the best effect value), the sequence of operators applied, the corresponding reward trajectory, and the final trained parameters of the main network. The optimal solution currently found can be used as the path planning result, and the sequence of operators applied, the corresponding reward trajectory can be used as a reference and applied to other path planning problems. This method can achieve adaptive operator selection by learning from historical experience and generalize to solution sets that are difficult to achieve in complex optimization problems.

[0122] In order to verify the superiority of the vehicle path planning method based on reinforcement learning provided in the present application, experiments are carried out based on multiple instances, which belong to Solomon benchmark dataset. Each instance is represented by letters and numbers. The letter instance category includes C, R and RC instances. C instances are instances with clustered customer point locations, R instances are instances with uniformly random distributed customer points, and RC instances are instances with partially clustered and partially random distributed customer points. The first digit "1 / 2" in the number represents the time window / capacity group. 1 series represents tight time window and small vehicle capacity, and 2 series represents wide time window and large vehicle capacity. The other digits represent the number of the instance. For example, R111 represents an R instance, 1 series (tight time window, small capacity), and number 11, and C208 represents a C instance, 2 series (wide time window, large capacity), and number 08.

[0123] Table 1 shows the comparison results of the solution (path planning result) found by the vehicle path planning method based on reinforcement learning provided in the present application and the known optimal solution on the standard VRPTW instance, including the objective value, the number of vehicles used and the total path length, and the relative deviation of the objective value and the path length is calculated. The experimental results show that in centralized instances (such as C102, C109), the method of the present application can achieve near-optimal vehicle number and smaller objective value gap; while in more complex instances (such as R111, RC206), the deviation is relatively large, but in most cases the number of vehicles used is still consistent with the optimal solution, indicating that the method provided in the present application has strong ability in vehicle scheduling.

[0124] Table 1

[0125]

[0126] The results of comparing the method provided in the application with the policy results of the random selection operator are shown in Table 2. As can be seen from the results, the method provided in the application is significantly better than the random method in all examples, and in particular, in examples such as C109 and RC105, the path length is shortened by more than 20%, which reflects the good utilization capability of the method for the spatial structure. Even in the case of the same number of vehicles, the path quality of the random method is still significantly worse than the method provided in the application, which shows that the method provided in the application achieves a better balance between cost and the number of vehicles, and shows stronger robustness.

[0127] Table 2

[0128]

[0129] As Figures 4 to 6 shown, the training convergence processes of the method provided in the application and other reinforcement learning methods (DQN, PPO, Q-learning) in each of the four representative examples in C, R and RC are respectively shown. The yellow dotted line in the figure represents the method provided in the application, the blue solid line represents the Q-learning method, the green dashed line represents the DQN method, and the red dashed line represents the PPO method. The horizontal coordinate is the training batch, and the vertical coordinate is the action value.

[0130] As Figures 4-6 can be seen: C class (centralized distribution example): the method provided in the application converges quickly and has the best final result; DQN performs well; PPO and Q-Learning converge slowly and are unstable; R and RC classes (random distribution and mixed distribution examples): although the examples are more complex, the method provided in the application still shows faster convergence speed and better solution quality, and other methods generally have large fluctuations and early convergence problems. These results further verify the stability and effectiveness of the method provided in the application in various scenarios.

[0131] The reinforcement learning-based vehicle path planning device provided in the application is described below. The reinforcement learning-based vehicle path planning device described below can be correspondingly referred to the reinforcement learning-based vehicle path planning method described above. As Figure 7 shown, the reinforcement learning-based vehicle path planning device provided in the application comprises:

[0132] The first expected return acquisition module 710 is configured to input a state vector at a t time step into a first model to obtain a first expected return value corresponding to a plurality of first sample actions output by the first model, each first sample action corresponds to calling a path planning operator, and the state vector includes a sequence of operators that have been called;

[0133] The action selection module 720 is configured to select one action as a target first sample action from the plurality of first sample actions, update the state vector based on the target first sample action, and obtain a state vector at a t+1 time step;

[0134] The second expected return acquisition module 730 is configured to input the state vector at the t+1 time step into the second model, and acquire a plurality of second sample actions output by the second model and a second expected return value corresponding to each second sample action;

[0135] The target expected return determination module 740 is configured to determine a target expected return value based on the second expected return value, determine a training loss based on the target expected return value and a first expected return value corresponding to the target first sample action, and update parameters of the first model based on the training loss;

[0136] The soft update module 750 is configured to perform soft update on parameters of the second model based on the parameters of the first model after a plurality of time steps;

[0137] The result output module 760 is configured to obtain a vehicle path planning result based on output data of the first model after training.

[0138] Figure 8 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 8As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete communications with each other through the communications bus 840. The processor 810 can invoke a logic instruction in the memory 830 to execute a vehicle path planning method based on reinforcement learning. The vehicle path planning method based on reinforcement learning includes: inputting a state vector at a t time step into a first model, obtaining a first expected return value corresponding to a plurality of first sample actions output by the first model, each first sample action corresponding to invoking a path planning operator, and the state vector including a sequence of operators that have been invoked; selecting one action as a target first sample action from the plurality of first sample actions, updating the state vector based on the target first sample action to obtain a state vector at a t+1 time step; inputting the state vector at the t+1 time step into a second model, obtaining a plurality of second sample actions output by the second model and a second expected return value corresponding to each second sample action; determining a target expected return value based on the second expected return value, determining a training loss based on the target expected return value and a first expected return value corresponding to the target first sample action, and updating parameters of the first model based on the training loss; after a plurality of time steps, soft updating parameters of the second model based on the parameters of the first model; and obtaining a vehicle path planning result based on output data of the first model after training is completed.

[0139] In addition, the logic instruction in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0140] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the vehicle path planning method based on reinforcement learning provided by the above-mentioned methods. The vehicle path planning method based on reinforcement learning comprises: inputting a state vector at a t time step into a first model to obtain first expected return values corresponding to a plurality of first sample actions output by the first model, each first sample action corresponding to calling a path planning operator, and the state vector comprising a sequence of operators that have been called; selecting one action as a target first sample action from the plurality of first sample actions, updating the state vector based on the target first sample action to obtain a state vector at a t+1 time step; inputting the state vector at the t+1 time step into a second model to obtain a plurality of second sample actions output by the second model and second expected return values corresponding to each second sample action; determining a target expected return value based on the second expected return values, determining a training loss based on the target expected return value and a first expected return value corresponding to the target first sample action, updating parameters of the first model based on the training loss; after a plurality of time steps, performing soft updating of parameters of the second model based on the parameters of the first model; and obtaining a vehicle path planning result based on output data of the first model after training is completed.

[0141] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the vehicle path planning method based on reinforcement learning provided by the above-mentioned methods. The vehicle path planning method based on reinforcement learning comprises: inputting a state vector at a t time step into a first model to obtain first expected return values corresponding to a plurality of first sample actions output by the first model, each first sample action corresponding to calling a path planning operator, and the state vector comprising a sequence of operators that have been called; selecting one action as a target first sample action from the plurality of first sample actions, updating the state vector based on the target first sample action to obtain a state vector at a t+1 time step; inputting the state vector at the t+1 time step into a second model to obtain a plurality of second sample actions output by the second model and second expected return values corresponding to each second sample action; determining a target expected return value based on the second expected return values, determining a training loss based on the target expected return value and a first expected return value corresponding to the target first sample action, updating parameters of the first model based on the training loss; after a plurality of time steps, performing soft updating of parameters of the second model based on the parameters of the first model; and obtaining a vehicle path planning result based on output data of the first model after training is completed.

[0142] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0143] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0144] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for vehicle path planning based on reinforcement learning, characterized in that, The method comprises the following steps: inputting a state vector at a t time step into a first model to obtain first expected return values corresponding to a plurality of first sample actions output by the first model, each of the first sample actions corresponding to calling a path planning operator, and the state vector comprising a sequence of operators that have been called; selecting one of the plurality of first sample actions as a target first sample action, and updating the state vector based on the target first sample action to obtain the state vector at a t+1 time step; inputting the state vector at the t+1 time step into a second model to obtain a plurality of second sample actions output by the second model and second expected return values corresponding to each of the second sample actions; determining a target expected return value based on the second expected return values, determining a training loss based on the target expected return value and the first expected return value corresponding to the target first sample action, and updating parameters of the first model based on the training loss; after a plurality of time steps, performing soft updating of parameters of the second model based on the parameters of the first model; obtaining a vehicle path planning result based on output data of the first model after training.

2. The reinforcement learning-based vehicle path planning method according to claim 1, characterized in that, Each of the path planning operators corresponds to a path adjustment operation, and each of the path planning operators is associated with a parameter k, which is used to control a size of the path adjustment operation corresponding to the path planning operator.

3. The reinforcement learning-based vehicle path planning method according to claim 1, characterized in that, The selecting one of the plurality of first sample actions as the target first sample action comprises: Based on the first formula, the exploration rate is determined, and the first formula is: wherein, represents the exploration rate corresponding to the t time step, represents an exploration rate reference value, and T is an adjustment parameter, and the size of T is negatively related to the size of the time step; selecting the target first sample action from the plurality of first sample actions based on the exploration rate.

4. The reinforcement learning-based vehicle path planning method according to claim 1, characterized in that, The determining the target expected return value based on the second expected return values comprises: obtaining a second sample action corresponding to the maximum second expected return value as a target second sample action; determining a reward value at the t time step based on effect values of path planning schemes corresponding to the plurality of second sample actions; determining the target expected return value based on the reward value at the t time step.

5. The reinforcement learning-based vehicle path planning method according to claim 4, characterized in that, The determining the reward value at the t time step based on the effect values of the path planning schemes corresponding to the plurality of second sample actions comprises: obtaining a maximum value and an average value of effect improvement values of the path planning schemes corresponding to the plurality of second sample actions, obtaining a reward change trend value reflecting a change trend of the reward values of the latest N time steps; determining the reward value at the t time step based on the maximum value, the average value and the reward change trend value.

6. The reinforcement learning-based vehicle path planning method according to claim 1, characterized in that, The first model is a multi-layer fully connected neural network, and each hidden layer in the first model has a Dropout mechanism.

7. A vehicle path planning apparatus based on reinforcement learning, characterized by, The method comprises the following steps: a first expected return obtaining module is configured to input a state vector at a t time step into a first model to obtain first expected return values corresponding to a plurality of first sample actions output by the first model, each of the first sample actions corresponding to calling a path planning operator, and the state vector comprising a sequence of operators that have been called; The action selection module is configured to select one action from the plurality of first sample actions as a target first sample action, update the state vector based on the target first sample action, and obtain the state vector at the t+1 time step; The second expected return acquisition module is configured to input the state vector at the t+1 time step into a second model, acquire a plurality of second sample actions output by the second model and a second expected return value corresponding to each second sample action; The target expected return determination module is configured to determine a target expected return value based on the second expected return value, determine a training loss based on the target expected return value and the first expected return value corresponding to the target first sample action, and update parameters of the first model based on the training loss; The soft update module is configured to perform soft update on parameters of the second model based on the parameters of the first model after a plurality of time steps; The result output module is configured to obtain a vehicle path planning result based on output data of the first model after training.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The computer program is executed by the processor to implement the vehicle path planning method based on reinforcement learning according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the vehicle path planning method based on reinforcement learning according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the vehicle path planning method based on reinforcement learning according to any one of claims 1 to 6.

Images