Method and device for solving traveling salesman problem based on knowledge-guided deep reinforcement learning
Through the knowledge-guided deep reinforcement learning method of sliding average value and adaptive baseline combined with RIDGE algorithm, the UAV path planning is transformed into an undirected complete graph, solving the problem of insufficient generalization ability of deep reinforcement learning in different scale problems, and realizing the Better scale adaptability and quality of resolution.
Patent Information
- Application Number
- CN202411040310.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-07-31
AI Technical Summary
Existing deep reinforcement learning methods are limited by environmental consistency assumptions in UAV path planning and cannot effectively generalize to practical problem instances of different sizes, and light encoder and heavy decoder models perform poorly in unsupervised learning. , cannot provide stable solution quality assurance.
Using a deep reinforcement learning method based on knowledge guidance, the sliding average algorithm and adaptive baseline combined with RIDGE algorithm are used to transform the traveler problem into an undirected complete graph, improve the generalization ability and decision-making prudence of the model, and introduce theoretical boundaries to accelerate convergence. and improve solution quality.
The scale generalization ability of the model is improved, ensuring good performance is shown in the problem of path planning of different scales, solving the contradiction between environmental consistency assumptions and scale generalization ability, and enhancing the ability to adapt to unseen situations.
Smart Images

Figure CN118569466B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of traveling salesman problem, and in particular to a method and device for solving the traveling salesman problem based on knowledge-guided deep reinforcement learning. Background Art
[0002] The traveling salesman problem (TSP problem) is widely used in vehicle routing, warehouse management, drone path planning, etc. The goal is to find the shortest path and distance matrix among given n cities so that the traveler can visit each city once and return to the starting city. Due to various extensions and additional constraints of actual scenarios such as time windows, vehicle capacity, and online planning, the solution of the TSP problem is more complicated.
[0003] In the actual use of drones to collect wildlife information, how to plan the optimal path to efficiently collect data from each monitoring point can be regarded as a classic traveling salesman problem: each monitoring point is a city that needs to be reached, and the drone goes to obtain data from each monitoring point and returns to the initial point. In addition, during the collection process, due to the limitations of the drone's own energy and the life cycle of the monitoring point, the number of monitoring points that can be visited may fluctuate, which places certain requirements on the generalization performance of the path planning algorithm. Therefore, a stable algorithm with good generalization performance is needed to plan the path of the drone.
[0004] At present, deep reinforcement learning (DRL) and light encoder and heavy decoder models (LEHD) are usually used to solve the TSP problem of drones collecting wildlife information at different monitoring points. Among them, since the deep reinforcement learning (DRL) of this TSP problem is limited by the environmental consistency assumption, it is usually trained on fixed-scale instances, so it cannot be used for actual problem instances that exceed the training scale. In this way, when drones are needed to collect wildlife information from a large number of monitoring points, it cannot provide stable solution quality assurance. The light encoder and heavy decoder model (LEHD) shows strong generalization ability when there is a labeled training data set, but it is not suitable for unsupervised learning methods such as deep reinforcement learning (DRL). Summary of the invention
[0005] The purpose of the present invention is to provide a method and device for solving the traveling salesman problem based on knowledge-guided deep reinforcement learning. By designing a reinforcement algorithm in combination with theoretical results, the contradiction between the environmental consistency assumption of the reinforcement learning method and the scale generalization capability requirement is bridged, thereby providing an optimal solution to the TSP problem of using drones to collect wildlife information at monitoring points.
[0006] The present invention is achieved through the following technical solutions:
[0007] In a first aspect, a method for solving a traveling salesman problem based on knowledge-guided deep reinforcement learning is provided, wherein the method is used to solve a path planning problem of a drone to be monitored, and comprises the following steps:
[0008] S100, obtaining a training sample set and a test sample set, wherein the training sample set is a randomly generated monitoring point geographic location data set, and the test sample set is a real monitoring point geographic location data set that the drone needs to actually monitor;
[0009] S200, training a deep reinforcement learning model, wherein the deep reinforcement learning model is used to determine the shortest path from the drone to each test sample set, including the following steps:
[0010] S210, inputting a training sample set into the deep reinforcement learning model, creating each training sample in the training sample set as an undirected complete graph, and obtaining a training undirected graph set; each training undirected graph in the training undirected graph set corresponds one-to-one to a training sample;
[0011] S220, based on the training undirected atlas, using a sliding average algorithm, determining a current decision-making ability of the deep reinforcement learning model;
[0012] S230, combining the current decision-making ability and theoretical boundary of the deep reinforcement learning model to obtain an adaptive baseline;
[0013] S240, determining the RIDGE algorithm through the adaptive baseline to obtain a trained deep reinforcement learning model;
[0014] S300, testing the deep reinforcement learning model, including the following steps:
[0015] S310, inputting a test sample set into the trained deep reinforcement learning model, creating each test sample in the test sample set as an undirected complete graph, and obtaining a test undirected graph set; each test undirected graph in the test undirected graph set corresponds one-to-one to a test sample;
[0016] S320: Determine the optimal solution of the test undirected graph through the trained deep reinforcement learning model to obtain the shortest path from the drone to each test sample set.
[0017] Specifically, the formula of the sliding average algorithm is as follows:
[0018] ;
[0019] in, represents a sliding average; Indicates the length of the sliding window; Indicates the number of training samples in the training sample set; represents the first Batch training sample set; represents the first Batch training sample set; Indicates The first batch The solution set of the path planning problem for training samples; Indicates The first batch The average reward of the set of solutions to the path planning problem for training samples.
[0020] Specifically, the The calculation formula is as follows:
[0021] ;
[0022] in, Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample The total cost of the solution to the path planning problem; Represents the number of solutions to the path planning problem in each training sample.
[0023] Specifically, combining the current decision-making ability and theoretical boundary of the deep reinforcement learning model to obtain an adaptive baseline, the specific steps include:
[0024] S231, calling the baseline of the previous moment, and comparing the baseline of the previous moment with the sliding average value;
[0025] S232: If the baseline at the previous moment is less than or equal to the sliding average, the baseline at the previous moment is used as a reference and adjusted upward to obtain the current baseline;
[0026] S233: If the baseline at the previous moment is greater than the sliding average, the sliding average is used as a benchmark to adjust downward to obtain the current baseline.
[0027] Specifically, the formula of the adaptive baseline is as follows:
[0028] ;
[0029] in, Indicates the shortest path length in the current monitored area; represents the learning rate, which measures the degree to which theoretical knowledge guides the deep reinforcement learning model in solving the path planning of the monitoring area; Indicates the shortest path length of the area to be monitored in the previous round; It represents the lower bound of the theoretical shortest path length in the monitored area. The lower bound related constant representing the theoretical shortest path length in the monitored area, Indicates the number of monitoring points in the training area to be monitored.
[0030] Specifically, the formula of the RIDGE algorithm is as follows:
[0031] ;
[0032] in, represents the gradient operator; Represents the current deep reinforcement learning model Solve the cumulative benefits of the training samples below; Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample A reward for solving a path planning problem; Indicates that in the training sample Next, select The first batch The first training sample A strategy for solving the path planning problem.
[0033] In a second aspect, a traveling salesman problem solving device based on knowledge-guided deep reinforcement learning is provided, wherein the device is used to solve the path planning problem of a drone to be monitored, and the solving device is used to implement the above-mentioned solving method;
[0034] The solving device comprises:
[0035] A collection module, used to obtain a training sample set and a test sample set, wherein the training sample set is a randomly generated monitoring point geographic location data set, and the test sample set is an actual monitoring point geographic location data set that the drone needs to actually monitor;
[0036] A processing module, which is connected to the acquisition module and is used to perform the following operations:
[0037] S200: Training a deep reinforcement learning model, where the deep reinforcement learning model is used to determine the shortest path to each test sample set, including the following steps:
[0038] S210, inputting a training sample set into the deep reinforcement learning model, creating each training sample in the training sample set as an undirected complete graph, and obtaining a training undirected graph set;
[0039] Each training undirected graph in the training undirected graph set corresponds to a training sample one by one;
[0040] S220, based on the training undirected atlas, using a sliding average algorithm, determining a current decision-making ability of the deep reinforcement learning model;
[0041] S230, combining the current decision-making ability and theoretical boundary of the deep reinforcement learning model to obtain an adaptive baseline;
[0042] S240, determining the RIDGE algorithm through the adaptive baseline to obtain a trained deep reinforcement learning model;
[0043] S300, testing the deep reinforcement learning model, including the following steps:
[0044] S310, inputting a test sample set into the trained deep reinforcement learning model, creating each test sample in the test sample set as an undirected complete graph, and obtaining a test undirected graph set; each test undirected graph in the test undirected graph set corresponds one-to-one to a test sample;
[0045] S320: Determine the optimal solution of the test undirected graph through the trained deep reinforcement learning model to obtain the shortest path from the drone to each test sample set.
[0046] Furthermore, the processing module is also used to calculate a sliding average;
[0047] The formula of the sliding average algorithm is as follows:
[0048] ;
[0049] in, represents a sliding average; Indicates the length of the sliding window; Indicates the number of training samples in the training sample set; represents the first Batch training sample set; represents the first Batch training sample set; Indicates The first batch The solution set of the path planning problem for training samples; Indicates The first batch The average reward of the path planning problem solution set for training samples;
[0050] Said The calculation formula is as follows:
[0051] ;
[0052] in, Indicates The first batch The first training sample Solution to the path planning problem; Indicates the first The first batch The first training sample The total cost of the solution to the path planning problem; Represents the number of solutions to the path planning problem in each training sample.
[0053] Furthermore, the processing module is further used to combine the current decision-making ability and theoretical boundary of the deep reinforcement learning model to obtain an adaptive baseline, and the specific steps include:
[0054] S231, calling the baseline of the previous moment, and comparing the baseline of the previous moment with the sliding average value;
[0055] S232: If the baseline at the previous moment is less than or equal to the sliding average, the baseline at the previous moment is used as a reference and adjusted upward to obtain the current baseline;
[0056] S233: If the baseline at the previous moment is greater than the sliding average, the sliding average is used as a benchmark to adjust downward to obtain the current baseline;
[0057] The formula of the adaptive baseline is as follows:
[0058] ;
[0059] in, Indicates the shortest path length in the current monitored area; represents the learning rate, which measures the degree to which theoretical knowledge guides the deep reinforcement learning model in solving the path planning of the monitoring area; Indicates the shortest path length of the area to be monitored in the previous round; It represents the lower bound of the theoretical shortest path length in the monitored area. The lower bound related constant representing the theoretical shortest path length in the monitored area, Indicates the number of monitoring points in the training area to be monitored.
[0060] Furthermore, the server is also used to update the RIDGE algorithm, and the updated RIDGE algorithm formula is as follows:
[0061] ;
[0062] in, represents the gradient operator; Represents the current deep reinforcement learning model Solve the cumulative benefits of the training samples below; Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample A reward for solving a path planning problem; Indicates that in the training sample Next, select The first batch The first training sample A strategy for solving the path planning problem.
[0063] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0064] In solving the path planning problem of the drone to be monitored, an undirected complete graph is created for the training data and the test data to represent the environment of the traveling salesman problem. Due to the variability of the traveling salesman problem, the created undirected complete graph is variable, and the deep reinforcement learning model improves its generalization ability after training and learning; by converting the path planning problem of the drone to be monitored into the form of an undirected complete graph, and combining the sliding average algorithm and the adaptive baseline, the deep reinforcement learning model can learn a more general solution strategy, not just a strategy for problems of a specific scale, which enables the model to show better scale generalization ability when facing path planning problems of different scales; the RIDGE algorithm is determined based on the adaptive baseline, which combines the current Pre-decision-making ability and theoretical boundaries mean that when making decisions, the deep reinforcement learning model not only considers its own decision-making ability, but also refers to the theoretical optimal boundary, making the model more cautious and rational in the decision-making process, avoiding suboptimal solutions caused by over-reliance on the current state or historical experience; at the same time, since the theoretical boundaries are taken into consideration, the model is also more efficient and targeted when exploring new strategies; the introduction of knowledge guidance in deep reinforcement learning means that the deep reinforcement learning model not only relies on data-driven learning methods during training, but also combines prior knowledge or theoretical knowledge in the field; in path planning problems, by incorporating theoretical knowledge into the model training process, the convergence speed of the model can be accelerated, the quality of the solution can be improved, and the model's ability to adapt to unprecedented situations can be enhanced.
[0065] In summary, the above methods can not only improve the performance of the model in the test environment, but also enhance the scale generalization ability of the model, so that the model can show good performance when facing traveling salesman problems of different scales, and solve the contradiction between the environmental consistency assumption and the scale generalization ability requirement in the path planning problem of the drone to be monitored. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:
[0067] Figure 1 This is the main flow chart of the method for solving the traveling salesman problem based on knowledge-guided deep reinforcement learning of the present invention. DETAILED DESCRIPTION
[0068] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.
[0069] In the following embodiments, how to plan the optimal path to collect data from each monitoring point in the scenario of using a drone to collect information on wild animals is used as an example to illustrate. Each monitoring point is a monitoring point to be reached, and the drone obtains data from each monitoring point and returns to the initial point.
[0070] In this scenario, the training sample set is a randomly generated monitoring point geographic location data set. The monitoring point is used to collect information about wild animals in the area, such as the latitude and longitude data of the monitoring point geographic location data. The test sample set is an actual monitoring point geographic location data set, such as a national park monitoring station. In other words, the training sample set is a randomly generated monitoring point geographic location data set, and the test sample set is an actual monitoring point geographic location data set.
[0071] In addition, it should be understood that the traveling salesman problem solving method based on knowledge-guided deep reinforcement learning of the present invention can also be applied to other fields, such as earthquake search and other scenarios. Example
[0072] Combination Figure 1 This embodiment 1 provides a method for solving the traveling salesman problem based on knowledge-guided deep reinforcement learning, the method comprising the following steps:
[0073] S100, obtaining a training sample set and a test sample set, wherein the training sample set is a randomly generated monitoring point geographic location data set, and the test sample set is an actual monitoring point geographic location data set that the drone needs to actually monitor. In the present invention, the training sample set can be a randomly generated coordinate data set, and the test sample set is a coordinate data set of actual monitoring points.
[0074] S200, training a deep reinforcement learning model, wherein the deep reinforcement learning model is used to determine the shortest path from the drone to each test sample set, including the following steps:
[0075] S210. Input a training sample set into the above-mentioned deep reinforcement learning model, and create each training sample in the training sample set as an undirected complete graph to obtain a training undirected graph set; each training undirected graph in the above-mentioned training undirected graph set corresponds one-to-one to a training sample.
[0076] Training an undirected graph set refers to using an undirected graph model for data modeling and learning in deep learning. An undirected graph model (such as a Markov random field) is a type of statistical model used to describe the interdependence between variables. In the present invention, training an undirected graph set can describe the interdependence of randomly generated monitoring point geographic location data sets.
[0077] S220. Based on the above-mentioned training undirected graph set, use the sliding average algorithm to determine the current decision-making ability of the deep reinforcement learning model; wherein, the current decision-making ability of the deep reinforcement learning model refers to the ability of the model to take the best action according to the current state in a given environment.
[0078] S230. Combining the current decision-making ability and theoretical boundary of the above deep reinforcement learning model, an adaptive baseline is obtained; wherein the theoretical boundary refers to the best performance that the deep reinforcement learning model can achieve in theory, and the adaptive baseline is used to reduce the variance of the estimate, thereby accelerating the learning process and improving the stability of the algorithm. The "baseline" here usually refers to a value (or function) that is used to compare with the actual reward to calculate the advantage (i.e., how good or bad a certain action is relative to the average performance).
[0079] S240, determining the RIDGE algorithm through the above-mentioned adaptive baseline to obtain a trained deep reinforcement learning model;
[0080] S300, testing the deep reinforcement learning model, including the following steps:
[0081] S310, inputting a test sample set into the trained deep reinforcement learning model, creating each test sample in the test sample set as an undirected complete graph, and obtaining a test undirected graph set; each test undirected graph in the test undirected graph set corresponds one-to-one to a test sample;
[0082] S320. Determine the optimal solution of the test undirected graph through the trained deep reinforcement learning model to obtain the shortest path from the drone to each test sample set.
[0083] Since deep reinforcement learning models usually assume that the environment is stable and consistent, that is, the environmental characteristics observed by the model during training should also remain consistent during testing; however, when drones monitor ground conditions, the monitoring points and their number may change due to the complex and changeable environment of the monitoring points. It is necessary to effectively plan the monitoring locations of drones across scales, and this consistency is difficult to fully guarantee. Therefore, the above method is used to improve the generalization ability of the model. By creating an undirected complete graph to represent the environment of the drone's path planning problem for the actual detection points that need to be actually monitored, due to the variability of the drone's path planning problem for the actual detection points that need to be actually monitored, the created undirected complete graph is variable, and the deep reinforcement learning model improves its generalization ability after training and learning; by converting the drone's path planning problem for the actual detection points that need to be actually monitored into the form of an undirected complete graph, and combining the sliding average algorithm and the adaptive baseline, the deep reinforcement learning model can learn a more general solution strategy, not just a strategy for specific scale problems, which enables the model to show better scale generalization ability when facing path planning problems of different scales; the RIDGE algorithm is determined based on the adaptive baseline, This baseline combines the current decision-making ability and theoretical boundaries of deep reinforcement learning models, which means that when making decisions, deep reinforcement learning models not only consider their own decision-making ability, but also refer to the theoretical optimal boundaries, making the model more cautious and rational in the decision-making process, avoiding suboptimal solutions caused by over-reliance on the current state or historical experience; at the same time, since the theoretical boundaries are taken into consideration, the model is also more efficient and targeted when exploring new strategies; the introduction of knowledge guidance in deep reinforcement learning means that the deep reinforcement learning model not only relies on data-driven learning methods during training, but also combines prior knowledge or theoretical knowledge in the field; in path planning problems, by incorporating this knowledge into the model training process, the convergence speed of the model can be accelerated, the quality of the solution can be improved, and the model's ability to adapt to unseen situations can be enhanced.
[0084] In summary, the above methods can not only improve the performance of the model in the test environment, but also enhance the scale generalization ability of the model, so that the model can show good performance when facing path planning problems of different scales.
[0085] Example 2
[0086] Based on Example 1, the formula of the sliding average algorithm is as follows:
[0087]
[0088] in, represents a sliding average; represents the length of the sliding window, ; represents the number of training samples in the training sample set. Specifically, in the present invention, the training samples are randomly generated monitoring point geographic location data sets; represents the first A batch training sample set, that is, the batch index currently input into the deep reinforcement learning model; represents the first A batch of training sample sets, that is, the starting batch index; Indicates The first batch The solution set of the path planning problem for training samples; Indicates The first batch The average reward of the path planning problem solution set for training samples. Here, the training samples are randomly generated monitoring point location datasets.
[0089] The above sliding average algorithm is usually used to smooth data sequences in order to better understand the changing trend of the data; in the context of deep reinforcement learning, the sliding average algorithm is used to estimate the performance of the deep reinforcement learning model, especially when the deep reinforcement learning model is trained on multiple batches of training sample sets.
[0090] In deep reinforcement learning, the above sliding average can be used to: monitor the training progress of the deep reinforcement learning model. By observing the changes in the sliding average, you can understand whether the deep reinforcement learning model is improving or whether problems such as overfitting have occurred; use the sliding average to set an adaptive baseline, which can be dynamically adjusted as the deep reinforcement learning model is trained, which helps to determine a reasonable goal so that the deep reinforcement learning model can continue to improve its performance; based on the changes in the sliding average, decide whether to adjust the learning rate or other hyperparameters to accelerate the training of the deep reinforcement learning model or improve the performance of the deep reinforcement learning model.
[0091] In summary, by using the sliding average algorithm, we can better evaluate the performance of deep reinforcement learning models during training and make more informed decisions based on this.
[0092] Specific embodiments, the above The calculation formula is as follows:
[0093]
[0094] in, Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample The total cost of the solution to the path planning problem; Represents the number of solutions to the path planning problem in each training sample. Here, the training sample is a randomly generated dataset of monitoring point locations.
[0095] In the traveling salesman problem of the present invention, a solution is usually a sequential list of all monitoring points visited by the drone; in the traveling salesman problem of the present invention, since the solution space of the traveling salesman problem is very large, all possible solutions are usually not calculated, but a heuristic method or search algorithm is used to generate a solution set; then the above formula is used to find the solution with the smallest loss or cost among all candidate solutions, and its negative value is taken.
[0096] In deep reinforcement learning, it is usually desirable to maximize the reward, while the distance or cost in the traveling salesman problem is desirable to be minimized. Specifically, in the traveling salesman problem of the present invention, which uses drones to monitor all actual monitoring points that need to be actually monitored, it is desirable to minimize the path distance; therefore, taking a negative value can transform the minimization problem into a maximization problem, which is consistent with the goal of deep reinforcement learning.
[0097] The total cost is obtained by calculating the sum of the distances of all nodes of the training sample, and the calculation formula is as follows:
[0098]
[0099] in, Indicates The first batch The first training sample The total cost of the solution to the path planning problem; Indicates Batch No. The first of the training samples The Euclidean distance between the node and the first node; Indicates Batch No. The first of the training samples Nodes and The Euclidean distance between nodes.
[0100] Example 3
[0101] On the basis of Example 2, combining the current decision-making ability and theoretical boundary of the above deep reinforcement learning model, an adaptive baseline is obtained, and the specific steps include:
[0102] S231, calling the baseline of the previous moment, and comparing the baseline of the previous moment with the sliding average value;
[0103] S232: If the baseline at the previous moment is less than or equal to the sliding average, the baseline at the previous moment is used as a reference and adjusted upward to obtain the current baseline;
[0104] S233: If the baseline at the previous moment is greater than the sliding average, the sliding average is used as a benchmark to adjust downward to obtain the current baseline.
[0105] In a specific embodiment, the formula of the above adaptive baseline is as follows:
[0106]
[0107] in, Indicates the current baseline; represents a hyperparameter that measures the degree to which theoretical knowledge guides the training of deep reinforcement learning models; Indicates the baseline of the previous moment; represents the lower bound of the theoretical boundary, The lower bound correlation constant representing the theoretical boundary, represents the number of nodes in the training undirected graph. Specifically, in the context of the present invention, Indicates the shortest path length in the current monitored area; represents the learning rate, which measures the degree to which theoretical knowledge guides the deep reinforcement learning model in solving the path planning of the monitoring area; Indicates the shortest path length of the area to be monitored in the previous round; It represents the lower bound of the theoretical shortest path length in the monitored area. The lower bound related constant representing the theoretical shortest path length in the monitored area, Indicates the number of monitoring points in the training area to be monitored.
[0108] In the field of deep learning, "baseline" refers to an initial, simple model or a set of set standards used as a reference for subsequent model performance comparison. The baseline model is usually a model that is easy to implement and has acceptable performance. Its role is to provide a minimum standard for new methods or models to evaluate whether the new model is truly an improvement.
[0109] Combining the current decision-making capabilities and theoretical boundaries of the above deep reinforcement learning model, a strategy for adaptive baseline adjustment is given; first, the baseline of the previous moment is called and compared with the sliding average.
[0110] When the baseline at the previous moment is less than or equal to the current sliding average, the current baseline is the weighted sum of the baseline at the previous moment and the sliding average, where the weights are given by α and 1-α, allowing the baseline to move closer to the current performance indicator while maintaining a certain stability.
[0111] When the baseline at the previous moment is greater than the current sliding average, the current baseline is affected by the lower bound of the theoretical boundary and adjusted in combination with the sliding average, ensuring that the baseline does not overestimate the performance of the model while allowing it to make certain adjustments based on the current performance indicators.
[0112] In this way, the adaptive baseline can be adjusted according to the performance of the model and theoretical knowledge, thereby providing more reasonable rewards or evaluation criteria during the training process.
[0113] Example 4
[0114] Based on Example 3, the formula of the above RIDGE algorithm is as follows:
[0115]
[0116] in, represents the gradient operator; Represents the current deep reinforcement learning model Solve the cumulative benefits of the training samples below; Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample A reward for solving a path planning problem; Indicates that in the training sample Next, select The first batch The first training sample A strategy for solving the path planning problem.
[0117] Each solution to the path planning problem means starting from the initial node, visiting one node at each step, and selecting the next node from the remaining nodes until returning to the initial node. Specifically, this process can be regarded as the process of the drone visiting the next monitoring point from one monitoring point and finally returning to the initial monitoring point. This process can be regarded as a Markov decision process, and each decision at each step can be expressed as To parameterize, (where, represents training samples; represents the first solution to the traveling salesman problem nodes; Indicated in step before, the nodes that have been visited); therefore, in a given training sample The strategy for selecting the solution to the path planning problem is It can be defined as:
[0118]
[0119] in, Indicates that in a given training sample A strategy for selecting a solution to the path planning problem when .
[0120] For example, if the generated training samples are those that exist within the unit square nodes, whose length does not exceed , select any node in the unit square as the initial node;
[0121] First use , ( represents the shortest path length through all nodes of the training sample), find the shortest path length through all nodes of the training sample;
[0122] Reuse , calculate the upper bound related constant of the theoretical boundary and lower bound related constants ;
[0123] The RIDGE algorithm is improved by using the information of the shortest path range.
[0124] In order to obtain an effective baseline, it is necessary to balance the solving ability and theoretical boundaries of current deep reinforcement learning models; therefore, the decision-making ability of deep reinforcement learning models should be determined first, and then an adaptive baseline should be designed.
[0125] (1) The method for determining the decision-making ability of a deep reinforcement learning model is as follows:
[0126] In the past, it was usually randomly generated training samples, and extract Path In these studies, the baseline for each training sample is obtained by taking the current batch graph However, since the training samples are randomly generated, the shortest path length will fluctuate in different training samples. The baseline derived from a single training sample cannot capture the model's ability across different training samples and batches.
[0127] To solve this problem, the present invention uses a sliding average ( ,in Indicates that during training, The sliding window is used to measure the decision-making ability of the current deep reinforcement learning model, and the sliding average algorithm is used to calculate the sliding average. The introduction of the sliding window helps to alleviate the problem of inaccurate model ability estimation caused by randomly selecting training samples from different batches.
[0128] (2) The method for designing an adaptive baseline is as follows:
[0129] Combining the capabilities of the current model with theoretical constraints, it can be expressed as:
[0130]
[0131] Among them, Assume that it is the upper bound of the theoretical boundary, where is the upper bound related constant of the theoretical boundary of the training sample, Let be the lower bound of the theoretical boundary, where is the lower bound related constant of the theoretical boundary of the training samples, is the number of nodes in the training sample; introduce hyperparameters To measure the degree to which theoretical knowledge guides model training, that is, if ,but ;if ,but , and Indicates the magnitude of adaptive adjustment.
[0132] If the baseline of the previous round is less than the sliding average, then and sliding average On the contrary, if If it is greater than the sliding average, the lower bound of the theoretical shortest path and the current sliding average are used as guidance to adjust downward; therefore, the adaptive baseline The formula is:
[0133]
[0134] Adaptively adjust the baseline based on a sliding average of the shortest path lengths during training , and guided by the theoretical boundary, this adaptive adjustment ensures the theoretically optimal stable downward trend of the traveling salesman problem, thereby enhancing resistance to local disturbances and representing an evaluation metric for deep reinforcement learning models, while The model is trained during The impact of change forms a reciprocal relationship between the two; and The values gradually converge and influence each other; only when and The model can converge only when the baseline is close to the same. When it converges to the theoretical optimum, the model reaches the theoretical optimum.
[0135] In addition, in order to ensure better convergence of the model, the baseline is introduced As a measure, The present invention is concerned with the difference between and This is because different training samples produce different reward values during model training; in addition, in the early and late stages of training, the gap between the sliding mean and the baseline cannot be compared on the same scale. Therefore, by applying non-dimensionalization, the impact of the difference can be more effectively affected, which is beneficial to optimizing the learning process of the model.
[0136] Example 5
[0137] This embodiment 5 provides a traveling salesman problem solving device based on knowledge-guided deep reinforcement learning, and the solving device is used to implement the above-mentioned solving method;
[0138] The solving device comprises:
[0139] A collection module, used to obtain a training sample set and a test sample set, wherein the training sample set is a randomly generated monitoring point geographic location data set, and the test sample set is an actual monitoring point geographic location data set that the drone needs to actually monitor;
[0140] A processing module, which is connected to the acquisition module and is used to perform the following operations:
[0141] S200: Training a deep reinforcement learning model, where the deep reinforcement learning model is used to determine the shortest path to each test sample set, including the following steps:
[0142] S210, inputting a training sample set into the above-mentioned deep reinforcement learning model, creating each training sample in the training sample set as an undirected complete graph, and obtaining a training undirected graph set; each training undirected graph in the above-mentioned training undirected graph set corresponds one-to-one to a training sample;
[0143] S220, based on the above training undirected atlas, using a sliding average algorithm, determining the current decision-making ability of the deep reinforcement learning model;
[0144] S230, combining the current decision-making ability and theoretical boundary of the above deep reinforcement learning model to obtain an adaptive baseline;
[0145] S240, determining the RIDGE algorithm through the above-mentioned adaptive baseline to obtain a trained deep reinforcement learning model;
[0146] S300, testing the deep reinforcement learning model, including the following steps:
[0147] S310, inputting a test sample set into the trained deep reinforcement learning model, creating each test sample in the test sample set as an undirected complete graph, and obtaining a test undirected graph set; each test undirected graph in the test undirected graph set corresponds one-to-one to a test sample;
[0148] S320. Determine the optimal solution of the test undirected graph through the trained deep reinforcement learning model to obtain the shortest path from the drone to each test sample set.
[0149] Example 6
[0150] Based on Example 5, the processing module is further used to calculate a sliding average value;
[0151] The formula for the above sliding average algorithm is as follows:
[0152]
[0153] in, represents a sliding average; Indicates the length of the sliding window; Indicates the number of training samples in the training sample set; represents the first Batch training sample set; represents the first Batch training sample set; Indicates The first batch The solution set of the path planning problem for training samples; Indicates The first batch The average reward of the path planning problem solution set for training samples;
[0154] Above The calculation formula is as follows:
[0155]
[0156] in, Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample The total cost of the solution to the path planning problem; Represents the number of solutions to the path planning problem in each training sample.
[0157] Example 7
[0158] On the basis of Example 6, the processing module is further used to combine the current decision-making ability and theoretical boundary of the deep reinforcement learning model to obtain an adaptive baseline, and the specific steps include:
[0159] S231, calling the baseline of the previous moment, and comparing the baseline of the previous moment with the sliding average value;
[0160] S232: If the baseline at the previous moment is less than or equal to the sliding average, the baseline at the previous moment is used as a reference and adjusted upward to obtain the current baseline;
[0161] S233: If the baseline at the previous moment is greater than the sliding average, the sliding average is used as a benchmark to adjust downward to obtain the current baseline;
[0162] The formula for the above adaptive baseline is as follows:
[0163]
[0164] in, Indicates the current baseline; represents a hyperparameter that measures the degree to which theoretical knowledge guides the training of deep reinforcement learning models; Indicates the baseline of the previous moment; represents the lower bound of the theoretical boundary, The lower bound correlation constant representing the theoretical boundary, represents the number of nodes in the training undirected graph. More specifically, in the traveling salesman problem of drone path planning, Indicates the shortest path length in the current monitored area; represents the learning rate, which measures the degree to which theoretical knowledge guides the deep reinforcement learning model in solving the path planning of the monitoring area; Indicates the shortest path length of the area to be monitored in the previous round; It represents the lower bound of the theoretical shortest path length in the monitored area. The lower bound related constant representing the theoretical shortest path length in the monitored area, Indicates the number of monitoring points in the training area to be monitored.
[0165] Example 8
[0166] Based on Example 7, the processing module is further used to update the RIDGE algorithm. The updated RIDGE algorithm formula is as follows:
[0167]
[0168] in, represents the gradient operator; Represents the current deep reinforcement learning model Solve the cumulative benefits of the training samples below; Indicates The first batch The first training sample Processing module problem solution; Indicates The first batch The first training sample Rewards for solving processing module problems; Indicates that in the training sample Next, select The first batch The first training sample A strategy for solving modular problems.
[0169] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for planning the path of a drone to be monitored based on the traveling salesman problem solution guided by knowledge deep reinforcement learning, the method is based on the traveling salesman problem solution guided by knowledge deep reinforcement learning, and is characterized in that: The following steps are involved: S100, obtaining a training sample set and a test sample set, wherein the training sample set is a randomly generated monitoring point geographic location data set, and the test sample set is a real monitoring point geographic location data set that the drone needs to actually monitor; S200, training a deep reinforcement learning model, wherein the deep reinforcement learning model is used to determine the shortest path from the drone to each test sample set, including the following steps: S210, inputting a training sample set into the deep reinforcement learning model, creating each training sample in the training sample set as an undirected complete graph, and obtaining a training undirected atlas; each training undirected graph in the training undirected atlas corresponds to a training sample one by one; the training undirected atlas describes the interdependence of the randomly generated monitoring point geographic location data sets; S220, based on the training undirected atlas, using a sliding average algorithm, determining a current decision-making ability of the deep reinforcement learning model; S230, combining the current decision-making ability and theoretical boundary of the deep reinforcement learning model to obtain an adaptive baseline; S240, determining the RIDGE algorithm through the adaptive baseline to obtain a trained deep reinforcement learning model; The formula of the RIDGE algorithm is as follows: ; in, represents the gradient operator; Represents the current deep reinforcement learning model Solve the cumulative benefits of the training samples below; Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample A reward for solving a path planning problem; Indicates that in the training sample Next, select The first batch The first training sample A strategy for solving the path planning problem; S300, testing the deep reinforcement learning model, including the following steps: S310, inputting a test sample set into the trained deep reinforcement learning model, creating each test sample in the test sample set as an undirected complete graph, and obtaining a test undirected graph set; each test undirected graph in the test undirected graph set corresponds one-to-one to a test sample; S320: Determine the optimal solution of the test undirected graph through the trained deep reinforcement learning model to obtain the shortest path from the drone to each test sample set.
2. According to claim 1, a method for solving the traveling salesman problem of a drone for monitoring a path planning of a drone based on knowledge-guided deep reinforcement learning, characterized in that: The formula of the sliding average algorithm is as follows: ; in, represents a sliding average; Indicates the length of the sliding window; Indicates the number of training samples in the training sample set; represents the first Batch training sample set; represents the first Batch training sample set; Indicates The first batch The solution set of the path planning problem for training samples; Indicates The first batch The average reward of the set of solutions to the path planning problem for training samples.
3. According to claim 2, a method for solving the traveling salesman problem of a drone to be monitored based on knowledge-guided deep reinforcement learning is characterized in that: Said The calculation formula is as follows: ; in, Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample The total cost of the solution to the path planning problem; Represents the number of solutions to the path planning problem in each training sample.
4. According to claim 3, a method for solving the traveling salesman problem of a drone for monitoring a location based on knowledge-guided deep reinforcement learning is characterized in that: Combining the current decision-making ability and theoretical boundaries of the deep reinforcement learning model, an adaptive baseline is obtained. The specific steps include: S231, calling the baseline of the previous moment, and comparing the baseline of the previous moment with the sliding average value; S232: If the baseline at the previous moment is less than or equal to the sliding average, the baseline at the previous moment is used as a reference and adjusted upward to obtain the current baseline; S233: If the baseline at the previous moment is greater than the sliding average, the sliding average is used as a benchmark to adjust downward to obtain the current baseline.
5. According to claim 4, a method for solving the traveling salesman problem of a drone for monitoring a path planning of a drone based on knowledge-guided deep reinforcement learning, characterized in that: The formula of the adaptive baseline is as follows: ; in, Indicates the shortest path length in the current monitored area; represents the learning rate, which measures the degree to which theoretical knowledge guides the deep reinforcement learning model in solving the path planning of the monitoring area; Indicates the shortest path length of the area to be monitored in the previous round; It represents the lower bound of the theoretical shortest path length in the monitored area. The lower bound related constant representing the theoretical shortest path length in the monitored area, Indicates the number of monitoring points in the training area to be monitored.
6. A device for planning the path of a drone to be monitored based on the traveling salesman problem solved by knowledge-guided deep reinforcement learning, characterized in that: The solution device is used to implement the solution method described in any one of claims 1 to 5; The solving device comprises: A collection module, used to obtain a training sample set and a test sample set, wherein the training sample set is a randomly generated monitoring point geographic location data set, and the test sample set is an actual monitoring point geographic location data set that the drone needs to actually monitor; A processing module, which is connected to the acquisition module and is used to perform the following operations: S200: Training a deep reinforcement learning model, where the deep reinforcement learning model is used to determine the shortest path to each test sample set, including the following steps: S210, inputting a training sample set into the deep reinforcement learning model, creating each training sample in the training sample set as an undirected complete graph, and obtaining a training undirected graph set; Each training undirected graph in the training undirected graph set corresponds to a training sample one by one; S220, based on the training undirected atlas, using a sliding average algorithm, determining a current decision-making ability of the deep reinforcement learning model; S230, combining the current decision-making ability and theoretical boundary of the deep reinforcement learning model to obtain an adaptive baseline; S240, determining the RIDGE algorithm through the adaptive baseline to obtain a trained deep reinforcement learning model; S300, testing the deep reinforcement learning model, including the following steps: S310, inputting a test sample set into the trained deep reinforcement learning model, creating each test sample in the test sample set as an undirected complete graph, and obtaining a test undirected graph set; each test undirected graph in the test undirected graph set corresponds one-to-one to a test sample; S320: Determine the optimal solution of the test undirected graph through the trained deep reinforcement learning model to obtain the shortest path from the drone to each test sample set.
7. The device for planning the path of a drone to be monitored based on the traveling salesman problem solving method guided by knowledge in deep reinforcement learning according to claim 6, characterized in that: The processing module is also used to calculate a sliding average; The formula of the sliding average algorithm is as follows: ; in, represents a sliding average; Indicates the length of the sliding window; Indicates the number of training samples in the training sample set; represents the first Batch training sample set; represents the first Batch training sample set; Indicates The first batch The solution set of the path planning problem for training samples; Indicates The first batch The average reward of the path planning problem solution set for training samples; Said The calculation formula is as follows: ; in, Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample The total cost of the solution to the path planning problem; Represents the number of solutions to the path planning problem in each training sample.
8. The device for planning the path of a drone to be monitored based on the traveling salesman problem solving method guided by knowledge in deep reinforcement learning according to claim 7, characterized in that: The processing module is also used to combine the current decision-making ability and theoretical boundary of the deep reinforcement learning model to obtain an adaptive baseline, and the specific steps include: S231, calling the baseline of the previous moment, and comparing the baseline of the previous moment with the sliding average value; S232: If the baseline at the previous moment is less than or equal to the sliding average, the baseline at the previous moment is used as a reference and adjusted upward to obtain the current baseline; S233: If the baseline at the previous moment is greater than the sliding average, the sliding average is used as a benchmark to adjust downward to obtain the current baseline; The formula of the adaptive baseline is as follows: ; in, Indicates the shortest path length in the current monitored area; represents the learning rate, which measures the degree to which theoretical knowledge guides the deep reinforcement learning model in solving the path planning of the monitoring area; Indicates the shortest path length of the area to be monitored in the previous round; It represents the lower bound of the theoretical shortest path length in the monitored area. The lower bound related constant representing the theoretical shortest path length in the monitored area, Indicates the number of monitoring points in the training area to be monitored.
9. The device for planning the path of a drone to be monitored based on the traveling salesman problem solving method guided by knowledge in deep reinforcement learning according to claim 8, characterized in that: The processing module is also used to update the RIDGE algorithm. The updated RIDGE algorithm formula is as follows: ; in, represents the gradient operator; Represents the current deep reinforcement learning model Solve the cumulative benefits of the training samples below; Indicates The first batch The first training sample Solution to the path planning problem; Indicates The first batch The first training sample A reward for solving a path planning problem; Indicates that in the training sample Next, select The first batch The first training sample A strategy for solving the path planning problem.
Citation Information
Patent Citations
Multi-robot autonomous path planning method
CN115542921A
Travel salesman problem solving method based on graph convolution and reinforcement learning algorithm
CN116862083A
Learning method and learning device for supporting reinforcement learning by using human driving data as training data to thereby perform personalized path planning
US20200250486A1
