Method and device for solving traveling salesman problem based on cross-scale deep reinforcement learning
By using the BHH theorem and adjusting the reward function in the UAV monitoring task, the problem of insufficient generalization ability of the deep reinforcement learning model in different scale path planning problems is solved, and efficient path planning strategy solving is achieved.
Patent Information
- Application Number
- CN202411040404.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The existing deep reinforcement learning methods are difficult to effectively generalize to different scale path planning problems in UAV monitoring tasks, resulting in the inability to effectively solve the model in practical applications or the solution time is too long.
By introducing the BHH theorem, the ratio relationship between the path length and the number to be monitored is determined, and the reward function and REINFORCE algorithm are adjusted to improve the generalization ability of the model to adapt to path planning problems at different scales.
It realizes efficient solution to the UAV monitoring path planning problem at different scales, improves the generalization ability and computing efficiency of the model, and can provide the optimal path planning strategy at different scales at the same time.
Smart Images

Figure CN118607742B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of combinatorial optimization technology, and in particular to a method and device for solving the traveling salesman problem based on cross-scale deep reinforcement learning. Background Art
[0002] The Traveling Salesman Problem (TSP) is a fundamental problem in the field of combinatorial optimization, which requires starting from a city and traversing all cities back to the starting point, and the path must be as short as possible. It has a wide range of applications in various fields, including vehicle transportation, drone path planning, online planning, circuit layout, etc. Over the years, due to the inherent complexity of the NP (Nondeterministic polynominal)-Hard problem of TSP and the associated exponential computational requirements, researchers have conducted extensive research to design effective solutions for TSP. In recent years, the emergence of Deep Reinforcement Learning (DRL) has brought new ideas and methods to solving TSP. Researchers have adopted various network architectures, such as pointer networks, Transformer networks, and graph neural networks to model and solve TSP. These models are trained using reinforcement learning or a combination of reinforcement learning and supervised learning to solve the traveling salesman problem. In addition, there are some methods that combine traditional heuristic factors with reinforcement learning to solve TSP.
[0003] In actual scenarios, when drones are used to monitor the ground conditions of sites to be monitored, using drones to search and collect information at various sites to be monitored can be considered a traveling salesman problem. However, in the process of completing the monitoring task, the number of planned monitoring points is also changing due to the limitations of drone energy, the life cycle of monitoring points, or changes in the conditions of monitoring points. Most DRL methods in the TSP solution algorithm model are trained on a data set with a fixed number of nodes and tested on the corresponding node data set, resulting in limited model generalization ability and lack of reusability for large-scale practical problems. It is impossible to solve the path planning problem of drones to be monitored at different scales.
[0004] In order to improve the model's cross-scale generalization ability, the proposed solution is not feasible to train the model on datasets with different scales, or the use of meta-learning technology has the limitation of too long computing time.
[0005] Therefore, when drones monitor ground conditions, the monitoring points and their number may change due to the complex and changeable environment of the monitoring points. It is necessary to effectively plan the monitoring locations of drones across scales. However, the current solution of the TSP problem is inefficient or takes a long time to solve. The planning problem of the drone's monitoring path needs to be further optimized. Summary of the invention
[0006] In order to solve the technical problem of path planning for a constantly changing area to be monitored when a drone monitors ground conditions, the present invention aims to provide a method and device for solving the traveling salesman problem based on cross-scale deep reinforcement learning.
[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0008] According to one aspect of the present disclosure, a method for solving a traveling salesman problem based on cross-scale deep reinforcement learning is provided, comprising:
[0009] Obtain the information of the area to be monitored by the drone;
[0010] Convert the path planning problem of the drone to the monitored area into a TSP problem;
[0011] According to the information of the site to be monitored, in the DRL model training of the TSP problem, the ratio relationship between the optimal path length and the number of sites to be monitored is determined by the BHH theorem based on the path planning examples of each site to be monitored;
[0012] adjusting a reward function according to the ratio relationship;
[0013] A trained DRL model is obtained according to the adjusted reward function and the improved REINFORCE algorithm, and the path planning problem of the monitored area is solved by the trained DRL model.
[0014] In one embodiment of the present disclosure, the method further includes:
[0015] Obtain a mathematical representation of the path planning problem for the monitored area, wherein the process of path planning for the monitored area is represented as an undirected complete graph , and obtain the Euclidean distance between the locations to be monitored , and the total length of the corresponding solution of the path planning to be monitored, where Indicates the set of ground to be monitored. represents the set of paths between the locations to be monitored, represents the solution of the path planning to be monitored, express The total length of is expressed as follows:
[0016]
[0017] in, express On Areas to be monitored, yes Length;
[0018] Converting the path planning problem to be monitored into a Markov decision process to obtain a strategy for generating a solution to the path planning problem to be monitored and the reward function;
[0019] A neural network model is constructed to implement the Markov decision process.
[0020] In one embodiment of the present disclosure, the strategy expression for generating a solution to the path planning problem to be monitored is as follows:
[0021]
[0022] Among them, s represents the path planning instance of the monitored area;
[0023] The reward function The expression is as follows:
[0024]
[0025] in, .
[0026] In one embodiment of the present disclosure, according to the information of the site to be monitored, in the DRL model training of the TSP problem, the ratio relationship between the optimal path length and the number of sites to be monitored is determined by the BHH theorem based on the path planning instance of each site to be monitored, including:
[0027] The points in the path planning example to be monitored are in the unit square Under the condition of random uniform distribution, there is a relationship expression between the optimal path length and the number of monitored areas involved:
[0028]
[0029] in, represents a point within the unit square, Indicates the number of sites to be monitored, is the shortest path length that passes through all points, c1 and c2 are constants;
[0030] Determine the range of the ratio relationship β between the optimal path length of each path planning of the to-be-monitored site and the number of the to-be-monitored sites, the formula is as follows:
[0031] .
[0032] In one embodiment of the present disclosure, adjusting the reward function according to the ratio relationship includes:
[0033] According to the ratio relationship The reward function in DRL is adjusted by reducing The expression is as follows:
[0034]
[0035] in express The number of sites to be monitored.
[0036] In one embodiment of the present disclosure, the preset gradient update algorithm is an improved REINFORCE algorithm, wherein the calculation formula of the improved REINFORCE algorithm is as follows:
[0037]
[0038] Wherein, b represents the number of the path planning instances to be monitored in each batch.
[0039] In one embodiment of the present disclosure, the method further includes:
[0040] Monte Carlo is used to randomly select a starting location to be monitored for path sampling from each location to be monitored path planning instance.
[0041] According to another aspect of the present disclosure, a traveling salesman problem solving device is provided, comprising:
[0042] An information acquisition module is used to obtain the information of the area to be monitored by the drone;
[0043] A TSP conversion module, used for converting the path planning problem of the unmanned aerial vehicle to be monitored into a TSP problem;
[0044] A relationship determination module is used to determine the ratio relationship between the optimal path length and the number of sites to be monitored by using the BHH theorem based on the path planning examples of each site to be monitored in the DRL model training of the TSP problem according to the information of the site to be monitored by the drone;
[0045] A reward adjustment module, used for adjusting the reward function according to the ratio relationship;
[0046] The solution module is used to obtain a trained DRL model according to the adjusted reward function and the improved REINFORCE algorithm, and solve the path planning of the drone to be monitored by using the trained DRL model.
[0047] According to yet another aspect of the present disclosure, there is provided an electronic device, including: a processor; and
[0048] A memory, configured to store executable instructions of the processor;
[0049] The processor is configured to execute any one of the above-mentioned methods for solving the traveling salesman problem based on cross-scale deep reinforcement learning by executing the executable instructions.
[0050] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for solving the traveling salesman problem based on cross-scale deep reinforcement learning described in any one of the above is implemented.
[0051] The embodiment of the present disclosure provides a method for solving the traveling salesman problem based on cross-scale deep reinforcement learning, which obtains the information of the monitored area of the drone; converts the path planning problem of the monitored area of the drone into a TSP problem; according to the information of the monitored area, in the DRL model training of the TSP problem, the ratio relationship between the optimal path length and the number of monitored areas is determined by the BHH theorem based on each monitored area path planning instance; the reward function is adjusted according to the ratio relationship; the trained DRL model is obtained according to the adjusted reward function and the improved REINFORCE algorithm, and the path planning problem of the monitored area is solved by the trained DRL model. The monitored area changes continuously according to the actual situation. For the path planning problem of the monitored area of different scales, the reward function is refined and adjusted by using the idea of the BHH theorem, and the model is gradient updated by using the improved REINFORCE algorithm to obtain a solution model with stronger generalization ability, which is suitable for the planning of the monitored path of drones at different scales, and can simultaneously solve the optimal path planning strategy at different scales, so as to efficiently complete the large-scale monitoring of drones. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0053] Figure 1It is a flowchart of a method for solving the traveling salesman problem based on cross-scale deep reinforcement learning according to an embodiment of the present disclosure;
[0054] Figure 2 This is an embodiment of the present disclosure, a method for solving the traveling salesman problem based on cross-scale deep reinforcement learning. and the number of sites to be monitored Relationship change curve chart;
[0055] Figure 3 This is a schematic diagram of the structure of a device for solving the traveling salesman problem based on cross-scale deep reinforcement learning according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.
[0057] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0058] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.
[0059] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.
[0060] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.
[0061] To facilitate understanding, the terms involved in this application are first explained below.
[0062] Deep Reinforcement Learning (DRL): It is a combination of reinforcement learning (RL) and deep learning (DL). It optimizes reinforcement learning algorithms through deep learning techniques such as neural networks.
[0063] TSP (Traveling Salesman Problem): Suppose a traveling salesman wants to visit N cities. He must choose a path to take. The restriction of the path is that each city can only be visited once, and he must return to the original starting city in the end. The goal of path selection is to obtain a path with the minimum distance among all paths.
[0064] Markov Decision Process (MDP): It is a mathematical model of sequential decision, which is used to simulate the random strategies and rewards that can be achieved by agents in an environment where the system state has Markov properties. MDP is built based on a set of interacting objects, namely agents and environments, and its elements include state, action, strategy and reward. In the simulation of MDP, the agent will perceive the current system state, take actions on the environment according to the strategy, thereby changing the state of the environment and getting rewards. The accumulation of rewards over time is called return.
[0065] An intuitive way to improve the cross-scale generalization ability of the model is to train the model on datasets with different scales. However, due to the requirement for environmental consistency in reinforcement learning, training the model on cross-scale datasets is not feasible.
[0066] Recently, more and more research has directed attention to alternative methods to enhance the generalization ability of models. One of these methods is meta-learning, the idea of which is to first pre-train the initial model on a set of small-scale TSP instances, and then fine-tune it on several large-scale instances to enhance the model's ability to solve large-scale problems. However, meta-learning methods still require a lot of computing resources to fine-tune large-scale problems, resulting in excessively long computational time, even exceeding the solution time required to obtain the optimal solution to the problem using an exact algorithm. This limitation greatly restricts the application of existing meta-learning related methods in large-scale real-world problems, and in some cases, they may even lose their practical utility.
[0067] This application mainly aims at solving the traveling salesman problem based on cross-scale deep reinforcement learning.
[0068] This application is mainly aimed at the scenario of using drones to monitor the ground conditions of the sites to be monitored on the ground. Using drones to search and collect information at each site to be monitored can be regarded as a traveling salesman problem. Drones can be used for ground monitoring, and then data or information is collected at the monitoring site. For example, after a natural disaster, drones are used to collect rescue data at each site to be monitored. The number of rescue sites planned to go to each time will vary according to the rescue information, so it is not a fixed-scale TSP. At this time, if the traditional method of training on a fixed number of nodes is used, a lot of time and computing resources will be consumed, and the model directly trained with existing fixed nodes will not work well. Similarly, in the supervision of wild animals, using drones to search for information at each site to be monitored can be regarded as a traveling salesman problem. Due to the limitation of drone energy and the limitation of the life cycle of the monitoring point, the number of monitoring points required to collect information is also variable. Drones need to monitor different sites to be monitored, and the number of monitoring points for path planning between the sites to be monitored is also variable. In this scenario, a high generalization ability and a solution algorithm model capable of coping with TSPs of different scales are required. The existing technology has limitations and cannot be solved.
[0069] The following example is used to specifically illustrate the solution process of the traveling salesman problem:
[0070] Obtain a mathematical representation of the path planning problem for the monitored area, wherein the process of path planning for the monitored area is represented as an undirected complete graph , and obtain the Euclidean distance between the locations to be monitored , and the total length of the corresponding solution of the path planning to be monitored, where Indicates the set of ground to be monitored. represents the set of paths between the locations to be monitored, represents the solution of the path planning to be monitored, express The total length of is expressed as follows:
[0071] Formula 1
[0072] in, express On Areas to be monitored, yes Length;
[0073] Specifically, it is necessary to determine the optimal solution for planning the path among the N sites to be monitored. and The Euclidean distance between Indicates that Must meet exactly one visit The conditions for all points in , and the starting and ending points are the same. In each solution Starting from the initial point, at each step a decision is made based on the information from the remaining points to choose the next point. Euclidean distance It can be expressed as the cost of traveling from one monitored location to another, such as the power consumption of a drone.
[0074] Convert the process of path planning for the monitored area into a Markov decision process, and obtain a strategy for generating a solution for the path planning for the monitored area and the reward function;
[0075] Specifically, since its process is very consistent with the concept of Markov decision process (MDP), we regard the process of solving the TSP problem as an MDP.
[0076] A neural network model is constructed to implement the Markov decision process.
[0077] Specifically, a deep neural network is used to model the solution process, where neural network models include Transformer models, graph neural network models, etc. Each decision in the Markov decision process can be expressed as ,in Indicates the TSP instance, that is, the path planning instance to be monitored. Indicates Before step There are differences in the number of sites to be monitored that need to be planned in the planning examples of different sites to be monitored paths.
[0078] In a specific example, when a path planning instance to be monitored is given When , the strategy expression for generating the solution of the TSP problem is as follows:
[0079] Formula 2
[0080] in, represents the path planning instance to be monitored, It means that the model has parameters When given a problem instance s, we can give This path is taken as the probability of the solution.
[0081] In the original solution process,
[0082] The reward function The expression is as follows:
[0083] Formula 3
[0084] The above formula 3 indicates that all The expected reward value of the obtained path is, .
[0085] Specifically, the reward value of each decision step is set to the negative of the increase in path length. For a given TSP instance, the goal is to maximize the cumulative reward value. The reward value is determined by Formula 3.
[0086] In addition, according to the policy gradient theory, The derivative of , and use the gradient update algorithm to update the model:
[0087] Formula 4
[0088] Reference Figure 1 As shown, in one embodiment, a method for solving the traveling salesman problem based on cross-scale deep reinforcement learning is provided, comprising:
[0089] Step S101, obtaining the information of the area to be monitored by the drone;
[0090] Among them, the information of the area to be monitored refers to the location or position information that the drone is going to monitor. When the drone is performing a monitoring task, it must be informed of the information of the area to be monitored. The information of the area to be monitored is not fixed and changes according to the actual monitoring needs. The drone monitors the ground, and the objects to be monitored include but are not limited to animals, plants, buildings, etc. However, due to the complexity of the environment, the monitoring objects are not static. For example, in the scenario of using drones for monitoring after an earthquake, the drone will perform rescue data at the planned area to be monitored. Each time no one is performing a monitoring task, the area to be monitored and its number will change according to the rescue situation. For another example, in the scenario of monitoring wild protected animals, the drone needs to search for the location of the wild protected animals, and the area to be monitored will also change.
[0091] Step S102, converting the path planning problem of the drone to be monitored into a TSP problem;
[0092] Specifically, the drone goes to each monitored location to perform monitoring tasks, and the drone's path planning for the monitored location can be regarded as a traveling salesman problem.
[0093] Step S103, according to the information of the site to be monitored, in the DRL model training of the TSP problem, based on the path planning examples of each site to be monitored, the ratio relationship between the optimal path length and the number of sites to be monitored is determined by the BHH theorem;
[0094] Specifically, based on the TSP problem instance in the above embodiment, in order to improve the generalization ability of the model, the Beardwood-Halton-Hammersley (BHH) theorem is introduced to determine the ratio between the optimal path length of each monitored location path planning instance of different sizes and the number of monitored locations.
[0095] In a specific example, step S101 includes:
[0096] The points in the path planning example to be monitored are in the unit square Under the condition of random uniform distribution, there is a relationship expression between the optimal path length and the number of sites to be monitored:
[0097] Formula 5
[0098] in, represents a point within the unit square, point , Indicates the number of sites to be monitored, is the length of the shortest path through all points, and is a constant;
[0099] Specifically, there is a relationship in the above formula 5 between the path length of the optimal solution and the number of nodes involved in the TSP, and the relationship between the path length of the optimal solution in the path planning instance of the monitored area and the involved monitored area can be proved.
[0100] The range of the ratio relationship β between the optimal path length and the number of nodes of each TSP instance is determined as follows:
[0101] Formula 6
[0102] Specifically, we introduce the BHH theorem and assume are independent and randomly distributed in the unit square The range of the ratio relationship β between the optimal path length and the number of nodes in the path planning instances of different sizes to be monitored can be determined. According to the above formula 6, β ranges between 0.625 and 0.922, and β can be considered a measure of the model's performance in solving problems of different sizes, whereas previously only path length was considered an indicator of model performance.
[0103] Step S104, adjusting the reward function according to the ratio relationship;
[0104] Specifically, the main challenge of training across scale datasets in DRL is that the reward values of different scales of monitored path planning instances have different significance. For example, the unit difference in reward value between an instance with 20 monitored sites and an instance with 100 monitored sites has a completely different impact on the model's solving ability. Applicable to the planning of monitored paths for drones at different scales.
[0105] Using the BHH theorem formula, we can establish a fitting line by calculating the optimal solution of TSP at different scales through the traditional solver. Number of sites to be monitored The curve of the relationship between Figure 2 As shown, when When the value of varies between 100 and 10,000, The fluctuation does not exceed 8%. In addition, as the number of sites to be monitored increases, the fluctuation decreases rapidly. As it approaches infinity, The changes in The influence of can be ignored, and its value converges to .
[0106] use Represents a given cross-scale training range, for any given small value , there is always a , so that the following equation 7 holds true:
[0107] Formula 7
[0108] Due to the reward function in the above formula 3, it is impossible to solve the problem of training different sizes of path planning instances to be monitored together. In a specific example, it includes:
[0109] According to the proportional relationship The reward function in DRL is adjusted by reducing The expression is as follows:
[0110] Formula 8
[0111] in express The number of sites to be monitored.
[0112] Specifically, based on The reduction in the ratio refines the reward function in DRL to solve the problem of training different sizes of path planning instances to be monitored together. The reward function is adjusted to obtain the maximum optimized reward function. .
[0113] Step S105, obtaining a trained DRL model according to the adjusted reward function and the improved REINFORCE algorithm, and solving the path planning problem of the monitored area by using the trained DRL model.
[0114] Specifically, the above steps can be used to train the model on a cross-scale dataset that includes path planning examples for sites to be monitored with different numbers of monitored sites. This enables the model to be effectively generalized at different scales without extensive adjustments or expensive fine-tuning. It can also simultaneously solve for the optimal path planning strategy at different scales, and efficiently complete large-scale monitoring of drones.
[0115] Among them, the policy gradient algorithm uses gradient information to guide the optimization of the model. The traditional reinforcement algorithm is a classic policy gradient reinforcement learning algorithm. The REINFORCE algorithm directly improves the performance of the model in the policy parameter space through gradient ascent. Although there are more effective policy gradient algorithms, such as Actor-Critic (AC), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO), using these algorithms is more challenging than the REINFORCE algorithm in the context of TSP. Therefore, this embodiment selects an improved version of the REINFORCE algorithm for gradient update.
[0116] In a specific example, the gradient update algorithm used is the improved REINFORCE algorithm, wherein the calculation formula of the improved REINFORCE algorithm is as follows:
[0117] Formula 9
[0118] Wherein, b represents the number of the path planning instances to be monitored in each batch.
[0119] Optionally, one drone may correspond to one instance of path planning for the area to be monitored, and b instances may represent that b drones jointly perform path planning for the area to be monitored.
[0120] The following is a code example of the SCARF algorithm:
[0121]
[0122] By combining the BHH theorem with the REINFORCE algorithm, a deep reinforcement learning method for achieving true model generalization, namely SCARF, is proposed. The model can be trained on a cross-scale dataset that includes path planning examples of monitored sites with different monitored sites. This enables the model to generalize effectively at different scales without extensive adjustments or expensive fine-tuning processes.
[0123] In one example, the method further includes:
[0124] Monte Carlo is used to randomly select a starting location to be monitored for path sampling from each location to be monitored path planning instance.
[0125] Specifically, for each monitored path planning instance sampling Each sampling involves randomly selecting a starting location to be monitored and using Monte Carlo sampling to obtain Solution There are b instances in each batch, so for b instances in each batch, each instance samples n paths, and we can get Results. Obviously, large-scale problems are more difficult to solve than small-scale problems. Preferably, more samples are collected for larger-scale path planning instances to be monitored.
[0126] In this embodiment, by utilizing the BHH theorem, Scale-reduction REINFORCE (SCARF) is proposed by combining the BHH theorem with the REINFORCE algorithm. This is the first REINFORCE algorithm that solves the problem of training a DRL model on a cross-scale dataset for path planning in the monitored area. The above embodiment only needs to be trained on a small-scale dataset, and no additional fine-tuning is required to adapt to large-scale problems. This paradigm shift in training methods significantly enhances the generalization ability of the model at various scales, ensuring a more versatile and adaptable solution for TSP. The proposed method has been extensively experimentally evaluated on randomly generated and real-world large-scale dataset instances, while considering cross-scale and cross-distribution generalization tasks. When the drone performs a monitoring task, regardless of how the number of monitored areas changes, facing the path planning problem of monitored areas of different scales, the optimal path planning strategy can be accurately and efficiently provided. And drones of different task scales can simultaneously perform path planning for monitored areas.
[0127] In another embodiment, a traveling salesman problem solving device 300 based on cross-scale deep reinforcement learning is provided, comprising:
[0128] The information acquisition module 301 is used to acquire the information of the area to be monitored by the drone;
[0129] A TSP conversion module 302 is used to convert the path planning problem of the unmanned aerial vehicle to be monitored into a TSP problem;
[0130] The relationship determination module 303 is used to determine the ratio relationship between the optimal path length and the number of sites to be monitored by using the BHH theorem based on the path planning examples of each site to be monitored in the DRL model training of the TSP problem according to the information of the site to be monitored by the drone;
[0131] A reward adjustment module 304, configured to adjust the reward function according to the ratio relationship;
[0132] The solution module 305 is used to obtain a trained DRL model according to the adjusted reward function and the improved REINFORCE algorithm, and solve the path planning problem to be monitored by using the trained DRL model.
[0133] The present embodiment provides a traveling salesman problem solving device 300 based on cross-scale deep reinforcement learning, including an information acquisition module 301, a TSP conversion module 302, a relationship determination module 303, a reward adjustment module 304, and a solution module 305. Through the above modules of the traveling salesman problem solving device 300 based on cross-scale deep reinforcement learning provided by the present embodiment, when the drone performs a monitoring task, regardless of how the number of monitored areas changes, the optimal path planning strategy can be accurately and efficiently provided for the path planning problems of monitored areas of different scales. Moreover, the path planning problems of monitored areas of drones of different task scales can be carried out simultaneously, and large-scale monitoring of drones can be completed efficiently.
[0134] The method for solving the traveling salesman problem based on cross-scale deep reinforcement learning implemented by the above-mentioned modules in this embodiment has been described in detail in its corresponding method embodiment, and the present disclosure will not elaborate on it here.
[0135] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0136] In yet another embodiment, an electronic device is provided, including: a processor; and
[0137] A memory, configured to store executable instructions of the processor;
[0138] The processor is configured to execute any one of the above-mentioned methods for solving the traveling salesman problem based on cross-scale deep reinforcement learning by executing the executable instructions.
[0139] This embodiment also provides an electronic device capable of solving the above-mentioned traveling salesman problem, which is described in detail in the corresponding method embodiment and will not be described in detail in this disclosure.
[0140] In yet another embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned methods for solving the traveling salesman problem based on cross-scale deep reinforcement learning is implemented.
[0141] This embodiment also provides a computer-readable storage medium that can implement the above-mentioned method for solving the traveling salesman problem based on cross-scale deep reinforcement learning. It is described in detail in the corresponding method embodiment, and the present disclosure will not repeat it here.
[0142] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.
[0143] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".
[0144] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.
[0145] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
[0146] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0147] A computer-readable storage medium is provided in an exemplary embodiment of the present disclosure, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary implementations of the present invention described in the above "Exemplary Method" section of the present specification.
[0148] A program product for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0149] The program product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0150] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0151] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0152] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0153] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0154] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0155] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0156] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A method for solving the traveling salesman problem based on cross-scale deep reinforcement learning, It is characterized in that include: Obtain the information of the area to be monitored by the drone; Convert the path planning problem of the drone to the monitored area into a TSP problem; According to the information of the site to be monitored, in the DRL model training of the TSP problem, the ratio relationship between the optimal path length and the number of sites to be monitored is determined by the BHH theorem based on the path planning examples of each site to be monitored; adjusting a reward function according to the ratio relationship; Obtaining a trained DRL model according to the adjusted reward function and the improved REINFORCE algorithm, and solving the path planning problem of the monitored area by using the trained DRL model; The conversion of the path planning problem of the drone to be monitored into a TSP problem includes: Obtain a mathematical representation of the path planning problem for the monitored area, wherein the process of path planning for the monitored area is represented as an undirected complete graph , and obtain the Euclidean distance between the locations to be monitored , and the total length of the corresponding solution of the path planning to be monitored, where Indicates the set of ground to be monitored. represents the set of paths between the locations to be monitored, represents the solution of the path planning to be monitored, express The total length of is expressed as follows: ; in, express On Areas to be monitored, yes The number of sites to be monitored; Converting the path planning problem to be monitored into a Markov decision process to obtain a strategy for generating a solution to the path planning problem to be monitored and the reward function; A neural network model is constructed to implement the Markov decision process; the strategy expression for generating a solution to the path planning problem to be monitored is as follows: in, Indicates the path planning instance to be monitored; The reward function The expression is as follows: in, ,and Indicates all The expected reward value of the obtained path is It means that the model has parameters When given a problem instance In the case of The probability of this path being a solution; according to the information of the to-be-monitored land, in the DRL model training of the TSP problem, the ratio relationship between the optimal path length and the number of to-be-monitored lands is determined by the BHH theorem based on the path planning examples of each to-be-monitored land, including: The points in the path planning example to be monitored are in the unit square Under the condition of random uniform distribution, there is a relationship expression between the optimal path length and the number of monitored areas involved: ; in, represents a point within the unit square, Indicates the number of sites to be monitored, is the length of the shortest path through all points, and is a constant; Determine the ratio between the optimal path length of each path planning instance of the to-be-monitored site and the number of the to-be-monitored sites The range is as follows: ; The step of adjusting the reward function according to the ratio relationship comprises: According to the ratio relationship The reward function in DRL is adjusted by reducing The expression is as follows: ; in, express The number of sites to be monitored; the calculation formula of the improved REINFORCE algorithm is as follows: ; Wherein, b represents the number of the path planning instances to be monitored in each batch.
2. The method for solving the traveling salesman problem based on cross-scale deep reinforcement learning as claimed in claim 1, It is characterized in that The method further comprises: Monte Carlo is used to randomly select a starting location to be monitored for path sampling in each location to be monitored path planning instance.
3. A traveling salesman problem solving device, It is characterized in that include: An information acquisition module is used to obtain the information of the area to be monitored by the drone; A TSP conversion module, used for converting the path planning problem of the unmanned aerial vehicle to be monitored into a TSP problem; A relationship determination module is used to determine the ratio relationship between the optimal path length and the number of sites to be monitored by using the BHH theorem based on the path planning examples of each site to be monitored in the DRL model training of the TSP problem according to the information of the site to be monitored by the drone; A reward adjustment module, used for adjusting the reward function according to the ratio relationship; A solution module, used for obtaining a trained DRL model according to the adjusted reward function and the improved REINFORCE algorithm, and solving the path planning of the drone to be monitored by using the trained DRL model; The conversion of the path planning problem of the drone to be monitored into a TSP problem includes: Obtain a mathematical representation of the path planning problem for the monitored area, wherein the process of path planning for the monitored area is represented as an undirected complete graph , and obtain the Euclidean distance between the locations to be monitored , and the total length of the corresponding solution of the path planning to be monitored, where, Indicates the set of ground to be monitored. represents the set of paths between the locations to be monitored, represents the solution of the path planning to be monitored, express The total length of is expressed as follows: ; in, express On Areas to be monitored, yes The number of sites to be monitored; Converting the path planning problem to be monitored into a Markov decision process to obtain a strategy for generating a solution to the path planning problem to be monitored and the reward function; A neural network model is constructed to implement the Markov decision process; the strategy expression for generating a solution to the path planning problem to be monitored is as follows: in, Indicates the path planning instance to be monitored; The reward function The expression is as follows: in, ,and Indicates all The expected reward value of the obtained path is It means that the model has parameters When given a problem instance In the case of The probability of this path being a solution; according to the information of the to-be-monitored land, in the DRL model training of the TSP problem, the ratio relationship between the optimal path length and the number of to-be-monitored lands is determined by the BHH theorem based on the path planning examples of each to-be-monitored land, including: The points in the path planning example to be monitored are in the unit square Under the condition of random uniform distribution, there is a relationship expression between the optimal path length and the number of monitored areas involved: ; in, represents a point within the unit square, Indicates the number of sites to be monitored, is the length of the shortest path through all points, and is a constant; Determine the ratio between the optimal path length of each path planning instance of the to-be-monitored site and the number of the to-be-monitored sites The range is as follows: ; The step of adjusting the reward function according to the ratio relationship comprises: According to the ratio relationship The reward function in DRL is adjusted by reducing The expression is as follows: ; in, express The number of sites to be monitored; the calculation formula of the improved REINFORCE algorithm is as follows: ; Wherein, b represents the number of the path planning instances to be monitored in each batch.
4. An electronic device, It is characterized in that include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the traveling salesman problem solving method based on cross-scale deep reinforcement learning as described in any one of claims 1 to 2 by executing the executable instructions.
5. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for solving the traveling salesman problem based on cross-scale deep reinforcement learning as described in any one of claims 1 to 2 is implemented.
