A method and device for solving a UAV reconnaissance path planning problem by using a DQN-ALNS

Abstract

The application provides a UAV reconnaissance path planning method and device solved by DQN-ALNS, and the method comprises the following steps: training a DQN model; integrating the trained DQN model into ALNS to construct a hierarchical planning model; outputting a UAV reconnaissance path by using the hierarchical planning model; constantly updating and iterating the UAV reconnaissance path by using the destruction and repair operators of the hierarchical planning model; and judging whether the iteration is finished or not by using the simulated annealing criterion. The application proposes a new algorithm of DQN-ALNS, which integrates a deep Q network (DQN) and an adaptive large neighborhood search (ALNS), and can explore the solution space according to the search state of the current solution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle path planning, in particular to a DQN-ALNS solving unmanned aerial vehicle reconnaissance path planning method and device. BACKGROUND

[0002] Unmanned aerial vehicles (UAVs) have been widely used in many fields due to their high cost-effectiveness, flexible deployment, and ability to eliminate threats to human life. In recent years, the research on unmanned aerial vehicles in regional reconnaissance and target reconnaissance has been particularly prominent. These tasks are usually modeled as path planning problems. However, due to the importance and benefits of targets, which may vary due to factors such as damage or urgency, the efficiency of actual reconnaissance, and the timeliness of completing tasks, it is necessary to adjust and optimize the reconnaissance path.

[0003] Current path algorithms cannot fully utilize the current search state information. This leads to the selection of operators being based on fixed probability sampling, thereby ignoring short-term dependencies that may be beneficial to the optimization of the operator selection process.

[0004] In summary, the existing technology has the following problems: it is unable to plan the priority of unmanned aerial vehicle reconnaissance targets on demand, and the reconnaissance efficiency is low. SUMMARY

[0005] The purpose of the present application is to solve the problem of how to plan the priority of unmanned aerial vehicle reconnaissance targets and improve the efficiency of actual reconnaissance.

[0006] To this end, on the one hand, the present application provides a DQN-ALNS solving unmanned aerial vehicle reconnaissance path planning method, which comprises the following steps:

[0007] training a DQN model;

[0008] integrating the trained DQN model into ALNS to construct a hierarchical planning model;

[0009] outputting an unmanned aerial vehicle reconnaissance path using the hierarchical planning model;

[0010] continuously updating and iterating the unmanned aerial vehicle reconnaissance path through the destruction and repair operators of the hierarchical planning model;

[0011] using the simulated annealing criterion to determine whether the iteration is complete.

[0012] On the other hand, the present application also provides a DQN-ALNS solving unmanned aerial vehicle reconnaissance path planning device, comprising:

[0013] a training unit for training a DQN model;

[0014] A construction unit is configured to integrate the trained DQN model into the ALNS to construct a hierarchical planning model;

[0015] An output unit is configured to output the UAV reconnaissance path by using the hierarchical planning model.

[0016] An iteration unit is configured to constantly update the iteration UAV reconnaissance path by using the destruction and repair operators of the hierarchical planning model.

[0017] A termination unit is configured to determine whether the iteration is terminated by using the simulated annealing criterion.

[0018] The technical scheme has the following beneficial effects: the selection probability of the operator is quickly adjusted in the iteration process, so that the DQN-ALNS can utilize new information in the search process and plan a more optimal UAV reconnaissance path. The present application provides a new algorithm of DQN-ALNS, which integrates the deep Q network (DQN) and the adaptive large neighborhood search (ALNS), and can explore the solution space according to the search state of the current solution.

[0019] The present application provides a mixed integer linear programming model for the UAV reconnaissance path planning problem, which considers the target priority and is an extension of the vehicle path planning problem. The Gurobi solver is used to solve small-scale problems, and the sensitivity analysis of the UAV power and the target priority under different distributions is performed. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flow chart of a UAV reconnaissance path planning method solved by a DQN-ALNS provided by an embodiment of the present application;

[0021] Figure 2 is a structural schematic diagram of a UAV reconnaissance path planning device solved by a DQN-ALNS provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0023] In the problem of UAV reconnaissance path planning, the application solves the problem of priority weighting, which represents the importance of the target in a weighted manner. The application proposes a Deep Q Network-Adaptive Large Neighborhood Search (DQN-ALNS) algorithm, which uses reinforcement learning to select operators to solve the problem of priority-weighted UAV reconnaissance path planning.

[0024] The Adaptive Large Neighborhood Search (ALNS) algorithm is known for its block structure exploration in the solution space, which makes ALNS show high robustness when dealing with various instances with different characteristics, and effectively avoids falling into local optimal solutions. However, in the traditional ALNS algorithm, the selection of the operator depends on the roulette wheel update mechanism, which is based only on the historical performance of the operator, and fails to fully utilize the information of the current search state. This leads to the selection of the operator based on sampling with fixed probability, thereby ignoring the short-term dependencies that may be beneficial to the optimization of the operator selection process.

[0025] In order to overcome the limitations of the traditional Adaptive Large Neighborhood Search (ALNS) algorithm in selecting operators, the application introduces a reinforcement learning method. The agent can predict and select the action that can bring the maximum expected future reward in each decision-making step. In addition, the Deep Q Network (DQN) algorithm uses the powerful function approximation ability of artificial neural networks to train the agent in the decision-making stage and enable it to interact with the environment. After training, the agent can recognize changes in the dynamic environment and adapt to different environments with similar settings. Therefore, integrating DQN into the ALNS operator selection can enable the algorithm to make more intelligent decisions at each iteration, thereby improving overall performance. The DQN-ALNS algorithm proposed in the application represents the search state through a set of features (including problem instances, changes in solutions, and running history), and selects a destroy-repair operator to update the UAV reconnaissance path at each iteration. This method allows the selection probability of the operator to quickly adjust during the iteration process, enabling DQN-ALNS to utilize new information in the search process and plan a more optimal UAV reconnaissance path.

[0026] In an embodiment of the application, as Figure 1 , a DQN-ALNS solving UAV reconnaissance path planning method is provided, the method comprising the following steps:

[0027] S101: training a DQN model;

[0028] DQN is a model-free RL method for solving MDPs that relies on estimating the state-action value function Q(s, a). As shown in Table 1, unlike the approximate estimation of general temporal difference methods, DQN approximates the value function using a deep neural network, while training the network parameters by minimizing the objective function (12).

[0029]

[0030] where γ denotes the discount factor, Q(s, a, w) represents the primary Q-network, and Q'(s, a, w) represents the target Q-network. The interaction of the agent with the environment generates tuples (s, a, r, s') and the network parameters w are updated according to the Deep-Q network training algorithm. After the training process is completed, the optimal policy π * (a|s, w) will be used to solve unknown instances in the test set.

[0031]

[0032] Table 1 Deep-Q network training algorithm

[0033]

[0034] S102: Integrate the trained DQN model into the ALNS to build a hierarchical planning model;

[0035] As mentioned before, the learned policy can take actions more effectively by always choosing the action with the highest expected cumulative reward with respect to the state-action value function of the problem. Both the Q-value update and the roulette-wheel update use a weighting factor to balance the performance estimates that represent history and current. The key difference is that the Q-value update is conditioned on the state, so it can capture more information that can be used to select relevant operators, while the roulette-wheel update just averages the gains of the operators regardless of whether they are suitable for the current solution. However, when integrating the trained learning model into the ALNS, some problems can be encountered. Because the objective function weighting changes the shape of the search space, making some regions steeper or flatter, and due to the nature of the greedy policy, it can reduce the ability of the search process to escape from local optimal solutions. To expand the search range and obtain a probabilistic policy, the SoftMax function shown in equation (14) is used in this application, where the temperature parameter τ is used to adjust the degree of greediness of the policy. Specifically, when τ tends to infinity, the selection probabilities of each action tend to be uniform; while when τ tends to zero, the selection probability of the action with the highest expected cumulative reward approaches 1.

[0036]

[0037] S103: outputting the UAV reconnaissance path by using the hierarchical planning model;

[0038] The action space A includes a combination of destroy-repair operators that can be applied to the current solution in each iteration, and the size of the action space A is not limited. Similar to the ALNS destroy and repair solution operations, the action is composed of two parts of destroy and repair operators, which increases the diversity of the solution by constantly destroying and reconstructing the current solution. The DRL agent selects an action to construct a new solution. Each action is a combination of one destroy operator and one repair operator in Table 3 and Table 4. Therefore, the size of the action space |A| is 24, that is, 6 destroy operators x 4 repair operators.

[0039] S104: constantly updating the iteration UAV reconnaissance path by using the destroy and repair operators of the hierarchical planning model;

[0040] The action space A includes a combination of destroy-repair operators that can be applied to the current solution in each iteration, and in each iteration of the DQN-ALNS algorithm, the agent selects an action a∈A and applies it to the current solution. The solution is the planned path of the UAV.

[0041] The transition function mainly reflects the change of the solution in the search process. Similar to ALNS, after the agent applies the selected action to the current solution, the destroy operator removes some nodes from the current solution and puts these nodes into the deletion list. Then, the repair operator reinserts the nodes from the deletion list into the solution, empties the deletion list, and generates a new solution.

[0042] S105: determining whether the iteration is ended by using the simulated annealing criterion;

[0043] The acceptance criterion in simulated annealing is used. The acceptance criterion depends on the difference in the objective function between the existing solution X and the new solution X', denoted as ΔE = Π(x') - Π(x), and the temperature parameter T which gradually decreases throughout the search process. If the objective function of a new solution is lower than that of the current solution, that is, ΔE < 0, then the new solution is always accepted. In addition, the probability of accepting a worse new solution is e -ΔE / T In order to determine the initial temperature T0, all solutions of the first 100 iterations of the search are accepted, and all unimproved new solutions, that is, ΔE > 0, are recorded, as shown in formula (15). Then, the average value of these positive increments The following can be obtained:

[0044]

[0045] In order to reduce the temperature, the cooling schedule of Crama and Schyn is used, and the search is terminated when a certain number of iterations is reached.

[0046] Integrate the trained DQN model into ALNS to build a hierarchical planning model, including:

[0047] The trained DQN model is used as the operator selection model to replace the operator selection part in ALNS to form a hierarchical planning model.

[0048] The hierarchical planning model includes:

[0049]

[0050] Where: S is the state space, A is the action space, γ is the discount factor, α is the learning rate; τ is the temperature parameter;

[0051] Q(s,a,w) is the value function; D is the initialization experience replay buffer, N is the capacity; w is the parameter of the randomly initialized main Q network; w T =w is the parameter of the randomly initialized target Q network; π is the policy function.

[0052] The destruction repair operator of the hierarchical planning model continuously updates and iterates the UAV reconnaissance path; including:

[0053] Based on the trained DQN model, a set of damage repair operators are selected as actions to update the drone reconnaissance path;

[0054] Use the SA criteria to evaluate the updated UAV reconnaissance path and determine whether to accept the updated UAV reconnaissance path;

[0055] Output the drone reconnaissance path accepted by SA criteria.

[0056] The training of the DQN model specifically includes:

[0057] The network parameters of the DQN model are trained by minimizing the objective function.

[0058] The present invention also provides a UAV reconnaissance path planning device solved by DQN-ALNS, such as Figure 2 As shown, including:

[0059] Training unit 21, used for training the DQN model;

[0060] A construction unit 22 is used to integrate the trained DQN model into the ALNS to build a hierarchical planning model;

[0061] An output unit 23 is configured to output a UAV reconnaissance path using the hierarchical planning model;

[0062] Iteration unit 24, used for continuously updating and iterating the UAV reconnaissance path through the destruction and repair operators of the hierarchical planning model;

[0063] A termination unit 25 is configured to determine whether the iteration is terminated or not by using a simulated annealing criterion.

[0064] The construction unit 22 comprises:

[0065] The trained DQN model is used to replace the operator selection part in the ALNS as the model of the operator selection, so as to form a hierarchical planning model.

[0066] The hierarchical planning model comprises:

[0067]

[0068] Wherein, S is a state space, A is an action space, γ is a discount factor, α is a learning rate, τ is a temperature parameter;

[0069] Q(s,a,w) is a value function, D is an initialized experience replay buffer, N is a capacity, w is a parameter of a randomly initialized main Q network, and w T is a parameter of a randomly initialized target Q network, and π is a policy function.

[0070] The iteration unit 24 comprises:

[0071] The trained DQN model is used to replace the operator selection part in the ALNS as the model of the operator selection, so as to form a hierarchical planning model.

[0072] The trained DQN model is used to replace the operator selection part in the ALNS as the model of the operator selection, so as to form a hierarchical planning model.

[0073] The trained DQN model is used to replace the operator selection part in the ALNS as the model of the operator selection, so as to form a hierarchical planning model.

[0074] The trained DQN model is used to replace the operator selection part in the ALNS as the model of the operator selection, so as to form a hierarchical planning model.

[0075] The network parameters of the DQN model are trained by minimizing the objective function.

[0076] The working principle of the UAV reconnaissance path planning device based on the DQN-ALNS solving method is the same as the principle of the UAV reconnaissance path planning method based on the DQN-ALNS solving method, and details are not repeated here.

[0077] The UAV reconnaissance path planning problem is provided, the mixed integer linear programming model considers the target priority, and is an extension of the vehicle path planning problem.

[0078] The application proposes a new algorithm of DQN-ALNS, which integrates deep Q network (DQN) and adaptive large neighborhood search (ALNS), and can explore the solution space according to the search state of the current solution.

[0079] The above technical solutions of the embodiments of the application will be described in detail below in combination with specific application examples, and technical details not introduced in the implementation process can be referred to the related description in the foregoing.

[0080] Embodiment 1

[0081] The application provides a DQN-ALNS solving unmanned aerial vehicle reconnaissance path planning method, and the optimization target of the application is to minimize the total time and the number of unmanned aerial vehicles used for completing the reconnaissance task by optimizing the flight path of each unmanned aerial vehicle for accessing the reconnaissance target, and simultaneously weighting the product of the reconnaissance target priority and the completion time of reconnaissance.

[0082] The parameter variable symbols used in the model have the meanings shown in Table 2:

[0083] Table 2 problem parameters

[0084]

[0085] Mixed integer linear programming mathematical model:

[0086]

[0087] s.t.

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095] y0=Q,u0=0 (13)

[0096] In this model, equation (1) minimizes the number of UAVs used, the total flight time of the reconnaissance missions, and the weighted product of the target priority and the reconnaissance time. Equations (2)-(4) are path planning constraints, and equations (5)-(7) are endurance time constraints. Equation (2) ensures that each target is only reconnoitered once. Equations (3) and (4) ensure that the number of fly-ins and fly-outs for the base and each target are equal. M is a positive number when it is infinite, and equation (5) ensures the feasibility of the continuity of the time after the UAV visits the target. Equation (6) ensures that the power is non-negative when the UAV leaves the reconnaissance target. Equation (7) ensures that the power is non-negative when arriving at the reconnaissance target and can return to the base. Equation (8) is a variable value constraint condition. Equation (9) defines the initial power of the UAV and the takeoff time.

[0097] DQN-ALNS algorithm:

[0098] DQN-ALNS considers the priority of the targets by integrating the DQN and ALNS methods. In the traditional ALNS algorithm, each destruction and repair operator is assigned a certain weight, which determines their selection probability in the search process. The effectiveness of the ALNS algorithm in finding solutions makes it the preferred method in many path planning problems in recent years. In addition, the application models the destruction-repair operator selection sequence as a Markov decision process, in which the agent receives a reward associated with the simulated annealing (SA) criterion. This method enhances the traditional ALNS algorithm by utilizing the agent's selection of destruction-repair operators in each iteration. The agent selects the operator based on the search state of the current solution, while the traditional roulette selection relies on the historical performance indicators of the operators. This indicates that the operator selection has higher potential within the DQN framework. See Table 3 for the pseudo-code of DQN-ALNS, and Algorithm 1 for details.

[0099] Table 3 DQN-ALNS algorithm

[0100]

[0101]

[0102] The construction process of the greedy initial solution is elaborated in detail in the subsequent application. Next, according to the trained DQN model, a set of destroy-repair operators is selected as actions to update the solution. Then, the SA criterion is used to evaluate the new solution to determine whether to accept the new solution. After a certain number of iterations, the best solution is output. The structure of DQN is adopted, which uses the Q value update mechanism instead of the traditional roulette update method in ALNS. Q value update is closely related to the state, so it can capture more detailed information to select the appropriate operator more accurately. In contrast, the roulette update method averages the benefits of the operators without considering the relevance of the operators to the current solution. The specific composition of the action space is introduced, which is a set of destroy-repair operators designed based on experience. These actions are composed of destroy and repair operators, which increase the diversity of solutions by constantly destroying and rebuilding the current solution. The agent updates the current solution by selecting actions. The DQN-ALNS acceptance criteria and termination conditions for new solutions are described in detail.

[0103] Constructing a greedy initial solution

[0104] To promote the rapid convergence of the algorithm and improve the quality of the solution, this study proposes an initial solution generation algorithm based on greedy heuristic. The algorithm starts from base 0 and selects reconnaissance targets in turn according to a specific metric - priority density, until all targets are covered. Assuming that the previous visited target is i, the priority density vector is the ratio of the priority vector and the distance vector , as shown in equation (10).

[0105]

[0106] Then, the target with high priority density is preferentially visited without violating the constraint condition. If the constraint condition is violated, an additional flight path is added from base 0.

[0107] DQN

[0108] This application uses the Deep Q Network (DQN) method to optimize the operator selection of ALNS. The operator selection is considered as an independent Markov decision process (MDP), in which the agent selects the best combination of destroy-repair operators. Finally, the learned model is integrated into the ALNS loop for selecting operators during the SA process.

[0109] MDP is a tuple (S, A, P, R). In each state s e S, the agent selects an action a e A from the action space, and receives a reward r according to the reward function R(s, a). Afterwards, the state transitions to a new state s' depending on P(s'|s, a), the transition function that controls the environment changes. The interaction happens in episodes, each of which is a finite sequence of (s, a, r, s') pairs until a terminal state is reached. The agent specifies its actions through a policy, which is a function that selects actions, π(a|s). The state-action value function Q(s, a) is the expected reward the agent gets by choosing action a in state s.

[0110] State:

[0111] As shown in Table 4, each state contains features from the problem instance, the change of solution, and the history of run. Static features, such as the location, priority, and required surveillance power of each surveillance target, are closely related to the nature of the problem, as they remain unchanged during the iterative search process. Features associated with the current solution cover the surveillance path constructed based on the current solution and the search process itself. For example, given a surveillance path, the remaining power of each surveillance target on the flight path is calculated by the present application. The objective function value of the current solution, the history optimal value, and the search generation, provide the agent with an overview of the current solution in the search process. The objective function difference, temperature, and the number of non-improving steps inform the agent how likely the new solution is accepted. The objective function difference, temperature, and the number of non-improving steps, together determine the probability of the agent accepting the new solution. The reduction of the objective function, the change of solution, and the history of search, reflect the immediate change from the previous solution to the current solution. In particular, the reduction of the objective function reveals the difference of the objective function between the previous solution and the current solution. The change of solution indicates whether the solution has changed from the previous step to the current step. The history of search indicates whether the current solution has been encountered in the search process. Finally, according to the research of Oding et al., the features of the history of run include the action taken recently and its impact. For example t-1 represents the action taken 1 step before the current step t, which affects e t-1 the value of the objective function, that is, the reward introduced later.

[0112] Table 4 State Features

[0113]

[0114]

[0115] Action and Transition Function:

[0116] The action space A consists of combinations of destroy-repair operators that can be applied to the current solution at each iteration, which will be described in detail in Section 4.4. At each iteration of the DQN-ALNS algorithm, the agent selects an action a e A and applies it to the current solution.

[0117] The transition function mainly reflects the change of the solution in the search process. Similar to ALNS, after the agent applies the selected action to the current solution, the destroy operator removes several nodes from the current solution and puts them into the deletion list. Then, the repair operator reinserts the nodes from the deletion list into the solution, empties the deletion list, and generates a new solution. Since the construction of the new solution and the acceptance of the new solution are two independent processes, whether the new solution is accepted needs to be determined according to the simulated annealing (SA) criterion.

[0118] The reward function:

[0119] At the end of the episode when the agent reaches the next state, the reward it obtains can be used to evaluate the improvement of the solution quality by the acceptance of the new solution. Specifically, a good reward function should strike a balance between providing continuous incentives and avoiding excessive rewards, while ensuring that the agent does not receive rewards for not actually optimizing the expected goal. To this end, this application adopts a reward function (11) with the above properties, which is similar to the mechanism for scoring the performance of operators in the ALNS framework, aiming to encourage the agent to find better solutions than the current solution and give higher reward scores to such solutions. In addition, if a slightly worse solution is found, but as long as it is accepted by the acceptance criterion, a small amount of reward will also be given.

[0120]

[0121] Training method:

[0122] DQN is a model-free RL method for solving MDPs, which relies on estimating the state-action value function Q(s, a). As shown in Table 5, unlike the approximate estimation of general temporal difference methods, DQN uses a deep neural network to approximate the value function, while training the network parameters by minimizing the objective function (12).

[0123]

[0124] where γ represents the discount factor, Q(s, a, w) represents the main Q network, and Q(s, a, w) represents the target Q network. The interaction of the agent and the environment generates tuples (s, a, r, s'), and the network parameters w are updated according to Algorithm 2. After the training process is complete, the optimal policy * (a|s, w) will be used to solve unknown instances in the test set.

[0125]

[0126] Table 5 Deep-Q network training algorithm 2

[0127]

[0128] As mentioned before, the learned policy can take actions more effectively with respect to the state-action value function, always choosing the action with the highest expected cumulative reward. Both the traditional ALNS roulette wheel update and the Q-value update use a weighting factor to balance the performance estimates representing history and current. The key difference is that the Q-value update is conditioned on the state, thus capturing more information that can be used to select relevant operators, while the roulette wheel update simply averages the gains of the operators regardless of whether they are suitable for the current solution. However, when integrating the trained learning model into ALNS, some problems can be encountered. Because the objective function weighting changes the shape of the search space, making some regions steeper or flatter, and due to the nature of the greedy policy, it can reduce the ability of the search process to escape from local optimal solutions. To expand the search range and obtain a probabilistic policy, the SoftMax function as shown in equation (14) is adopted, where the temperature parameter τ is used to adjust the degree of greediness of the policy. Specifically, when τ tends to infinity, the selection probabilities of each action tend to be uniform; while when τ tends to zero, the selection probability of the action with the highest expected cumulative reward approaches 1.

[0129]

[0130] Combination of destroy-repair operators:

[0131] The action space A includes the combination of destroy-repair operators that can be applied to the current solution in each iteration, and there is no limit to the size of the action space A. Similar to the operations of ALNS destroy and repair solutions, an action a∈A is composed of two parts, a destroy operator and a repair operator, which increases the diversity of solutions by constantly destroying and reconstructing the current solution. The DRL agent selects an action to construct a new solution. Each action is a combination of one destroy operator and one repair operator in Table 3 and Table 4. Therefore, the size of the action space |A| is 24 (6 destroy operators x 4 repair operators).

[0132] Destroy operator:

[0133] Table 6 lists all the destroy operators, which delete nodes in a targeted manner within and between paths. The random destroy operator randomly deletes nodes at a rate of 0.1 to 0.4 to intensify the disturbance of the solution. The worst value destroy operator deletes 5 to 20 nodes with the largest impact on the objective function by sorting them in descending order of their impact on the objective function. The correlation destroy operator deletes 4 to 8 nodes with the largest correlation to other nodes by randomly selecting two nodes, calculating the correlation of other nodes to them, and deleting the 4 to 8 nodes with the largest correlation to adjust and optimize the current solution. The worst path delete operator deletes 1 to 3 paths with the largest impact on the objective function by sorting them in descending order of their impact on the objective function. The shortest time delete operator deletes 1 to 3 paths with the earliest return time of the UAV to improve the utilization of the UAV. The minimum priority delete operator calculates the sum of the priorities of the nodes in each path and deletes the path with the smallest sum.

[0134] Table 6 destroy operator list

[0135]

[0136] Repair operators:

[0137] Table 7 lists all the repair operators, which are the values added to the objective function due to the insertion of nodes at corresponding positions. The greedy repair operator focuses on local optimization by selecting the position with the smallest repair cost to optimize the insertion of nodes. The regret repair operator considers the long-term effect by calculating the regret values of the first 5 nodes and inserting the unassigned nodes according to the greedy strategy. The maximum variance repair operator focuses on the variance of the inserted nodes by sorting them to optimize the insertion order to improve stability. The priority repair operator prioritizes nodes with high priority to ensure that nodes with high priority are repaired first.

[0138] Table 7 repair operator list

[0139]

[0140] Acceptance criteria and stopping conditions

[0141] The acceptance criteria in simulated annealing is used. The acceptance criteria depends on the difference in the objective function between the existing solution X and the new solution X', denoted as ΔE = Π(x') - Π(x), and the temperature parameter T, which gradually decreases throughout the search process. If the objective function of a new solution is lower than that of the current solution, i.e., ΔE < 0, then the new solution is always accepted. In addition, the probability of accepting a worse new solution is e -ΔE / T To determine the initial temperature T0, all solutions of the first 100 iterations of the acceptance search are accepted, and all unimproved new solutions, i.e., ΔE > 0, are recorded, as shown in equation (15). Then, the average value of these positive increments is calculated It can be obtained that:

[0142]

[0143] To reduce the temperature, the cooling schedule of Crama and Schyn is used, and the search is terminated when a certain number of iterations is reached.

[0144] The application makes the selection probability of the operator quickly adjust in the iteration process, so that the DQN-ALNS can utilize new information in time in the search process and plan a better UAV reconnaissance path. The application provides a mixed integer linear programming model of the UAV reconnaissance path planning problem, which considers target priority and is an extension of the vehicle path planning problem. Small-scale problems are solved by using the Gurobi solver, and sensitivity analysis is performed on the power of the UAV and the target priority under different distributions. The application proposes a new algorithm of DQN-ALNS, which integrates deep Q network (DQN) and adaptive large neighborhood search (ALNS), and can explore the solution space according to the search state of the current solution.

[0145] It should be understood that the specific order or hierarchy of steps in the processes disclosed is an example. Based upon design preferences, it should be understood that the specific order or hierarchy of steps in the processes can be re-arranged while remaining within the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, and as such claims should not be construed as necessarily limited to the particular opening of closing sequence.

[0146] In the above detailed description, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This disclosed approach is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are explicitly recited in each claim. On the contrary, as reflected in the appended claims, the inventiveness lies in less than all features of the disclosed single embodiments. Thus, the appended claims are hereby expressly incorporated into the detailed description, with each claim acting as a separate embodiment of the application.

[0147] The disclosed embodiments are to be considered in all respects as illustrative and not restrictive, since additional embodiments will become apparent to those skilled in the art in view of the foregoing description. The particular embodiments disclosed above are reflective of the inventiveness required to be claimed by patent laws and are felt to be points in a space of possibilities as vast as the human imagination. While the present disclosure has been illustrated by a description of various embodiments, it is not the intention of the applicant to restrict or in any way limit the scope of the appended claims to such detail. Additional modifications, within the scope of the present disclosure, will readily occur to those skilled in the art. The disclosure having now been described by way of example, those skilled in the art will be aware of modifications and variations which fall within the scope of the appended claims. The various embodiments described herein can be implemented in hardware, software, or a combination thereof. The various embodiments and / or modules can each be implemented and / or executed by, for example, one or more computer processors at one or more locations.

[0148] The above description includes examples of one or more embodiments. Of course, not all possible combinations of components or methods described above will be employed to make or use the embodiments nor will all of the following described examples necessarily be realized. One of ordinary skill in the art, however, having the benefit of the present description, can understand how to make and use variations of the embodiments under the teachings and concepts described herein. Thus, the embodiments described herein are intended to embrace all such alterations, modifications, and variations that fall within the scope of the appended claims. Furthermore, the terms "comprises", "comprising", "includes", "including", "has", "having" and the like are to be construed open-ended, as "comprising", "including" and "having" are to be interpreted in the same manner as "consisting of", "consisting essentially of" and "substantially consisting of" under 35 U.S.C. § 112, Paragraph 6, as that terminology is interpreted in the context of the specification as a whole. Additionally, the terms "a" and "an" are defined as "one or more" in the context of the specification as a whole.

[0149] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, and steps described in connection with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.

[0150] The various illustrative logical blocks and modules described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the general purpose processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other such configuration.

[0151] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0152] In one or more exemplary designs, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or data

[0153] The above detailed description describes the purpose, technical solutions and advantages of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the present application.

Claims

1. A UAV reconnaissance path planning method solved by DQN-ALNS, characterized by: The method comprises the following steps: Train the DQN model; Integrate the trained DQN model into the adaptive large neighborhood search (ALNS) to build a hierarchical planning model. Outputting the UAV reconnaissance path using the hierarchical planning model; The UAV reconnaissance path is continuously updated and iterated through the destruction and repair operators of the hierarchical planning model; Use the simulated annealing criterion to determine whether the iteration is completed; The method comprises the following steps: using a mixed integer programming modeling method to construct a UAV reconnaissance path planning problem model that takes into account the reconnaissance target priority, in which a minimization objective function is constructed based on the number of UAVs used, the total flight time of the reconnaissance mission, and the weighted product of the target priority and the reconnaissance time; The trained DQN model is integrated into ALNS to build a hierarchical planning model, including: The trained DQN model is used as the operator selection model to replace the operator selection part in ALNS to form a hierarchical planning model; The hierarchical planning model includes: Among them: S is the state space, A is the action space; is the temperature parameter; Q(s,a,w) is the value function; w is the parameter of the randomly initialized main Q network; π is the policy function.

2. The UAV reconnaissance path planning method solved by DQN-ALNS according to claim 1 is characterized in that: The destruction and repair operators of the hierarchical planning model are used to continuously iterate the UAV reconnaissance path; including: Based on the trained DQN model, a set of damage repair operators are selected as actions to update the drone reconnaissance path; Use the simulated annealing criterion SA criterion to evaluate the updated UAV reconnaissance path and determine whether to accept the updated UAV reconnaissance path; Output the drone reconnaissance path accepted by the simulated annealing criterion SA criterion.

3. The UAV reconnaissance path planning method solved by DQN-ALNS according to claim 1 is characterized in that: The training of the DQN model specifically includes: The network parameters of the DQN model are trained by minimizing the objective function.

4. A DQN-ALNS-based UAV reconnaissance path planning device, wherein the device adopts the DQN-ALNS-based UAV reconnaissance path planning method according to claim 1, characterized in that: include: Training unit, used to train the DQN model; A construction unit for integrating the trained DQN model into ALNS to build a hierarchical planning model; An output unit, configured to output a UAV reconnaissance path using the hierarchical planning model; An iteration unit, configured to continuously update and iterate the UAV reconnaissance path through the destruction and repair operators of the hierarchical planning model; The termination unit is used to determine whether the iteration is finished using the simulated annealing criterion.

5. The UAV reconnaissance path planning device solved by DQN-ALNS according to claim 4 is characterized in that: The building block comprises: It is used to replace the operator selection part in ALNS with the trained DQN model as the operator selection model to form a hierarchical planning model.

6. The UAV reconnaissance path planning device solved by DQN-ALNS according to claim 4 is characterized in that: The iteration unit comprises: Used to select a set of destruction and repair operators as actions to update the drone reconnaissance path based on the trained DQN model; Used to evaluate the updated UAV reconnaissance path using the SA criterion and determine whether to accept the updated UAV reconnaissance path; Used to output drone reconnaissance paths accepted by SA guidelines.

7. The UAV reconnaissance path planning device solved by DQN-ALNS according to claim 4 is characterized in that: The training of the DQN model specifically includes: The network parameters of the DQN model are trained by minimizing the objective function.

Images