UAV collaborative search method and system based on heuristic Q-learning algorithm

Through the heuristic Q-learning algorithm-based drone collaborative search method, the problem of low drone search efficiency and reliability is solved, efficient and fast search path planning and target discovery is achieved, and the drone search is improved Overall performance.

CN114625167BActive Publication Date: 2024-11-01SHANDONG NEW GENERATION INFORMATION IND TECH RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210281173.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-22
Publication Date
2024-11-01
Estimated Expiration
2042-03-22

AI Technical Summary

Technical Problem

The existing drone search methods are inefficient and have low reliability, especially when the prediction time domain is too long, the real-time requirement cannot be met. The classic rolling time domain planning method does not consider the value of the adjacent grid, so there is low reliability. question.

Method used

Adopt the heuristic Q-learning algorithm to collaborative search method, and design a search reward mechanism by establishing a drone dynamic model and search environment perception map, and use the heuristic Q-learning algorithm to plan the search path to realize the drone The best search for. The method includes information interoperability between multiple drones and fusion of search environment map perception information, updating the search environment perception map until all targets are searched.

Benefits of technology

Improve the efficiency and reliability of drone search, realize a multi-drone search solution with strong robustness and short single-step decision-making time, can quickly complete search tasks, increase the probability of discovering targets and reduce the completion of tasks. time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114625167B_ABST
    Figure CN114625167B_ABST
Patent Text Reader

Abstract

The invention discloses a UAV collaborative search method and system based on a heuristic Q-learning algorithm, which belongs to the field of UAV collaborative search. The technical problem to be solved by the invention is that the existing UAV search method is inefficient and has low reliability. The technical scheme is as follows: The method is specifically as follows: S1, establishing a UAV power model and a search environment perception map; S2, establishing a UAV search reward and a heuristic reward mechanism: based on the UAV power model, the environment perception map and the unmanned state established in step S1, an excellent UAV search reward mechanism is designed; S3, the UAV adopts a heuristic Q-learning algorithm, and utilizes the UAV search reward mechanism established in step S2 to explore the search area, and plans the search path to achieve the optimal search of the UAV; S4, the UAV updates the search environment perception map according to the search result of step S3; S5, the attribute values ​​of multiple UAV environment perception maps are fused; S6, repeating steps S3 and S5 until all targets are searched.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle collaborative search, and in particular to a method and system for unmanned aerial vehicle collaborative search based on a heuristic Q-learning algorithm. Background Art

[0002] In recent years, drones have been widely used in post-disaster rescue, target strikes, military confrontation and other fields. Drone search is an important field of drones. The drone flies above the search sub-area, uses the onboard sensors to explore the area, and designs an excellent drone search strategy so that the drone can capture the moving target in the area as quickly as possible. Therefore, in order to improve the efficiency of drone search, various strategies have been proposed. The classic rolling horizon planning generates the optimal search path online and is widely used in the drone search process.

[0003] The classic rolling horizon planning selects the extreme value of the prediction path without considering the value of the adjacent grid. In addition, when the prediction horizon is too long, the single-step planning time is too long and cannot meet the real-time requirements. Relying on various numerical solutions or heuristic algorithms, it is often required that the objective function is differentiable or differentiable, which has the problem of low reliability during use. Summary of the invention

[0004] The technical task of the present invention is to provide a UAV collaborative search method and system based on a heuristic Q-learning algorithm to solve the problems of low efficiency and low reliability of existing UAV search methods.

[0005] The technical task of the present invention is achieved in the following way: a UAV collaborative search method based on a heuristic Q-learning algorithm, the method is as follows:

[0006] S1. Establish UAV power model and search environment perception map;

[0007] S2. Establishing drone search reward and heuristic reward mechanism: Based on the drone power model, environmental perception map and unmanned state established in step S1, design an excellent drone search reward mechanism;

[0008] S3, the drone uses the heuristic Q-learning algorithm and the drone search reward mechanism established in step S2 to explore the search area and plan the search path to achieve the optimal search of the drone;

[0009] S4. The drone updates the search environment perception map according to the search results of step S3;

[0010] S5. Attribute value fusion of multi-UAV environment perception map: UAVs communicate with UAVs within the communication range, exchange and fuse the perception information of the search environment map;

[0011] S6. Repeat steps S3 to S5 until all targets are searched.

[0012] Preferably, the establishment of the environment search map in step S1 is specifically as follows:

[0013] Rasterize the search area into a discretized grid map of N*M;

[0014] The length and width of the grid are dx and dy respectively; dx and dy are the distances that the UAV flies at an average level flight speed for a planned duration.

[0015] Preferably, the environment search map in step S1 is divided into a target existence probability map and an uncertainty map;

[0016] The target existence probability map model is established as follows:

[0017] (1) Define p(x,y,k)∈(0,1) to represent the probability that the target is located at (x,y) at time k; p(x,y,0)=0.5 means that the probability that the target exists at (x,y) or not exists at (x,y) at the initial time is the same. The formula for p(x,y,0) is as follows:

[0018]

[0019] Among them, v n Indicates the peak width of target existence probability; c n Indicates the probability peak of the target; (x n ,y n ) indicates where the target may appear;

[0020] (2) During the detection process, the UAV has a detection probability p d and false alarm probability p f , which is defined as:

[0021] p(Z(x,y,k)=1|δ(x,y,k)=1)=p d ;

[0022] p(Z(x,y,k)=1|δ(x,y,k)=0)=p f ;

[0023] Among them, Z(x,y,k) represents the target detection result of UAV k at (x,y); δ(x,y,k) represents the existence of target k at (x,y);

[0024] (3) As the drone searches the area, the attribute values ​​of the target existence probability map are continuously updated according to the Bayesian theorem; the update formula of the target existence probability map is:

[0025]

[0026]

[0027] Among them, φ k represents the detection range of the drone at time k;

[0028] The uncertainty map model is established as follows:

[0029] (1) Define χ(x, y, k) to represent the uncertainty information of (x, y) at time k and initialize the uncertainty map; the formula is as follows:

[0030] χ(x,y,0)=-p(x,y,0)log2 p(x,y,0);

[0031] (2) The uncertain map attribute value decays exponentially with the increase in the number of drone visits, and the formula is as follows:

[0032] χ(x,y,k)=τ*χ(x,y,k)

[0033] Among them, τ∈(0,1) represents the decay exponent.

[0034] Preferably, the drone search reward and heuristic reward mechanism in step S2 are as follows:

[0035] S201, search reward r is composed of r A , r B , r C , r D It consists of four parts, and the formula is as follows:

[0036] r=w1r A +w2r B +w3r C +w4r D ;

[0037] Among them, r A Indicates the probability value reward of the target; r B represents the uncertain value reward; r C represents the flight cost reward; r D represents the expected search reward;

[0038] S202, target existence probability value reward r A The settings are as follows:

[0039] r A=(1-σ(x,y,k))*p(x,y,k);

[0040]

[0041] Among them, σ(x,y,k) indicates whether the drone believes that there is a target at (x,y); when p(x,y,k)>p_max, the drone believes that there is a target at (x,y);

[0042] S203, uncertain value reward r B The settings are as follows:

[0043] r B =χ(x,y,k+1)-χ(x,y,k);

[0044] S204, Flight Cost Reward r C The settings are as follows:

[0045]

[0046] S205, expected exploration cost r D The settings are as follows:

[0047]

[0048] S206, heuristic reward mechanism setting: the heuristic reward mechanism attracts the drone to move to the place where the probability of the global target is the largest and away from the map boundary;

[0049] S207, heuristic reward and search reward together constitute the reward mechanism of heuristic reinforcement learning, the formula is as follows:

[0050]

[0051] Among them, D and F represent the heuristic factors; D represents the distance from any position on the map to the position with the maximum probability of the global map target; F represents the distance from the position of the map drone to the ground. Figure 4 The minimum distance between the edges; c and d represent the adjustment coefficients.

[0052] Preferably, the heuristic Q-learning algorithm in step S3 is as follows:

[0053] S301, establish a Q table, initialize Q(s,a) to an arbitrary value, and initialize the tracking trace e(s,a)=0;

[0054] S302, loop:

[0055] Initialize drone state s;

[0056] Loop (for each step in the round):

[0057] According to the deep learning Q table, action a is selected according to the heuristic action selection strategy

[0058] Take action a, get heuristic reward r and next state s'

[0059]

[0060] e(s,a)←1

[0061] s←s′

[0062] Where e(s,a) is the qualification trace of the drone taking action a in state s; γ is the discount factor; δ is the increment of Q.

[0063] Update the value function and qualification trace of all states s' of the drone

[0064] Q(s,a)←Q(s,a)+αδe(s,a)

[0065] e(s,a)←γλe(s,a)

[0066] In the formula, α is the learning rate and λ is the learning step size.

[0067] Until the round ends

[0068] Until all Q(s,a) converge;

[0069] S303, output the final strategy, the formula is as follows:

[0070]

[0071] In the formula, π * is the optimal strategy of the drone; Q(s,a) is the action value function value of the drone performing action a in state s;

[0072] A heuristic action selection mechanism is set up, and the heuristic information of the harmonic function E(s,a) is used to improve the action selection strategy, so that the UAV tends to drive to the area with the maximum probability value of the target. The formula is as follows:

[0073]

[0074] Where E(s,a) is the harmonic function value of the drone performing action a in state s; ε∈(0,1) is the random probability value; a random Randomly select an action for the action set;

[0075] When the agent selects an action in state s, it has an ε probability of selecting the action with the largest sum of the value function and the harmonic function in this state, and a 1-ε probability of randomly selecting an action. The formula is as follows:

[0076]

[0077] Among them, p k represents the position of the drone at time k; p p_max Indicates the location where the target has the largest probability value; η represents a positive coefficient, which is used to adjust the size of the harmonic function E(s,a).

[0078] Preferably, the environment perception map update in step S4 is specifically as follows:

[0079] S401, the UAV explores the environment under the guidance of the UAV search method based on the heuristic Q-learning algorithm and enters a new exploration area;

[0080] S402: The UAV updates the target existence probability map and the uncertainty map according to the exploration results of the UAV and the updating rules of the environment perception map.

[0081] Preferably, the attribute value fusion of the multi-UAV environment perception map in step S5 is specifically as follows:

[0082] During the flight, the distributed UAVs communicate with the UAVs within the communication range, exchange environment perception map information, and fuse the attribute values ​​of the environment perception map. The formula is as follows:

[0083]

[0084]

[0085] Where n represents the number of drones within the communication range (including itself); v i,k (x, y) represents the attribute value (target existence probability value, uncertainty value, etc.) of the environment perception map at (x, y) of UAV i at time k; V k (x, y) represents the attribute value of the environment perception map after information fusion at (x, y) at time k.

[0086] A UAV collaborative search system based on heuristic Q-learning algorithm, the system includes:

[0087] Establish module 1, which is used to establish the UAV power model and search environment perception map;

[0088] Establish Module 2 to establish drone search reward and heuristic reward mechanism;

[0089] The planning module is used to use the heuristic Q-learning algorithm and the drone search reward mechanism to explore the search area and plan the search path to achieve the optimal search of the drone;

[0090] An updating module, used to update the search environment perception map according to the search results;

[0091] The fusion module is used to fuse the attribute values ​​of the multi-UAV environment perception map, that is, the UAVs communicate with the UAVs within the communication range, exchange and fuse the search environment map perception information.

[0092] An electronic device comprising: a memory and at least one processor;

[0093] Wherein, the memory stores a computer program;

[0094] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the drone collaborative search method based on the heuristic Q-learning algorithm as described above.

[0095] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the above-mentioned drone collaborative search method based on the heuristic Q-learning algorithm.

[0096] The UAV collaborative search method and system based on the heuristic Q-learningG algorithm of the present invention have the following advantages:

[0097] (I) The present invention considers the grid mean, is data-driven, has nothing to do with derivatives, does not require prediction, and has excellent convergence and robustness. It also uses a heuristic reward mechanism to design reasonable rewards, and uses prior knowledge to guide the intelligent agent to better complete tasks during operation. It also makes online decisions on the best search path to improve search efficiency.

[0098] (ii) The present invention provides a multi-UAV search solution with strong robustness, short single-step decision time, and high search efficiency, thereby guiding the UAVs to quickly complete the search task;

[0099] (iii) Compared with the existing technology, the present invention improves the search efficiency of the drone with a shorter single-step search;

[0100] (iv) The present invention realizes the rapid collaborative search of fixed areas by drones. The drones scan and collect information about environmental sub-areas, and self-learn the environment to search for the optimal path. The robustness is strong, and the search efficiency of drones is improved.

[0101] (V) The heuristic information preset in the present invention guides the UAV to move away from the boundary and move more quickly to the area where the target is more likely to exist, thereby increasing the probability of finding the target, improving the UAV search efficiency, and reducing the time it takes for the UAV to complete the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] The present invention is further described below in conjunction with the accompanying drawings.

[0103] Attached Figure 1 It is a flowchart of the UAV collaborative search method based on the heuristic Q-learning algorithm;

[0104] Attached Figure 2 Environmental grid map and UAV dynamics model map;

[0105] Attached Figure 3 This is a diagram of the reinforcement learning framework;

[0106] Attached Figure 4 It is the framework diagram of the heuristic Q-learning algorithm;

[0107] Attached Figure 5 The initial position distribution map of the UAV and the search target;

[0108] Attached Figure 6 Initial target existence probability map for the UAV;

[0109] Attached Figure 7 Initial uncertainty map for the drone;

[0110] Attached Figure 8 The probability map of the target existence of UAV 1 when the mission is completed;

[0111] Attached Fig. 9 The probability map of the target existence of UAV 2 when the mission is completed;

[0112] Attached Fig.10 This is the probability map of the existence of UAV 3 target when the mission is completed. DETAILED DESCRIPTION

[0113] The UAV collaborative search method and system based on the heuristic Q-learning algorithm of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0114] Embodiment 1:

[0115] As attached Figure 1 As shown, this embodiment provides a UAV collaborative search method based on a heuristic Q-learning algorithm, and the method is specifically as follows:

[0116] S1. Establish UAV power model and search environment perception map;

[0117] S2. Establishing drone search reward and heuristic reward mechanism: Based on the drone power model, environmental perception map and unmanned state established in step S1, design an excellent drone search reward mechanism;

[0118] S3, the drone uses the heuristic Q-learning algorithm and the drone search reward mechanism established in step S2 to explore the search area and plan the search path to achieve the optimal search of the drone;

[0119] S4. The drone updates the search environment perception map according to the search results of step S3;

[0120] S5. Attribute value fusion of multi-UAV environment perception map: UAVs communicate with UAVs within the communication range, exchange and fuse the perception information of the search environment map;

[0121] S6. Repeat steps S3 to S5 until all targets are searched.

[0122] The specific details of establishing the environment search map in step S1 of this embodiment are as follows:

[0123] Rasterize the search area into a discretized grid map of N*M;

[0124] The length and width of the grid are dx and dy respectively; dx and dy are the distances that the UAV flies at an average level flight speed for a planned duration.

[0125] The drone is a fixed-wing drone. Subject to dynamic constraints, the drone turns 45 degrees left, goes straight, and turns 45 degrees right within each planning time, as shown in the attached Figure 2 shown.

[0126] The environment search map in step S1 of this embodiment is divided into a target existence probability map and an uncertainty map;

[0127] As attached Figure 6 As shown, the target existence probability map model is established as follows:

[0128] (1) Define p(x,y,k)∈(0,1) to represent the probability that the target is located at (x,y) at time k; p(x,y,0)=0.5 means that the probability that the target exists at (x,y) or not exists at (x,y) at the initial time is the same. The formula for p(x,y,0) is as follows:

[0129]

[0130] Among them, v n Indicates the peak width of target existence probability; c n Indicates the probability peak of the target; (x n ,yn ) indicates where the target may appear;

[0131] (2) During the detection process, the UAV has a detection probability p d and false alarm probability p f , which is defined as:

[0132] p(Z(x,y,k)=1|δ(x,y,k)=1)=p d ;

[0133] p(Z(x,y,k)=1|δ(x,y,k)=0)=p f ;

[0134] Among them, Z(x,y,k) represents the target detection result of UAV k at (x,y); δ(x,y,k) represents the existence of target k at (x,y);

[0135] (3) As the drone searches the area, the attribute values ​​of the target existence probability map are continuously updated according to the Bayesian theorem; the update formula of the target existence probability map is:

[0136]

[0137] Among them, φ k represents the detection range of the drone at time k;

[0138] As attached Figure 7 As shown, the establishment of uncertainty map model is as follows:

[0139] (1) Define χ(x, y, k) to represent the uncertainty information of (x, y) at time k and initialize the uncertainty map; the formula is as follows:

[0140] χ(x,y,0)=-p(x,y,0)log2 p(x,y,0);

[0141] (2) The uncertain map attribute value decays exponentially with the increase in the number of drone visits, and the formula is as follows:

[0142] χ(x,y,k)=τ*χ(x,y,k)

[0143] Among them, τ∈(0,1) represents the decay exponent.

[0144] As attached Figure 3 As shown, the drone search reward and heuristic reward mechanism in step S2 of this embodiment are specifically as follows:

[0145] S201, search reward r is composed of r A , r B , r C, r D It consists of four parts, and the formula is as follows:

[0146] r=w1r A +w2r B +w3r C +w4r D ;

[0147] Among them, r A Indicates the probability value reward of the target; r B represents the uncertain value reward; r C represents the flight cost reward; r D represents the expected search reward;

[0148] S202, target existence probability value reward r A The settings are as follows:

[0149] r A =(1-σ(x,y,k))*p(x,y,k);

[0150]

[0151] Among them, σ(x,y,k) indicates whether the drone believes that there is a target at (x,y); when p(x,y,k)>p_max, the drone believes that there is a target at (x,y);

[0152] S203, uncertain value reward r B The settings are as follows:

[0153] r B =χ(x,y,k+1)-χ(x,y,k);

[0154] S204, Flight Cost Reward r C The settings are as follows:

[0155]

[0156] S205, expected exploration cost r D The settings are as follows:

[0157]

[0158] S206, heuristic reward mechanism setting: the heuristic reward mechanism attracts the drone to move to the place where the probability of the global target is the largest and away from the map boundary;

[0159] S207, heuristic reward and search reward together constitute the reward mechanism of heuristic reinforcement learning, the formula is as follows:

[0160]

[0161] Among them, D and F represent the heuristic factors; D represents the distance from any position on the map to the position with the maximum probability of the global map target; F represents the distance from the position of the map drone to the ground. Figure 4 The minimum distance between the edges; c and d represent the adjustment coefficients.

[0162] As attached Figure 4 As shown, the heuristic Q-learning algorithm in step S3 of this embodiment is specifically as follows:

[0163] S301, establish a Q table, initialize Q(s,a) to an arbitrary value, and initialize the tracking trace e(s,a)=0;

[0164] S302, loop:

[0165] Initialize drone state s;

[0166] Loop (for each step in the round):

[0167] According to the deep learning Q table, action a is selected according to the heuristic action selection strategy

[0168] Take action a, get heuristic reward r and next state s'

[0169]

[0170] e(s,a)←1

[0171] s←s′

[0172] Where e(s,a) is the qualification trace of the drone taking action a in state s; γ is the discount factor; δ is the increment of Q.

[0173] Update the value function and qualification trace of all states s' of the drone

[0174] Q(s,a)←Q(s,a)+αδe(s,a)

[0175] e(s,a)←γλe(s,a)

[0176] In the formula, α is the learning rate and λ is the learning step size.

[0177] Until the round ends

[0178] Until all Q(s,a) converge;

[0179] The key codes are as follows:

[0180] Repeat:

[0181] Initializes

[0182] Repeat (for each step in the round)

[0183] According to the Q table, select action a according to the heuristic action selection strategy

[0184] Take action a, get heuristic reward r and next state s'

[0185]

[0186] e(s,a)←1

[0187] s←s′

[0188] For all states'

[0189] Q(s,a)←Q(s,a)+αδe(s,a)

[0190] e(s,a)←γλe(s,a)

[0191] Until the round ends

[0192] Until all Q(s,a) converge;

[0193] S303, output the final strategy, the formula is as follows:

[0194]

[0195] In the formula, π * is the optimal strategy of the drone; Q(s,a) is the action value function value of the drone performing action a in state s;

[0196] A heuristic action selection mechanism is set up, and the heuristic information of the harmonic function E(s,a) is used to improve the action selection strategy, so that the UAV tends to drive to the area with the maximum probability value of the target. The formula is as follows:

[0197]

[0198] Where E(s,a) is the harmonic function value of the drone performing action a in state s; ε∈(0,1) is the random probability value; a random Randomly select an action for the action set;

[0199] When the agent selects an action in state s, it has an ε probability of selecting the action with the largest sum of the value function and the harmonic function in this state, and a 1-ε probability of randomly selecting an action. The formula is as follows:

[0200]

[0201] Among them, p krepresents the position of the drone at time k; p p_max Indicates the location where the target has the largest probability value; η represents a positive coefficient, which is used to adjust the size of the harmonic function E(s,a).

[0202] The environment perception map update in step S4 of this embodiment is specifically as follows:

[0203] S401, the UAV explores the environment under the guidance of the UAV search method based on the heuristic Q-learning algorithm and enters a new exploration area;

[0204] S402: The UAV updates the target existence probability map and the uncertainty map according to the exploration results of the UAV and the updating rules of the environment perception map.

[0205] The attribute value fusion of the multi-UAV environment perception map in step S5 of this embodiment is specifically as follows:

[0206] During the flight, the distributed UAVs communicate with the UAVs within the communication range, exchange environment perception map information, and fuse the attribute values ​​of the environment perception map. The formula is as follows:

[0207]

[0208]

[0209] Where n represents the number of drones within the communication range (including itself); v i,k (x, y) represents the attribute value (target existence probability value, uncertainty value, etc.) of the environment perception map at (x, y) of UAV i at time k; V k (x, y) represents the attribute value of the environment perception map after information fusion at (x, y) at time k.

[0210] In order to verify the search effect of the drone search method based on the heuristic Q-learning algorithm of the present invention, Matlab simulation is performed on the present invention to verify the feasibility and search efficiency of the present invention, as follows:

[0211] (1) Take a rectangular search area of ​​2km*2km and divide the search area into 20*20 square grids with a grid size of 100m*100m;

[0212] (2) The initial positions of the three drones are [450, 250], [1750, 1250], and [950, 450], and the initial positions of the three targets are [1450, 250], [800, 1500], and [1300, 700];

[0213] (3) The initial position distribution of the drone and the target is shown in the attached figure. Figure 5 As shown, the detection range of the drone is 300m*300m, the detection period is set to 4s, the detection probability is 0.95, and the false alarm probability is 0.015;

[0214] (4) In the heuristic Q-learning algorithm, the discount factor γ = 0.9, the learning rate α = 0.01, the greedy selection coefficient ε = 0.6, the trace discount factor λ = 0.9; and the uncertainty map attenuation exponent τ = 0.5.

[0215] The simulation results show that after 164 seconds, the UAV successfully searches for all targets in the area. Figure 8 , 9 As shown in Figure 10, the target existence probability map of drones 1, 2, and 3 after the mission is completed. The drones can cover most of the search area collaboratively and complete the coverage search of the area. The drone collaborative search method based on the heuristic Q-learning algorithm can better guide the drones to complete the search task, have better convergence and robustness, and improve the search efficiency of the drones.

[0216] Embodiment 2:

[0217] This embodiment provides a UAV collaborative search system based on a heuristic Q-learning algorithm, the system comprising:

[0218] Establish module 1, which is used to establish the UAV power model and search environment perception map;

[0219] Establish Module 2 to establish drone search reward and heuristic reward mechanism;

[0220] The planning module is used to use the heuristic Q-learning algorithm and the drone search reward mechanism to explore the search area and plan the search path to achieve the optimal search of the drone;

[0221] An updating module, used to update the search environment perception map according to the search results;

[0222] The fusion module is used to fuse the attribute values ​​of the multi-UAV environment perception map, that is, the UAVs communicate with the UAVs within the communication range, exchange and fuse the search environment map perception information.

[0223] Embodiment 3:

[0224] This embodiment also provides an electronic device, including: a memory and a processor;

[0225] Wherein, the memory stores computer-executable instructions;

[0226] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the drone collaborative search method based on the heuristic Q-learning algorithm in any embodiment of the present invention.

[0227] Embodiment 4:

[0228] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor, so that the processor executes the drone collaborative search method based on the heuristic Q-learning algorithm in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0229] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0230] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0231] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0232] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0233] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A UAV collaborative search method based on heuristic Q-learning algorithm, characterized in that: The method is as follows: S1. Establish UAV dynamics model and search environment perception map; S2. Establishing a drone search reward and heuristic reward mechanism: Based on the drone power model, environmental perception map, and unmanned state established in step S1, design a drone search reward mechanism; S3, the drone uses the heuristic Q-learning algorithm and the drone search reward mechanism established in step S2 to explore the search area and plan the search path to achieve the optimal search of the drone; S4. The drone updates the search environment perception map according to the search results of step S3; S5. Attribute value fusion of multi-UAV environment perception map: UAVs communicate with UAVs within the communication range, exchange and fuse the perception information of the search environment map; S6, repeat steps S3 to S5 until all targets are found; Among them, the drone search reward and heuristic reward mechanism are as follows: S201, search reward r is composed of r A , r B , r C , r D It consists of four parts, and the formula is as follows: r=w1r A +w2r B +w3r C +w4r D ; Among them, r A Indicates the probability value reward of the target; r B represents the uncertain value reward; r C represents the flight cost reward; r D represents the expected search reward; S202, target existence probability value reward r A The settings are as follows: r A =(1−σ(x,y,k))*p(x,y,k)? Among them, σ(x,y,k) indicates whether the drone believes that there is a target at (x,y); when p(x,y,k)>p_max, the drone believes that there is a target at (x,y); S203, uncertain value reward r B The settings are as follows: r B =χ(x,y,k+1)-χ(x,y,k); S204, Flight Cost Reward r C The settings are as follows: S205, expected exploration cost r D The settings are as follows: S206, heuristic reward mechanism setting: the heuristic reward mechanism attracts the drone to move to the place where the probability of the global target is the largest and away from the map boundary; S207, heuristic reward and search reward together constitute the reward mechanism of heuristic reinforcement learning, the formula is as follows: Among them, D and F represent heuristic factors; D represents the distance from any position on the map to the position with the maximum probability of the global map target; F represents the minimum distance from the position of the map drone to the four edges of the map; c and d represent adjustment coefficients; The heuristic Q-learning algorithm is as follows: S301, establish a Q table, initialize Q(s,a) to an arbitrary value, and initialize the tracking trace e(s,a)=0; S302, loop: Initialize drone state s; Loop (for each step in the round): According to the deep learning Q table, action a is selected according to the heuristic action selection strategy; Take action a and get the heuristic reward r and the next state s', which is as follows: e(s,a)←1; s←s′; Where e(s, a) is the qualification trace of the drone taking action a in state s; γ is the discount factor; δ is the increment of Q; The value function and qualification trace of all states s' of the drone are updated in the following form: Q(s,a)←Q(s,a)+αδe(s,a); e(s, a)←γλe(s, a); In the formula, α is the learning rate; λ is the learning step size; Until the round number of steps ends; Until all Q(s, a) converge; S303, output the final strategy, the formula is as follows: In the formula, π * is the optimal strategy of the drone; Q(s, a) is the action value function value of the drone performing action a in state s; A heuristic action selection mechanism is set up, and the heuristic information of the harmonic function E(s, a) is used to improve the action selection strategy, so that the UAV tends to drive to the area with the maximum probability value of the target. The formula is as follows: Where E(s, a) is the harmonic function value of the drone performing action a in state s; ε∈(0, 1) is the random probability value; a random Randomly select an action for the action set; When the agent selects an action in state s, it has an ε probability of selecting the action with the largest sum of the value function and the harmonic function in this state, and a 1-ε probability of randomly selecting an action. The formula is as follows: Among them, p k represents the position of the drone at time k; p p_max Indicates the location where the target has the largest probability value; η represents a positive coefficient, which is used to adjust the size of the harmonic function E(s, a).

2. The UAV collaborative search method based on the heuristic Q-learning algorithm according to claim 1 is characterized in that: The establishment of the environment search map in step S1 is specifically as follows: Rasterize the search area into a discretized grid map of N*M; The length and width of the grid are dx and dy respectively; dx and dy are the distances that the UAV flies at an average level flight speed for a planned duration.

3. The UAV collaborative search method based on heuristic Q-learning algorithm according to claim 1 is characterized in that: The environment search map in step S1 is divided into a target existence probability map and an uncertainty map; The target existence probability map model is established as follows: (1) Define p(x, y, k) ∈ (0, 1) to represent the probability that the target is located at (x, y) at time k; p(x, y, 0) = 0.5 means that the probability of the target being at (x, y) and not being at (x, y) at the initial time is the same. The formula for p(x, y, 0) is as follows: Among them, v n Indicates the peak width of target existence probability; c n Indicates the probability peak of the target; (x n ,y n ) indicates where the target may appear; (2) During the detection process, the UAV has a detection probability p d and false alarm probability p f , which is defined as: p(Z(x,y,k)=1|δ(x,y,k)=1)=p d 4 p(Z(x,y,k)=1|δ(x,y,k)=0)=p f 4 Among them, Z(x, y, k) represents the target detection result of UAV k at (x, y); δ(x, y, k) represents the existence of target k at (x, y) at time k; (3) As the drone searches the area, the attribute values ​​of the target existence probability map are continuously updated according to the Bayesian theorem; the update formula of the target existence probability map is: Among them, φ k represents the detection range of the drone at time k; The uncertainty map model is established as follows: (1) Define χ(x, y, k) to represent the uncertainty information of (x, y) at time k and initialize the uncertainty map; the formula is as follows: χ(x,y,0)=-p(x,y,0)log2p(x,y,0); (2) The uncertain map attribute value decays exponentially with the increase in the number of drone visits, and the formula is as follows: χ(x,y,k)=τ*χ(x,y,k); Wherein, τ∈(0, 1) represents the decay exponent.

4. The UAV collaborative search method based on heuristic Q-learning algorithm according to claim 1 is characterized in that: The environment perception map update in step S4 is specifically as follows: S401, the UAV explores the environment under the guidance of the UAV search method based on the heuristic Q-learning algorithm and enters a new exploration area; S402: The UAV updates the target existence probability map and the uncertainty map according to the exploration results of the UAV and the updating rules of the environment perception map.

5. The UAV collaborative search method based on the heuristic Q-learning algorithm according to any one of claims 1 to 4, characterized in that: The attribute value fusion of the multi-UAV environment perception map in step S5 is as follows: During the flight, the distributed UAVs communicate with the UAVs within the communication range, exchange environment perception map information, and fuse the attribute values ​​of the environment perception map. The formula is as follows: Where n represents the number of drones within the communication range; v i,l (x, y) represents the attribute value of the environment perception map at (x, y) of UAV i at time k; V k (x, y) represents the attribute value of the environment perception map after information fusion at (x, y) at time k.

6. A UAV collaborative search system based on heuristic Q-learning algorithm, characterized in that: The system includes, Establish module 1, which is used to establish the UAV power model and search environment perception map; Establish Module 2 to establish drone search reward and heuristic reward mechanism; The planning module is used to use the heuristic Q-learning algorithm and the drone search reward mechanism to explore the search area and plan the search path to achieve the optimal search of the drone; An updating module, used to update the search environment perception map according to the search results; The fusion module is used to fuse the attribute values ​​of the multi-UAV environment perception map, that is, the UAVs communicate with the UAVs within the communication range, exchange and fuse the search environment map perception information; Among them, the drone search reward and heuristic reward mechanism are as follows: S201, search reward r is composed of r A , r B , r C , r D It consists of four parts, and the formula is as follows: r=w1r A +w2r B +w3r C +w4r D ; Among them, r A Indicates the probability value reward of the target; r B represents the uncertain value reward; r C represents the flight cost reward; r D represents the expected search reward; S202, target existence probability value reward r A The settings are as follows: r A =(1-σ(x,y,k))*p(x,y,k)? Among them, σ(x, y, k) indicates whether the drone believes that there is a target at (x, y); when p(x, y, k)>p_max, the drone believes that there is a target at (x, y); S203, uncertain value reward r B The settings are as follows: r B =X(x,y,k+1)-χ(x,y,k): S204, Flight Cost Reward r C The settings are as follows: S205, expected exploration cost r D The settings are as follows: S206, heuristic reward mechanism setting: the heuristic reward mechanism attracts the drone to move to the place where the probability of the global target is the largest and away from the map boundary; S207, heuristic reward and search reward together constitute the reward mechanism of heuristic reinforcement learning, the formula is as follows: Among them, D and F represent heuristic factors; D represents the distance from any position on the map to the position with the maximum probability of the global map target; F represents the minimum distance from the position of the map drone to the four edges of the map; c and d represent adjustment coefficients; The heuristic Q-learning algorithm is as follows: S301, establish a Q table, initialize Q(s, a) to an arbitrary value, and initialize the tracking trace e(s, a) = 0; S302, loop: Initialize drone state s; Loop (for each step in the round): According to the deep learning Q table, action a is selected according to the heuristic action selection strategy; Take action a and get the heuristic reward r and the next state s', which is as follows: e(s,a)←1; s←s′; Where e(s, a) is the qualification trace of the drone taking action a in state s; γ is the discount factor; δ is the increment of Q; The value function and qualification trace of all states s' of the drone are updated in the following form: Q(s,a)←Q(s,a)+αδe(s,a); e(s, a)←γλe(s, a); In the formula, α is the learning rate; λ is the learning step size; Until the round number of steps ends; Until all Q(s, a) converge; S303, output the final strategy, the formula is as follows: In the formula, π * is the optimal strategy of the drone; Q(s, a) is the action value function value of the drone performing action a in state s; A heuristic action selection mechanism is set up, and the heuristic information of the harmonic function E(s, a) is used to improve the action selection strategy, so that the UAV tends to drive to the area with the maximum probability value of the target. The formula is as follows: Where E(s,a) is the harmonic function value of the drone performing action a in state s; ε∈(0,1) is the random probability value; a random Randomly select an action for the action set; When the agent selects an action in state s, it has an ε probability of selecting the action with the largest sum of the value function and the harmonic function in this state, and a 1-ε probability of randomly selecting an action. The formula is as follows: Among them, p k represents the position of the drone at time k; p p_max Indicates the location where the target has the largest probability value; η represents a positive coefficient, which is used to adjust the size of the harmonic function E(s,a).

7. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the drone collaborative search method based on the heuristic Q-learning algorithm as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the drone collaborative search method based on the heuristic Q-learning algorithm as described in any one of claims 1 to 5.