Optimal search method under SAR (synthetic aperture radar) grid scene of unmanned aerial vehicle

Abstract

The invention belongs to the field of unmanned aerial vehicle mobile search, and particularly relates to an optimal search method under an unmanned aerial vehicle SAR (synthetic aperture radar) grid scene, which comprises the following steps: constructing an SAR unmanned aerial vehicle search system model; constructing a search target function of maximizing the residual resources and the number of searched people according to the recognized people number constraint, the search region area constraint and the unmanned aerial vehicle resource constraint; solving the residual resource minimization and search number maximization search objective function by using a deep reinforcement learning method to obtain an optimal search path; the system optimizes the deflection angle and the moving distance of the unmanned aerial vehicle in the search model in real time according to the optimal search path scheme; and detecting the finally generated path area by using a target detection algorithm to identify trapped persons, and optimizing the search efficiency of the search model. According to the method, the optimal search path of the unmanned aerial vehicle under the condition of minimum resource consumption in the search scene can be reflected, meanwhile, human eye recognition is replaced by the recognition algorithm, manpower is liberated, and the search efficiency is improved.

Description

Technical Field

[0001] The invention belongs to the field of unmanned aerial vehicle mobile search, and in particular relates to an optimal search method in a unmanned aerial vehicle SAR grid scene. Background Art

[0002] In recent years, unmanned aerial vehicles (UAVs) have been widely used in various fields. At the same time, UAVs are maneuverable and flexible, can respond to operational needs quickly at the first time, have a fast reach speed, and are waterproof and can perform tasks in bad weather; real-time image transmission: aerial high-definition images are transmitted back to the front and rear command posts in real time, allowing commanders and fighters at all levels to promptly grasp the flood situation, dam conditions and rescue situation; information support: panoramic annotation, two-dimensional and three-dimensional modeling of key areas can be carried out, and the pre-disaster and post-disaster conditions can be compared to provide effective data support for command decision-making; and safe and reliable: the use of drones for reconnaissance, embankment patrol, rescue and other operations also provides protection for the safety of rescue personnel.

[0003] Drones are widely used due to their excellent durability, low cost, simple deployment and versatility; they are highly useful in a variety of applications such as smart agriculture, security, power line inspection, mapping, surveillance, SAR, express delivery, disaster area coverage, etc. For example, after a disaster, drones can be used to search for trapped people.

[0004] When a flood occurs, communication lines and infrastructure such as power plants, highways, and bridges are damaged or destroyed, and people are trapped by the flood and need rescue, otherwise it will cause unimaginable losses. Since disasters are unpredictable, quickly finding trapped people is crucial to facilitating disaster relief assistance. Drones are widely used due to their flexibility, maneuverability, and rapid deployment. Using drones for search has become the key to future search and rescue. Due to limited energy supply, drone emergency search needs to be adaptable and energy-efficient.

[0005] The use of drones has considerable advantages in search and image collection, but there are still various problems: for example, in terms of resource management, drones have small batteries and relatively short flight time, and there is a problem of insufficient resources to use drones to scan the entire disaster area. For the existing scanning area, the problem is to plan the path for drone scanning with limited resources so that the drone consumes the least energy to scan each task point. After collecting a large number of images, there is a problem of processing speed, evaluation and identification methods for objects of interest in search and rescue operations (including people, clothes or other details). If this work is mainly done manually, it will take a lot of time and is not necessarily reliable, reducing the possibility of survival of victims.

[0006] In order to solve these problems, researchers have proposed some path planning and image recognition solutions for post-disaster drone search and identification system planning. They mainly include: (1) For drone image detection, a lightweight YOLOv4 target detection algorithm for small targets is proposed (reference: LIU J, HU C, ZHOU J. et al. Object detection algorithm based on lightweight yolov4 for UAV [C]. IEEE International Conference on Intelligent Computing and Signal Processing, Xi'an, China, 2022: 425-429.): Based on the YOLO network model, the speed and accuracy of drone target detection are improved through network structure and image prediction box selection algorithm. (2) Multi-UAV coverage path planning algorithm (Reference: DONG D, LIUC. Collaborative coverage path planning of UAV cluster based on deep reinforcement learning [C]. IEEE 3rd International Conference on Frontiers Technology of Information and Computer, Greenville, SC, USA, 2021: 201-207.): Utilize all points of interest in the area, where each UAV starts from a random position and carries a camera during the mission; propose to use deep reinforcement learning of double Q learning network (DDQN, Double DQN) to learn the global optimal control strategy of the UAV team under certain power constraints, so as to effectively cooperate and explore a wider area. (3) An algorithm based on real-time target detection and search based on UAVs is proposed (Reference: Goncalves L, Damas B. Automatic detection of rescue targets in maritime search and rescue missions using UAVs [C]. IEEE International Conference on Unmanned Aircraft Systems, Dubrovnik, Croatia, 2022: 1638-1643.): The UAV carries an RGB camera and only forwards the candidate images and locations that may contain targets given by the detector. The YOLOv4-tiny detection network is used to detect maritime rescue targets, achieving good rescue detection capabilities.

[0007] However, some existing algorithms detect the images sent by drones and then manually intervene in the movement of drones, which lacks the search autonomy of drones and may not maximize the use of resources; some only study the path planning of drones, but lack a complete system framework for the entire drone search system. Therefore, how to optimize the drone path and detect drone images at the same time with limited drone resources has very important practical value and theoretical significance. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides an optimal search method in a UAV SAR grid scene, comprising:

[0009] S1: Build a SAR UAV search system model;

[0010] S2: Based on the constraints of the number of people to be identified, the area of ​​the search area, and the UAV resources, a search objective function is constructed to maximize the remaining resources and the number of people to be searched;

[0011] S3: Using deep reinforcement learning methods to solve the search objective functions of minimizing the remaining resources and maximizing the number of searchers, and obtain the best search path; the system optimizes the deflection angle and moving distance of the drone in the search model in real time according to the best search path solution;

[0012] S4: Use the target detection algorithm to detect and identify the trapped persons in the finally generated path area, and optimize the search efficiency of the search model.

[0013] Beneficial effects of the present invention:

[0014] The present invention considers that when scanning the disaster area, the drone takes pictures based on its own line of sight, proposes to divide the disaster area according to the line of sight, designs an optimal path based on DDPG in a grid scene and a search system combined with yolov8. First, a SAR system model is constructed based on the regional location and the drone; then, with the goal of maximizing the remaining resources of the drone and the number of searchers, a problem of solving the optimal search path for the drone is established; to solve this problem, a drone coverage path planning algorithm based on DDPG is proposed; finally, with the goal of the highest search accuracy, a target detection algorithm based on yolov8 is proposed. The simulation results show that the search framework constructed by the two algorithms can obtain greater drone search efficiency and system energy efficiency, thereby improving the performance of the drone search system, thereby verifying the effectiveness of the proposed system algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a model diagram of the drone search system in the present invention;

[0016] Figure 2 This is a schematic diagram of the DDPG algorithm framework in the present invention;

[0017] Figure 3 This is the target detection and recognition diagram using yolov8 in the present invention;

[0018] Figure 4 It is the reward convergence diagram of the algorithm proposed in this invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The present invention provides an optimal search method in a UAV SAR grid scene, comprising:

[0021] S1: Build a SAR UAV search system model;

[0022] like Figure 1 As shown in the figure, a drone search system in a SAR system is constructed. The system includes: a fully charged drone carrying a camera, a temporary base station (time state, TS) for processing information, and a target area to be searched; there are many trapped people in the target area, and the coordinates of the kth trapped person are u k =[x k ,y k ];

[0023] (1) Drone model:

[0024] Define the flight altitude of the drone as h and the flight speed as V fly ; α is the angle range of the scene captured by the drone’s onboard camera, and the drone’s viewing width is W k =2htan(α / 2), the drone battery capacity is E uav .

[0025] The coverage range r of the drone can be expressed as:

[0026] r≤W k (1)

[0027] (2) Channel model:

[0028] In a typical air-to-ground (A2G) channel model, the probability of a line-of-sight (LoS) link between the ground and the drone is defined as follows:

[0029]

[0030] Among them, a and b are constants related to the environment type, and h b represents the base station height, h represents the drone height, r us is the horizontal distance between the UAV and the temporary base station TS.

[0031] In addition, the probability of a non line-of-sight (NLoS) link is:

[0032] P NLoS =1-P LoS (3) The path loss models of UAV LoS and NLoS links are:

[0033]

[0034] Among them, f c is the carrier frequency, d us represents the distance between the UAV and the temporary base station, η LoS represents the additional loss in the LoS case, η NLoS represents the additional loss in the NLoS case.

[0035] Under the LoS and NLoS models, the average path loss of the A2G link is:

[0036] L us =L LoS ×P LoS +L NLoS ×P NLoS (6) The signal-to-noise ratio (SINR) received by the temporary base station from the drone is calculated as:

[0037]

[0038] where p us Indicates the transmitting power of the UAV, L us represents the average path loss of the A2G link, N0 represents the noise power, P Los and P NLoS They represent the LoS probability and NLoS probability between the UAV and the base station respectively.

[0039] S2: Identify the number of people constraints, search area constraints and drone resource constraints, and build a search model that maximizes the remaining resources and the number of people to search;

[0040] (1) Drone mobility model:

[0041] The position of the drone after moving is obtained through the distance and angle of movement in the action space. Assume that the moving distance action obtained from the action space is d n , the moving angle is θ n , the current position of the drone is The distance d between the drone and the temporary base station at this time us for:

[0042]

[0043] The offset distance in the drone coordinates is:

[0044]

[0045] The position update expression is: The time consumed by the drone during this movement is:

[0046]

[0047] in, They represent the x-axis coordinate and y-axis coordinate of the drone with the temporary base station as the coordinate origin; s ,y s They respectively represent the locations of temporary base stations.

[0048] (2) System energy consumption model:

[0049] Based on the flight speed v of the drone fly , the flight propulsion energy consumption of the UAV is P fly , the propulsion energy consumption can be approximately modeled as:

[0050]

[0051] P0 and P i is the blade power and induced power in the hovering state; v tip is the blade tip speed, v0 is the rotor induced speed when the UAV is hovering, d0 is the fuselage drag ratio; s is the rotor solidity; A is the rotor disc area, and ρ is the air density.

[0052] The calculation method for the blade power P0 in formula (11) is:

[0053]

[0054] Where Ω is the blade angular velocity, R0 is the rotor disc radius, and δ is the shape drag coefficient;

[0055] At the same time, the induced power in the hovering state is:

[0056]

[0057] W is the weight of the rescuer, W0 is the weight of the UAV, and κ is the induced power correction factor.

[0058] Based on the signal-to-noise ratio of the transmitted signal and formula (7), the transmission power p of the drone can be obtained: us :

[0059] p us =SINR us ×(p us -L us +N0)+L us (14)

[0060] Then the corresponding transmission energy consumption is calculated by the UAV transmission power, t n Indicates the duration of the nth action, that is, the task execution time.

[0061] The energy consumption optimization model is:

[0062]

[0063] S3: Use deep reinforcement learning methods to solve the source minimization and search number maximization model to obtain the optimal search path; the system optimizes the deflection angle and moving distance of the drone in the search model in real time according to the optimal search path solution.

[0064] S31: Construct a Markov decision process based on the objective function.

[0065] The problem is modeled as a Markov decision process. MDP refers to a random process with Markov properties, which consists of a 4-tuple {S, P, A, R}. The entire wireless communication system except the intelligent controller is regarded as the environment. The intelligent controller acts as an intelligent agent to interact with the environment to move the UAV distance and angle. The UAV position, the number of people identified, the area of ​​the coverage area, and the remaining energy of the UAV are regarded as states, the UAV movement distance and angle are regarded as actions, and the number of people searched and the remaining energy function of the UAV are regarded as rewards. Specifically, it includes:

[0066] State space: system state s at time slot t t ∈S includes the current location of the drone, the remaining energy, the current coverage area, and the number of people searched, so the system state s in time slot t t It is expressed as:

[0067]

[0068] in, Represents the current location of the drone, e t Represents the current remaining energy of the drone, m t Represents the number of trapped persons currently searched, cov t Represents the area currently covered.

[0069] Define the action space: The agent observes the system state s t Select the action, that is, the distance the drone moves each time and the angle the drone deflects. Action a t ∈A is expressed as:

[0070] {d t ,θ t},t∈N

[0071] C6:r≤d t ≤10r

[0072] C7:0≤θπ t ≤2π (17)

[0073] Among them, r is the UAV’s line of sight radius. The moving distance is greater than the UAV’s radius each time, that is, each movement goes out of the UAV’s line of sight range; θ represents the UAV’s deflection angle, d t represents the distance the drone moves at the current moment, θ t represents the deflection angle of the current UAV movement, N represents a natural number set, C6 and C7 represent the sixth and seventh constraints respectively;

[0074] Define the reward function: According to the objective function, the immediate reward is expressed as:

[0075]

[0076] Among them, ξ1,ξ2 are weight parameters Δm t =m t -m t-1 Represents the change rate of the number of people found by the drone, Δe t =e t -e t-1 Represents the rate of change of the remaining energy of the drone, m t Indicates the number of people currently searched by the drone, m t-1 Indicates the number of people searched last time, e t Indicates the current remaining energy of the drone, e t-1 represents the remaining energy of the drone in the previous state, Δp is the negative penalty constant for flying out of the boundary, Represents a real number.

[0077] S32: According to the DDPG network structure, construct the corresponding Actor network and Critic network structure.

[0078] DDPG's Actor network and Critic network structure: The Actor network structure is an input layer, three fully connected layers, an output layer, and uses the tanh activation function. Specifically, it includes:

[0079]

[0080] The Actor network uses the output layer of the tanh activation function to generate the corresponding action a t The Critic network connects the output of the fully connected layer with the state as input to the output of another fully connected layer with the action as input. The connected result is mapped to the Q value through the fully connected layer and the output layer, which is used to evaluate the quality of the action generated by the Actor network.

[0081] S33: According to Markov decision, a dynamic optimization algorithm based on DDPG is used to optimize the moving distance and angle of the drone each time, and finally the best path to the destination is obtained.

[0082] The overall framework of the proposed algorithm is as follows: Figure 2 As shown in Figure 2, the network training process specifically includes:

[0083] (1) Using parameter θ μ ,θ μ′ ,θ Q and θ Q′ Initialize the Actor network μ(s) respectively t |θ μ ), Actor target network μ′(s t |θ μ′ ), Critic network Q(s t ,a t |θ Q ) and Critic target network Q′(s t ,a t |θ Q′ ) and initialize the experience buffer;

[0084] (2) The Actor network is based on the current state s t Generate action a t , the agent performs action a t Then the next time slot state s can be observed t+1 and instant rewards t Update the status s at the same time t+1 , and the experience tuple (s t ,a t ,r t ,s t+1) is stored in the experience buffer with a capacity of M;

[0085] (3) The agent samples a batch of experience tuples from the experience buffer; specifically,

[0086] By assigning priorities to experiences according to their TD errors, the TD errors are expressed as:

[0087] δ=y t -Q(s t ,a t |θ Q )=r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ )-Q(s t ,a t |θ Q ) (19)

[0088] Among them, y t represents the target Q value, γ represents the discount rate, and Q() represents the state s under the current strategy t and action a t The Q value function, s t represents the system status in time slot t, a t represents the flight action of the drone, θ Q Represents the parameters of the Q value function, representing the weights and biases of the neural network, r t represents the immediate reward, μ′() represents the decision in a given state s t+1 The action to be taken next t+1 The policy function, Q′() represents the next state s t+1 Under this condition, the Q value function of the target Q value of the action is calculated by using the strategy μ′, s t+1 represents the system state at time slot t+1, θ μ′ represents the parameters of the target policy network, represents the weights and biases of the policy network, θ Q′ Represents the parameters of the target Q value network, representing the weights and biases of the target Q value network.

[0089] (4) Use the sampled experience tuples to train the neural network. Specifically include:

[0090] Use gradient descent to minimize the loss function in the Critic network to update the Critic network parameters θ Q , the loss function is expressed as:

[0091]

[0092] The parameters of the Critic network are updated using the gradient descent method. The update formula is expressed as:

[0093]

[0094] Among them, α Q represents the Critic network learning rate, δ represents the TD error, Represents the gradient of the Critic network parameters.

[0095] The policy gradient used to update the Actor network parameters in the Actor network is expressed as:

[0096]

[0097] Use the policy gradient method to update the parameters of the Actor network. By training the neural network, the flight distance and angle of the drone are optimized to obtain the optimal flight path. Represents the gradient of the Q value of the Critic network for the current state and action, Represents the gradient of the Actor network to the policy.

[0098] S4: Use the target detection algorithm to detect and identify the trapped persons in the finally generated path area, and optimize the search efficiency of the search model.

[0099] For the images transmitted by the drone back to the temporary base station, the yolov8 target detection algorithm is used to identify whether there are trapped people in the image. In the training phase, the pedestrian training set from the drone's perspective is used for training, the image features are extracted, and the size of the feature map is reduced through the convolution layer and the pooling layer, while retaining useful feature information, updating the weights, and then saving the training results.

[0100] In the algorithm reasoning process, the key point is that yolov8 adopts a positive and negative sample allocation strategy. In the early stage of training, the model may find it difficult to distinguish between positive and negative samples. At this time, the weight penalty value is large, and more attention will be paid to samples that are easily misclassified. As training progresses, the model components become more powerful and can better distinguish samples, so the weight penalty value will dynamically become lower.

[0101] By first conducting sample training and then using the algorithm to detect the images transmitted by the drone, the search efficiency is significantly improved. Figure 3 Some images of pedestrian target detection using drone.

[0102] The implementation process of the UAV path planning algorithm based on DDPG is shown below:

[0103]

[0104]

[0105] Figure 4 The reward convergence diagram of the proposed DDPG path planning scheme is given. Figure 4 It can be seen that the proposed DDPG solution can effectively ensure the minimum resource consumption and the maximum number of people covered.

[0106] The above simulation results only represent an embodiment of the model under ideal conditions and do not limit the present invention. All changes, equivalent substitutions or improvements that follow the spirit and principles of the present invention should be within the scope of protection of the present invention.

[0107] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An optimal search method in a UAV SAR grid scenario, characterized in that: include: S1: Build a SAR UAV search system model; S2: Based on the constraints of the number of people to be identified, the area of ​​the search area, and the UAV resources, a search objective function is constructed to maximize the remaining resources and the number of people to be searched; S3: Using deep reinforcement learning methods to solve the search objective functions of minimizing the remaining resources and maximizing the number of searchers, and obtain the best search path; the system optimizes the deflection angle and moving distance of the drone in the search model in real time according to the best search path solution; S4: Use the target detection algorithm to detect and identify the trapped persons in the finally generated path area, and optimize the search efficiency of the search model.

2. The optimal search method in a UAV SAR grid scene according to claim 1, characterized in that: The SAR UAV search system model includes: a temporary base station for processing information, a fully charged UAV, and a target area to be searched, where a large number of trapped persons are distributed.

3. The optimal search method in a UAV SAR grid scene according to claim 1, characterized in that: According to the constraints of the number of people to be identified, the area of ​​the search area, and the resources of drones, a search model that maximizes the remaining resources and the number of people to be searched is constructed, including: Subject to C1:0≤λ≤1, C2:0≤η≤1, C3:(0,0)≤pos i ≤(X,Y) C4:e fly ≤e uav C5:e search ≤and uav in, Represents the number of people searched so far after the drone moves to the pos position, all num Indicates the total number of people distributed in the area, pos0 indicates the initial position of the drone, pos i Represents the position reached by the drone after the i-th movement, pos goal represents the final destination (X, Y) that the drone will reach, and λ represents the weight of the number of people searching in the optimal search path model. It represents the energy consumed when flying to this position in time t. represents the energy consumed when searching for this position in time t, e uav represents the resources carried by the drone, η represents the weight of the drone's energy consumption in obtaining the best search path, C1, C2, C3, C4, and C5 represent the first, second, third, fourth, and fifth constraints, respectively, and e fly Indicates the energy consumed during flight, e search Represents the energy consumed during the search process.

4. The optimal search method in a UAV SAR grid scene according to claim 1, characterized in that: The deep reinforcement learning method is used to solve the search objective function of minimizing the remaining resources and maximizing the number of searchers, and the optimal search path is obtained, including: S31: Construct a Markov decision process based on the objective function; S32: According to the DDPG network structure, construct the corresponding Actor network and Critic network structure; S33: According to Markov decision, a dynamic optimization algorithm based on DDPG is used to optimize the moving distance and angle of the drone each time, and finally the best path to the destination is obtained.

5. The optimal search method in a UAV SAR grid scene according to claim 4, characterized in that: Construct a Markov decision process based on the objective function, including: Define the state space: the system state s at time slot t t ∈S includes the current location of the drone, the remaining energy, the current coverage area, and the number of people searched, so the system state s in time slot t t It is expressed as: in, Represents the current location of the drone, e t Represents the current remaining energy of the drone, m t Represents the number of trapped persons currently searched, cov t Represents the area currently covered, and S represents the system state s t The set to which it belongs is also called the state space; Define the action space: The agent observes the system state s t Select the action, that is, the distance the drone moves each time and the angle the drone deflects. Action a t ∈A is expressed as: {d} t ,i t },t∈N C6:r≤d t ≤10r C7:0≤θ t ≤2π Among them, r is the UAV’s line of sight radius. The moving distance is greater than the UAV’s radius each time, that is, each movement goes out of the UAV’s line of sight range; θ represents the UAV’s deflection angle, d t represents the distance the drone moves at the current moment, θ t represents the deflection angle of the current UAV movement, N represents a natural number set, C6 and C7 represent the sixth and seventh constraints respectively; Define the reward function: Get the immediate reward r according to the objective function t It is expressed as: r t =ξ1Δm t +ξ2Δe t +Δp Among them, ξ1, ξ2 are the first and second weight parameters Δm t =m t -m t-1 Represents the change rate of the number of people found by the drone, Δe t =e t -e t-1 represents the rate of change of the remaining energy of the drone, Δp represents the negative penalty constant for flying out of the boundary, and m t Indicates the number of people currently searched by the drone, m t-1 Indicates the number of people searched last time, e t Indicates the current remaining energy of the drone, e t-1 Indicates the remaining energy of the drone in the previous state. Represents a real value.

6. The optimal search method in a UAV SAR grid scene according to claim 4, characterized in that: According to the DDPG network structure, construct the corresponding Actor network and Critic network structure, including: The Actor network and Critic network structure are an input layer, three fully connected layers, and an output layer, and use the tanh activation function. The Actor network uses the output layer of the tanh activation function to generate the corresponding action a t ; The Critic network connects the output of the fully connected layer with the state as input to the output of another fully connected layer with the action as input. The connected result is mapped to the Q value through the fully connected layer and the output layer, which is used to evaluate the quality of the action generated by the Actor network.

7. The optimal search method in a UAV SAR grid scene according to claim 4, characterized in that: The DDPG-based dynamic optimization algorithm consists of an environment, an agent, an ER sampling method and a neural network.

8. The optimal search method in a UAV SAR grid scene according to claim 4, characterized in that: According to Markov decision making, a dynamic optimization algorithm based on DDPG is used to optimize the moving distance and angle of the drone each time. By continuously exploring different actions and states, the reward value based on energy changes and the number of people searched after each move is obtained, and the experience is stored in the buffer pool. Each random sampling is performed, the target network is updated, and the process is repeated until convergence is reached, and finally the best path to the destination is obtained, including: The ER sampling method assigns priorities to experiences based on their TD errors; The TD error δ is expressed as: δ=y t -Q(s t ,a t |θ Q )=r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ 0|θ Q′ )-Q(s t ,a t |θ Q ) Among them, y t represents the target Q value, γ represents the discount rate, and Q() represents the state s under the current strategy t and action a t The Q value function, s t represents the system status in time slot t, a t represents the flight action of the drone, θ Q Represents the parameters of the Q value function, representing the weights and biases of the neural network, r t represents the immediate reward, μ′() represents the decision in a given state s t+1 The action to be taken next t+1 The policy function, Q′() represents the next state s t+1 Under this condition, the Q value function of the target Q value of the action is calculated by using the strategy μ′, s t+1 represents the system state at time slot t+1, θ μ ′ represents the parameters of the target policy network, representing the weights and biases of the policy network, θ Q′ Represents the parameters of the target Q value network, representing the weights and biases of the target Q value network; The i-th empirical sampling probability p(i) is expressed as: Where J represents the number of experiences sampled from the experience buffer, α represents the hyperparameter used for the strength of priority playback, α≥0, α=0 is uniform sampling, represents the weight of the current sample, represents the weight of sample j; Use gradient descent to minimize the loss function L in the Critic network to update the Critic network parameters θ Q ; The loss function L is expressed as: Among them, N represents the number of samples used to calculate the loss function, δ 2 represents the square of TD error; The parameters of the Critic network are updated using the gradient descent method. The update formula is expressed as: Among them, α Q Represents the Critic network learning rate, Represents the gradient relative to the Critic network parameters; Use the policy gradient method to update the parameters of the Actor network. The update formula is expressed as: Among them, α μ represents the Actor network learning rate, δ represents the TD error, represents the gradient relative to the Actor network parameters, represents the learning rate of the Actor network, θ μ Represents the parameters of the policy network; The policy gradient used to update the Actor network parameters in the Actor network is expressed as: in, represents, s represents the current state in the environment, s i represents the i-th state, which is the state sampled from the buffer, μ(s i ) means in state s i The policy output action under μ ) means that in state s, through parameter θ μ The calculated policy function; Update the parameters of the Actor target network and the Critic target network. The update formula is expressed as: Among them, θ μ′ represents the Actor target network learning rate, θ Q′ Represents the Critic target network learning rate, τ∈[0,1] represents the update coefficient.

9. The optimal search method in a UAV SAR grid scene according to claim 1, characterized in that: The target detection algorithm adopts the yolov8 target detection algorithm.

Images