Unmanned aerial vehicle air combat target tracking method based on layered deep reinforcement learning framework
Through the method based on the hierarchical deep reinforcement learning framework, multi-target tracking and uncertainty problems in drone air combat are solved, effective target allocation and tracking in complex environments are achieved, and the robustness of the model is enhanced.
Patent Information
- Application Number
- CN202411572934.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-06
- Publication Date
- 2025-02-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In drone air combat, traditional rules and strategies are difficult to cope with complex and changing combat environments, especially in multi-target tracking and uncertainty issues, and it is difficult to achieve effective target allocation and tracking.
Using a method based on a layered deep reinforcement learning framework, a drone relative motion model and constraint model are established, reward functions are designed, and target tracking is assigned and implemented through upper and lower controllers.
The rational allocation and tracking of multiple goals in complex environments is achieved, which enhances the robustness of the model and can effectively respond to environmental changes and uncertainties.
Smart Images
Figure CN119440057A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of target tracking, and in particular relates to a method for tracking unmanned aerial vehicle air combat targets based on a hierarchical deep reinforcement learning framework. Background Art
[0002] With the rapid development of drone technology, drone air combat confrontation has become a research hotspot. Compared with traditional manned aircraft, drones usually have lower manufacturing and operating costs. At the same time, they do not need to carry personnel during combat, thus reducing the risk of pilots. However, drones will face dangerous, complex and ever-changing combat environments during air combat, including ever-changing enemy tactical actions. For target tracking in drone cluster air combat, traditional rules and strategies are difficult to cope with all possible situations in air combat. Therefore, it is necessary to use optimization algorithms, deep reinforcement learning and other methods to build a multi-target tracking system, so as to gradually accumulate its own advantages and complete the tracking of enemy drones.
[0003] In the article "Speed Decision and Task Allocation of UAV Clusters Based on Multi-Objectives", it is pointed out that in UAV clusters and multi-task and multi-objective UAV combat scenarios, it is necessary to make comprehensive decisions on the task allocation of UAVs based on the performance of the UAVs, the situation on the battlefield, etc. However, some traditional algorithms may find it difficult to balance the task allocation and tracking of multiple targets in a complex nonlinear environment, and in actual tasks, changes in the system and environment may cause the effectiveness of traditional control methods to decline. Traditional control methods can better complete the tracking of a single target, but in actual applications, the system needs to meet multiple different goals and constraints, and it is more difficult to handle the problem of multi-target tracking. To this end, the present invention provides a UAV air combat target tracking method based on a hierarchical deep reinforcement learning framework, which can effectively solve the multi-target allocation, tracking problems and uncertainty problems, and enhance the robustness of the model. Summary of the invention
[0004] The purpose of the present invention is to provide a UAV air combat target tracking method based on a hierarchical deep reinforcement learning framework, which can achieve the technical problem of quickly assigning targets to target UAVs and tracking them in complex environments.
[0005] To achieve the above object, the present invention provides a method for tracking unmanned aerial vehicle air combat targets based on a hierarchical deep reinforcement learning framework, comprising the following steps:
[0006] S1. Establish a relative motion model of UAVs to provide location information for subsequent tracking of different targets;
[0007] S2, design constraint model and reward function;
[0008] S3, using the upper controller to assign tracking targets;
[0009] S4. Based on the lower-level controller, control the own UAV to track the enemy UAV.
[0010] Furthermore, the step 1 comprises the following steps:
[0011] 1-1) Establish the UAV kinematic model,
[0012]
[0013] Among them, (x,y,z) is the position of the own drone, [x,y,z] T is its vector, V is the ground speed of the UAV, ξ and τ are the track inclination and track deviation, is the track roll angle, W x ,W y ,W z is the horizontal axial overload of the UAV, the horizontal component of the normal overload, and the vertical component of the normal overload. The angular velocity w x ,w y ,w z for g is the acceleration due to gravity;
[0014] 1-2) Use the coordinate transformation matrix to convert the UAV kinematic model into the UAV relative motion model.
[0015] Among them, the transformation matrix in step 1-2) is
[0016]
[0017] τ e =τ di -τ yi is the difference between the desired tracking position and the yaw angle of the own UAV, τ di is the yaw angle of the enemy UAV, τ yi is the yaw angle of the own UAV, di is the i-th target UAV in the target UAV swarm, and yi is the i-th UAV in the own UAV swarm;
[0018] Then we get the relative motion model of the UAV
[0019] Furthermore, the step 2 comprises the following steps:
[0020] 2-1) Set up the constraint model, including collision avoidance, communication constraints and constraints on the detection of enemy drones by the own drone; the collision avoidance and communication constraints are ||L i -L j ||∈[R min ,d max ], the constraint condition for the friendly UAV to detect the enemy UAV is (L∈Z)∩(||Li -L j ||≤R max ), where L i , L j are the distances between the friendly and enemy drones, R min is the relative minimum distance between UAVs at the same time, Z is the visible area of the UAV, and R max The maximum search range of your own drone;
[0021] 2-2) Set the reward function;
[0022] Among them, the reward function in step 2-2) includes the task cost cost i,1 , obstacle avoidance cost i,2 , time coordination cost i,3 , communication cost i,4 :
[0023] Among them, L q is the desired distance between drones to be maintained as much as possible, a 1 is the task cost parameter, d ij is the distance between the ith friendly UAV and the jth enemy UAV, d min =R min is a safe distance, and the f function converts the distance between drones to [0,1].
[0024] N 1 is the number of our own drones, N 2 is the number of enemy drones.
[0025] Furthermore, the step three comprises the following steps:
[0026] 3-1) Initialize your own drone, a, A, Further calculate the fitness of the drone and save the best fitness. Confirm assigned enemy drones, When , the friendly UAV will move away from the enemy UAV, otherwise it will move closer; is the convergence factor. As the number of iterations decreases from 2 to 0,
[0027] 3-2) Update the current position, a, and A of the own drone and calculate its fitness;
[0028] 3-3) Whether the maximum number of iterations has been reached, if not, continue iteration and repeat step S32, if reached, end iteration.
[0029] Furthermore, the step 4 comprises the following steps:
[0030] 4-1) Use UWB to obtain the location information of both drones and provide location data for tracking your own drone;
[0031] Among them, four UWB modules are used to form base stations M0, M1, M2, and M3, and four tags D1, D2, Y1, and Y2 are used to obtain the coordinate information of the own drone and the enemy drone;
[0032] 4-2) Use YOLO-V5 to detect the target drone, and process the pictures taken by the drone camera to obtain the target drone situation information. Perform feature extraction on the acquired target drone pictures, combine the acquired effective feature layer to predict the target detection results, and then obtain a complete prediction frame;
[0033] Among them, the picture taken by the camera of the own drone in step 4-2) is the latest frame. Specifically, a video frame skipping technology is designed to ensure that the drone of the own party always reads the latest frame when reading the video frame; two multi-threads thread1 and thread2 are created: thread1 is used to receive video frames frame0, frame1, …, frame∞ from the drone of the own party, and thread2 is used to create a queue with a capacity of 1 to store the video frames of thread1; when thread1 does not receive the latest frame, the queue of thread2 always stores the latest captured frame, ensuring that the video frame read by the host computer is always the latest frame.
[0034] 4-3) Based on the TD3 algorithm and the UAV's action and state space, combined with the reward function designed in step S22, a target tracking model of the UAV is established; the TD3 algorithm process specifically includes: constructing the actor network μ and the tar_actor network μ′, whose network weight parameters are θ μ ,θ μ′ ; Construct criticl1 network Q 1 、criticl2 network Q 2 ,tar_criticl1 network Q 1 ′、tar_criticl2 network Q 2 ′, and its network weight parameters are
[0035] in, m is a small amount of experience taken out; for the update of the target network, a soft update method is used to update the old target network parameters and the new corresponding network parameters. The weighted average is then assigned to the target network: θ μ′ =κθ μ +(1-κ)θ μ′ , κ is the introduced learning rate;
[0036] 4-4) Analyze and process the data during the tracking process based on the trained UAV target tracking model.
[0037] Compared with the prior art, the present invention has the following beneficial effects:
[0038] 1. The present invention discloses a method for tracking UAV air combat targets based on a hierarchical deep reinforcement learning framework. First, a relative motion model of the UAV is established to provide position information for subsequent tracking of different targets; then a constraint model and a reward function are designed.
[0039] 2. Based on the hierarchical deep reinforcement learning framework, the present invention designs an upper-level controller for allocating tracking targets, and a lower-level controller for controlling the own UAV to track the enemy UAV.
[0040] 3. The method proposed in the present invention can effectively solve the influence of factors such as complex environment, real-time images, and the number of tracked targets on UAVs in air combat confrontations, enable the UAVs of the UAVs to more reasonably allocate targets and track enemy UAVs, and provide tracking uncertainty to enhance the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 is a system flow chart of the method of the present invention;
[0042] Figure 2 This is a flowchart of the mission of the drone of the party in the present invention;
[0043] Figure 3 Schematic diagram of the coordinate system of the relative motion model of the UAV in the present invention;
[0044] Figure 4 It is the UWB position and radius positioning diagram in the present invention;
[0045] Figure 5 This is a diagram of the video frame skipping technology in the present invention. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0047] Example 1
[0048] The purpose of the present invention is to provide a method for tracking unmanned aerial vehicle air combat targets based on a hierarchical deep reinforcement learning framework, which includes the following steps:
[0049] Step 1) Establish a relative motion model of the UAV to provide location information for subsequent tracking of different targets;
[0050] Step 2) Design constraint model and reward function;
[0051] Step 3) using the upper controller to assign tracking targets;
[0052] Step 4) Based on the lower-level controller, control the own UAV to track the enemy UAV.
[0053] Step 1) Establishing the relative motion model of the drone to provide location information for subsequent tracking of different targets. The specific steps are as follows:
[0054] 1-1) Establishing the UAV kinematic model
[0055]
[0056] Among them, (x,y,z) is the position of the own drone, [x,y,z] T is its vector, V is the ground speed of the UAV, ξ and τ are the track inclination and track deviation, is the track roll angle, W x ,W y ,W z is the horizontal axial overload of the UAV, the horizontal component of the normal overload, and the vertical component of the normal overload. The angular velocity w x ,w y ,w z for g is the acceleration due to gravity;
[0057] 1-2) Use the coordinate transformation matrix to transform the UAV kinematic model into the relative motion model of the UAV. Specifically:
[0058] The transformation matrix is
[0059]
[0060] τ e =τ di -τ yi is the difference between the desired tracking position and the yaw angle of the own UAV, τ di is the yaw angle of the enemy UAV, τ yi is the yaw angle of the own UAV, di is the i-th target UAV in the target UAV swarm, and yi is the i-th UAV in the own UAV swarm;
[0061] Then we get the relative motion model of the UAV The coordinate system of the UAV relative motion model is shown as follows Figure 3 shown.
[0062] Step 2) The specific steps for designing the constraint model and reward function are as follows:
[0063] 2-1) Set up the constraint model, including collision avoidance, communication constraints and constraints on the detection of enemy drones by the own drone; the collision avoidance and communication constraints are ||L i -L j ||∈[R min ,d max ], the constraint condition for the friendly UAV to detect the enemy UAV is (L∈Z)∩(||L i -L j ||≤R max ), where L i , L j are the distances between the friendly and enemy drones, R min is the relative minimum distance between UAVs at the same time, Z is the visible area of the UAV, and R max The maximum search range of your own drone;
[0064] 2-2) Set the reward function, specifically:
[0065] The reward function includes the task cost i,1 , obstacle avoidance cost i,2 , time coordination cost i,3 , communication cost
[0066]
[0067] Among them, L q is the desired distance between drones to be maintained as much as possible, a 1 is the task cost parameter, d ij is the distance between the ith friendly UAV and the jth enemy UAV, d min =R min is a safe distance, and the f function converts the distance between drones to [0,1]. 1 is the number of our own drones, N 2 is the number of enemy drones;
[0068] Specifically,
[0069] Step 3) The specific steps of using the upper-level controller to assign tracking targets are as follows:
[0070] 3-1) First initialize your own drone, a, A, Then further calculate the fitness of the drone and save the best fitness. Confirm assigned enemy drones, When , the friendly UAV will move away from the enemy UAV, otherwise it will move closer; is the convergence factor. As the number of iterations decreases from 2 to 0,
[0071] 3-2) Further update the current position, a, and A of the own drone and calculate its fitness;
[0072] 3-3) Determine whether the maximum number of iterations has been reached. If not, continue iterating and repeat step S32. If reached, end the iteration.
[0073] Step 4) The specific steps of controlling the own UAV to track the enemy UAV based on the lower-level controller are as follows:
[0074] 4-1) Use UWB to obtain the location information of both drones and provide location data for tracking your own drone;
[0075] 4-1-1) Specifically, four UWB modules are used to form base stations M0, M1, M2, and M3, and four tags D1, D2, Y1, and Y2 are used to obtain the coordinate information of the UAV and the enemy UAV. Figure 4 Place as shown;
[0076] 4-1-2) Both the friendly UAV and the enemy UAV are flying within the base station range, and the enemy UAV is flying according to the set trajectory;
[0077] 4-2) Use YOLO-V5 to detect the target drone, and process the pictures taken by the drone camera to obtain the target drone situation information. Perform feature extraction on the acquired target drone pictures, combine the acquired effective feature layer to predict the target detection results, and then obtain a complete prediction frame;
[0078] Among them, the picture taken by our drone camera is the latest frame, specifically:
[0079] Design a video frame skipping technology, such as Figure 5, ensuring that the drone always reads the latest frame when reading the video frame; create two multi-threads thread1 and thread2: thread1 is used to receive video frames frame0, frame1, ..., frame∞ from the drone, and thread2 is used to create a queue with a capacity of 1 to store thread1's video frames; when thread1 does not receive the latest frame, thread2's queue always stores the latest captured frame, ensuring that the host computer always reads the latest frame.
[0080] 4-3) Based on the TD3 algorithm and the UAV's action and state space, combined with the reward function designed in step S22, a target tracking model of the UAV is established; the TD3 algorithm process specifically includes: constructing the actor network μ and the tar_actor network μ′, whose network weight parameters are θ μ ,θ μ′ ; Construct criticl1 network Q 1 、criticl2 network Q 2 ,tar_criticl1 network Q 1 ′、tar_criticl2 network Q 2 ′, and its network weight parameters are
[0081] in, m is a small amount of experience taken out; for the update of the target network, a soft update method is used to update the old target network parameters and the new corresponding network parameters. The weighted average is then assigned to the target network: θ μ′ =κθ μ +(1-κ)θ μ′ , κ is the introduced learning rate;
[0082] 4-4) Analyze and process the data during the tracking process based on the trained UAV target tracking model.
[0083] Example 2
[0084] In this example, the UAV performs well in a complex air combat environment in a simulated air combat experiment. The upper-level controller is designed based on the Gray Wolf optimization algorithm, which enables the UAV to efficiently and reasonably allocate targets. It can adjust the allocated targets according to the real-time combat situation and adapt to the highly complex air combat environment. The lower-level controller is based on the TD3 algorithm, combined with an optimized reward function, so that the UAV can fly more stably and make better tracking strategies in sudden situations. In addition, based on the video frame skipping technology, the UAV can obtain the latest situation in real time to ensure that the upper and lower controllers can adjust the strategy in time. The traditional direct use of cv2 to obtain video frames and detect the tracking target, the video frames processed by the upper computer may not be the video frames at the current moment, and the delay is very large, which does not meet the real-time requirements in high-intensity air combat. At the same time, for the control of UAVs, traditional PID control is linear control, which cannot adapt to complex dynamic environments. The dynamic characteristics of UAVs in air combat are even more complex, and parameter adjustment is also very difficult. Fuzzy logic control relies on pre-set rules and membership functions, and lacks adaptive capabilities. When the conditions in the air combat environment change greatly, these rules may not be adjusted in time, resulting in poor control effects, and then resulting in low efficiency in tracking mission execution.
[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for tracking UAV air combat targets based on a hierarchical deep reinforcement learning framework, characterized in that: The following steps are involved: S1. Establish a relative motion model of UAVs to provide location information for subsequent tracking of different targets; S2, design constraint model and reward function; S3, using the upper controller to assign tracking targets; S4. Based on the lower-level controller, control the own UAV to track the enemy UAV.
2. The method for tracking unmanned aerial vehicle air combat targets based on a hierarchical deep reinforcement learning framework according to claim 1 is characterized in that: In the above step S1, establishing a relative motion model of the drone to provide position information for subsequent tracking of different targets includes the following steps: S11: Building a UAV kinematic model Among them, (x,y,z) is the position of the own drone, [x,y,z] T is its vector, V is the ground speed of the UAV, ξ and τ are the track inclination and track deviation, is the track roll angle, W x ,W y ,W z is the horizontal axial overload of the UAV, the horizontal component of the normal overload, and the vertical component of the normal overload. The angular velocity w x ,w y ,w z for g is the acceleration due to gravity; S12: using a coordinate transformation matrix to transform the UAV kinematic model into a UAV relative motion model; The transformation matrix is Among them, τ e =τ di -τ yi is the difference between the desired tracking position and the yaw angle of the own UAV, τ di is the yaw angle of the enemy UAV, τ yi is the yaw angle of the own drone, di is the i-th target drone in the target drone group, yi is the i-th drone in the own drone group, and then the relative motion model of the drones is obtained 3. The method for tracking unmanned aerial vehicle air combat targets based on a hierarchical deep reinforcement learning framework according to claim 1, characterized in that: Designing the constraint model and reward function in the above step S2 includes two steps: S21: Set the constraint model, including collision avoidance, communication constraints and constraints on the detection of enemy drones by the own drone; the collision avoidance and communication constraints are ||L i -L j ||∈[R min ,d max ], the constraint condition for the friendly UAV to detect the enemy UAV is (L∈Z)∩(||L i -L j ||≤R max ), where L i , L j are the distances between the friendly and enemy drones, R min is the relative minimum distance between UAVs at the same time, Z is the visible area of the UAV, and R max The maximum search range of your own drone; S22: Setting the reward function For the above step S22, the reward function set includes the task cost cost i,1 , obstacle avoidance cost i,2 , time coordination cost i,3 , communication cost i,4 : Among them, L q is the desired distance between drones to be maintained as much as possible, a1 is the mission cost parameter, d ij is the distance between the ith friendly UAV and the jth enemy UAV, d min =R min is a safe distance, and the f function converts the distance between drones to [0,1]. N1 is the number of friendly drones, and N2 is the number of enemy drones.
4. The method for tracking unmanned aerial vehicle air combat targets based on a hierarchical deep reinforcement learning framework according to claim 1, characterized in that: In the above step S3, using the upper controller to allocate the tracking target includes the following steps: S31: Initialize your own drone, a, A, Further calculate the fitness of the drone and save the best fitness. Confirm assigned enemy drones, When , the friendly UAV will move away from the enemy UAV, otherwise it will move closer; is the convergence factor. As the number of iterations decreases from 2 to 0, S32: Update the current position, a, and A of the own UAV and calculate its fitness; S33: Check whether the maximum number of iterations has been reached. If not, continue iterating and repeat step S32. If reached, end the iteration.
5. The method for tracking unmanned aerial vehicle air combat targets based on a hierarchical deep reinforcement learning framework according to claim 1, characterized in that: In the above step S4, controlling the own UAV to track the enemy UAV based on the lower-layer controller includes the following steps: S41: Use UWB to obtain the location information of both drones and provide location data for tracking your own drone; S42: Use YOLO-V5 to detect the target drone, process the pictures taken by the drone camera to obtain the target drone situation information, extract features from the acquired target drone pictures, and predict the target detection results based on the acquired effective feature layers, thereby obtaining a complete prediction frame; S43: Based on the TD3 algorithm and the UAV's action and state space, combined with the reward function designed in step S22, a target tracking model of the UAV is established; the TD3 algorithm process specifically includes: constructing the actor network μ and the tar_actor network μ′, whose network weight parameters are θ μ ,θ μ′ ; Construct criticl1 network Q1, criticl2 network Q2, tar_criticl1 network Q1′, tar_criticl2 network Q2′, and their network weight parameters are in, m is a small amount of experience taken out; for the update of the target network, a soft update method is used to update the old target network parameters and the new corresponding network parameters. The weighted average is then assigned to the target network: θ μ′ =κθ μ +(1-δ)θ μ′ , κ is the introduced learning rate; S44: Analyze and process the data during the tracking process based on the trained UAV target tracking model.
6. The method for tracking unmanned aerial vehicle air combat targets based on a hierarchical deep reinforcement learning framework according to claim 1, characterized in that: For the above step S42, the picture taken by the camera of the own drone is the latest frame. A video frame skipping technology is designed to ensure that the drone reads the latest frame when reading the video frame. Two multi-threads thread1 and thread2 are created: thread1 is used to receive video frames frame0, frame1, ..., frame∞ from the drone, and thread2 is used to create a queue with a capacity of 1 to store the video frames of thread1. When thread1 does not receive the latest frame, the queue of thread2 always stores the latest captured frame, ensuring that the video frame read by the host computer is always the latest frame.