A reinforcement learning-based method for generating edge test scenarios for dynamic target tracking in UAVs

By constructing a Markov decision process and reinforcement learning algorithm for UAV target tracking tasks, edge test scenarios between UAVs and dynamic targets are generated, solving the problems of diversity and efficiency in UAV target tracking testing, and achieving efficient and low-cost test evaluation.

CN117540617BActive Publication Date: 2026-11-17BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311167353.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-11
Publication Date
2026-11-17
Estimated Expiration
2043-09-11

AI Technical Summary

Technical Problem

Existing drone target tracking testing methods cannot comprehensively evaluate the performance of drones under various environmental factors. Real-world testing is costly, and simulation testing is inefficient, failing to meet the testing needs of the rapidly developing drone industry.

Method used

By constructing a Markov decision process for UAV target tracking tasks and combining it with reinforcement learning algorithms, edge test scenarios between UAVs and dynamic targets are generated. The distribution and correlation of key influencing factors are analyzed, and a joint distribution is constructed using the copula function to simulate the impact of environmental factors on UAV tracking tasks, thereby training a dynamic target behavior model.

Benefits of technology

It enables efficient and low-cost testing of UAV dynamic target tracking, improves the diversity of test scenarios and testing efficiency, and provides an assessment of the reliability of UAV target tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117540617B_ABST
    Figure CN117540617B_ABST
Patent Text Reader

Abstract

This invention addresses unmanned aerial vehicles (UAVs) and proposes a reinforcement learning-based method for generating edge test scenarios for UAV dynamic target tracking. It comprises five steps: (1) constructing the UAV target tracking task process and defining the scene tracking success rate; (2) analyzing key influencing factors in the scene and combining them with real data to obtain the distribution types and correlations of each factor, constructing a joint distribution for correlated factors; (3) constructing a factor influence model to simulate the impact of key factors on the UAV target tracking task process; (4) establishing a Markov decision process for the UAV and dynamic target; and (5) constructing a target behavior model based on reinforcement learning, combining it with key factors to form a UAV target tracking test scenario.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention addresses the challenges of unmanned aerial vehicles (UAVs) and proposes a reinforcement learning-based method for generating edge test scenarios for dynamic target recognition and tracking. This method focuses on UAV target tracking, a typical task, analyzing key influencing factors affecting UAV tracking performance within the task scenario, determining the distribution types of each factor, and constructing a joint distribution for correlated factors. The values ​​of each factor in the scenario are obtained through sampling. Simultaneously, a Markov decision process between the UAV and the dynamic target is constructed, and reinforcement learning is used to train the dynamic target to better identify edge test scenarios for UAV target tracking, providing a reference for evaluating UAV tracking success rates. This invention belongs to the field of reliability engineering. Background Technology

[0002] As the autonomy and intelligence levels of drones continue to improve, they are gradually occupying an important position in the market and are widely used in important tasks such as military, police, urban management, and video shooting. At the same time, the reliability and safety of drones during missions are also receiving increasing attention, necessitating a realistic and efficient method for generating drone test scenarios to evaluate drone target tracking capabilities and provide a basis for drone design optimization.

[0003] During target tracking missions, unmanned aerial vehicles (UAVs) are often affected by various environmental factors. However, real-world testing methods cannot fully test the UAV's target tracking capabilities under various influences, and the test scenarios are relatively limited. Simulation testing requires testing the UAV in numerous scenarios, which contain many redundant scenarios and exhibit relatively simple target behaviors, resulting in high testing costs and low efficiency, failing to meet the testing needs of rapidly evolving UAVs. Therefore, this invention proposes a reinforcement learning-based method for generating edge test scenarios for UAV dynamic target tracking. By constructing a task profile of the UAV's target tracking process, the key influencing factors at each task stage are comprehensively analyzed. Combined with real-world scenario data, distribution models of each factor are obtained, and correlated influencing factors are identified, their joint distribution is constructed, and parameter combinations of each influencing factor in the scenario are sampled. Appropriate factor influence models are constructed for the analyzed key factors. A Markov decision process between the UAV and the dynamic target is constructed, and reinforcement learning is used to train the dynamic target's behavior model. Finally, the UAV is evaluated and tested, providing input for reliability assessment testing of UAV tracking of dynamic targets. Summary of the Invention

[0004] This invention provides a method for generating edge test scenarios for dynamic target tracking in unmanned aerial vehicles (UAVs) based on reinforcement learning. The main steps of this invention are as follows:

[0005] Step 1: Construct a task scenario for drone target tracking and define the drone tracking success rate;

[0006] The UAV dynamic target tracking task is divided into five stages: UAV takeoff, target identification, target tracking, return to base, and landing. The UAV tracking success rate is defined as the proportion of the total task time during which the UAV accurately acquires the dynamic target's position information. The calculation formula is as follows:

[0007]

[0008] Where t = 0 represents the time when the dynamic target first appears in the UAV's field of view during the target tracking task; S(t) frames All(t) represents the number of frames in which the UAV successfully tracked the target during the target tracking task. frames This indicates the total number of frames in the target tracking task.

[0009] Step 2: Analyze the key influencing factors in the task scenario, and combine real data to obtain the distribution type and correlation of each factor, and construct the joint distribution of the factors with correlation.

[0010] Based on the UAV target tracking mission scenario, this paper analyzes the key factors affecting mission success during UAV target tracking and obtains the distribution type of each factor using real data. Then, the Pearson correlation coefficient is used to determine the correlation between the factors, and the copula function is used to construct their joint distribution.

[0011] Step 1: Analyze the key influencing factors in each task scenario and collect real-world scenario data for each key factor;

[0012] Table 1 Key Influencing Factors

[0013] Task Phase Key influencing factors take off Weather, wind speed Identify target Weather, target characteristics, target speed, interfering targets, road conditions Tracking target Weather, wind speed, target characteristics, target speed, interfering targets, road conditions Return Weather, wind speed, drone location landing Weather, wind speed, drone location

[0014] Based on the analysis of key factors at each stage of the task scenario, the values ​​of each key factor in region A during time period T are collected to obtain a real-world scenario dataset.

[0015] Step 2: Combine the data to obtain a distribution model of the influencing factors;

[0016] The data of each key factor collected in step 1 are normalized, and the distribution of each key factor is fitted to obtain the distribution model of each factor.

[0017] Step 3: Factor Correlation Analysis

[0018] Based on the normalized key factor data, the correlation between factors is analyzed pairwise using the Pearson correlation coefficient, the formula of which is:

[0019]

[0020] Where r represents the correlation between the two factors, X i Y i This represents the value of the i-th sample among factors X and Y; This represents the average value of factors X and Y. Further, a correlation matrix is ​​constructed based on the correlation calculation results. If the correlation between two factors exceeds 0.6, proceed to step 4 until the joint distribution of all related factors is determined, then proceed to step 3; otherwise, proceed directly to step 3.

[0021] Step 4: Construct a joint distribution for correlated factors using the copula function.

[0022] Based on the correlation matrix obtained in step 3, a copula function is used to construct a joint distribution model of the correlated factors for all correlated factors. Assume there are n key influencing factors in the UAV ground target dynamic tracking scenario, and the distribution of each factor is denoted as Fi. a (x a ), F b (x b )...F n (x n If ), then its joint distribution is denoted as:

[0023] L(x a ,x b ,…x n )=C(F a (x a ),F b (x b ),…,F n (x n ))

[0024] Where, L(x) a ,x b ,…x n This represents the joint distribution of all relevant factors. The optimal Copula function needs to be selected as the joint distribution function using the AIC information criterion. The formula for calculating the AIC value is:

[0025] f AIC = -2ln(LF) + 2k

[0026] Among them, f AIC Let AIC represent the value, and LF represent the value of the likelihood function. The values ​​of various parameters of the scene are obtained by sampling from the distributions of each factor and their joint distribution using Monte Carlo sampling.

[0027] Step 3: Construct a factor impact model to simulate the impact of key factors on the UAV target tracking task process;

[0028] Step 1: Building a model of weather influencing factors;

[0029] 1) Fog Impact Modeling: The impact of heavy fog on drones is obtained through a fog formation model. The model is as follows:

[0030] I(x)=J(x)+A(1-t(x))

[0031] Where I(x) represents the image after adding the effect of heavy fog; J(x) represents the image captured by the drone in clear weather; t(x) represents the transmittance, which represents the effect of light intensity on the original image; and A represents the light intensity.

[0032] 2) Rain and Snow Impact Modeling: The impact of rain and snow on images is analyzed using a general rain and snow model. The model is as follows:

[0033]

[0034] Where O represents the image after adding rain and snow effects; B represents the image captured under the working conditions; S t Raindrops that are in the same direction and shape; α represents the cumulative amount of raindrops in layer s; α represents transmittance, which represents the effect of light intensity on the image during rainy weather; A represents light intensity.

[0035] 3) Wind speed impact modeling: The wind speed impact is represented by the force exerted by the wind on the UAV per unit time, and the magnitude of this force is described by momentum.

[0036] F wind =v wind ·m gas

[0037] Among them, F wind v represents the force exerted by the wind on the drone per unit time. wind Wind speed, m gas The mass of air acting on the drone is expressed by the following formula:

[0038] m gas =ρ gas ·S·v wind

[0039] Where, ρ gas S represents the air density, and S represents the contact area of ​​the drone's windward side.

[0040] Step 2: Occlusion factors affect model construction;

[0041] The obstruction range of a building is calculated using the drone's location and the location and height information of ground buildings. Assuming the road direction is the y-direction, the obstruction range is:

[0042]

[0043] Where w is the area of ​​obstruction by the building on the ground, (x uav y aav h uav Let (x) be the position of the UAV at time t, and (x) be the position of the UAV at time t obj y obj h obj The location and height of the building are shown in the figure. If the target is within the obstruction range, the drone will lose the target.

[0044] Step 3: Construction of the target and interference target impact model;

[0045] Factors influencing the target and interfering targets include target type, target size, and color, with target type and size being affected by target type. The types of targets and interfering targets (t) are determined through sampling. sp Type t t Size t si Color t c .

[0046] Step 4: Establish a Markov decision process for drones and dynamic targets;

[0047] By constructing a Markov decision process between the UAV and the ground dynamic target, the state space, action space, state transition probability, and reward function of the UAV and the ground dynamic target are defined, and an adversarial environment between the two is generated.

[0048] Step 1: Define the state space of the UAV and the dynamic target;

[0049] By placing the UAV and the dynamic target in a three-dimensional coordinate system, and using a Markov decision process, the state space for UAV and ground dynamic target tracking is established:

[0050] S:(x target x uav y target y uav v target v uav φ target φ uav h uav )

[0051] Where, x target This represents the target's position information in the x-direction of a three-dimensional coordinate system; x uav Represents the drone's position information in the x-direction; ytarget This represents the target's position information on the y-axis; y uav This represents the drone's position information on the y-axis; v target V represents the speed of the target; uav Represents the speed of the drone; φ target The deflection angle representing the target; φ uav The deflection angle of the drone; h uav The altitude information of the drone.

[0052] Step 2: Construct the state and action space of the UAV and dynamic targets;

[0053] In a scenario where a drone tracks a dynamic target, the action space for both the drone and the dynamic target is:

[0054] A: (a uav a target )

[0055] Among them, a uav It is a three-dimensional vector representing the acceleration of the UAV; a target It is a two-dimensional vector representing the acceleration of a dynamic target on the ground.

[0056] Step 3: Define the state transition probabilities between the UAV and the dynamic target's state and actions;

[0057] In the scenario of UAV tracking a dynamic target, the state transition probability function is:

[0058] S t+1 =f(S) t A t )

[0059] Among them, S t A represents the state of the UAV and the dynamic ground target at time t; t This indicates the actions taken by the drone and the dynamic ground target at that moment; S t+1 This indicates the state of the drone and the dynamic ground target at the next moment.

[0060] Step 4: Define the reward function for the Markov decision process of the UAV and the dynamic target;

[0061] The reward for a dynamic target consists of two parts: a positional reward and a tracking penalty. The reward function is determined by the relative position of the UAV and the dynamic target, and is expressed as:

[0062]

[0063] Where, r pos Represents positional reward; ε is a minimal number; h uav θ is the drone's flight altitude; θ is the drone's detection angle; ct The factor used to measure whether a ground target is being tracked has the following values:

[0064]

[0065] c t =1 represents a dynamic target escaping the drone's tracking, while c t =0 indicates that the target is being tracked by a drone.

[0066] Step 5: Construct a target behavior model based on reinforcement learning, and combine it with key factors to form a test scenario for UAV target tracking;

[0067] The constructed Markov decision process is solved using the TD3 reinforcement learning algorithm, which is used to solve problems in high-dimensional continuous action spaces.

[0068] Step 1: Initialize the current actor-critic network and the target actor-critic network in the TD3 network, and randomize their network parameters;

[0069] Step 2: In the current actor network and the target actor network, select the drone and the dynamic target in state S respectively. t Given the given conditions, actions A and A′ should be taken. Calculate the expected reward of each action, using the following function:

[0070]

[0071] Where Q(s,a) represents the dynamic target and the UAV in state S. t An estimate of the return or expected profit that can be obtained by taking action A. Represents the expected value r post γ represents the reward value of the dynamic objective at time t; γ represents the discount factor, used to measure the importance of the current reward and the future reward.

[0072] Step 3: Place the calculated expected reward and corresponding action into their respective experience pools. Using the current critic network and the target critic network, select the action A with the highest expected reward from the experience pools and selected actions. max and A′ max ;

[0073] Step 4: Compare and select action A max and A′ max Given the reward value, select the action corresponding to the smallest reward and update the current actor-critic network parameters. After updating the current actor-critic network parameters g (g is a positive integer) times, pass its parameters to the target actor-critic network.

[0074] For each state S t Execute steps 2-4 of step 5 repeatedly, r times (where r is a positive integer), so that in each state, the TD3 algorithm can select the optimal action A to maximize the expected reward. The model obtained after training r times (where r is a positive integer) is the target behavior model. Using this model, the optimal target behavior is selected, and combined with the key factor values ​​obtained in the above steps, an edge test scenario for UAV target tracking is generated. Attached Figure Description

[0075] Figure 1 A method for generating edge test scenarios for dynamic target tracking in UAVs based on reinforcement learning

[0076] Figure 2 UAV's task of identifying dynamic ground targets

[0077] Figure 3 UAV-based identification and tracking of dynamic ground targets: road plan diagram

[0078] Figure 4 Schematic diagram of building occlusion in drone target tracking scenario

[0079] Figure 5 Markov Decision Process for Unmanned Aerial Vehicles and Dynamic Ground Targets Detailed Implementation

[0080] Step 1: Construct a task scenario for drone target tracking and define the drone tracking success rate;

[0081] The UAV dynamic target tracking task is divided into five stages: UAV takeoff, target identification, target tracking, return to base, and landing. The UAV tracking success rate is defined as the proportion of the total task time during which the UAV accurately acquires the dynamic target's position information. The calculation formula is as follows:

[0082]

[0083] Where t = 0 represents the time when the dynamic target first appears in the UAV's field of view during the target tracking task; S(t) frames All(t) represents the number of frames in which the UAV successfully tracked the target during the target tracking task. frames This indicates the total number of frames in the target tracking task.

[0084] [Example] The process of constructing a drone target tracking task is as follows: Figure 2As shown. In this task, assuming a total of 3600 frames are tracked, the tracking success rate can be calculated by determining the total number of frames in which the UAV successfully and accurately acquires the target's location information. For example, if the UAV is able to identify and acquire the target's location information in 3389 out of 3600 frames, then its tracking success rate is:

[0085]

[0086] Step 2: Analyze the key influencing factors in the task scenario, and combine real data to obtain the distribution type and correlation of each factor, and construct the joint distribution of the factors with correlation.

[0087] Based on the UAV target tracking mission scenario, this paper analyzes the key factors affecting mission success during UAV target tracking and obtains the distribution type of each factor using real data. Then, the Pearson correlation coefficient is used to determine the correlation between the factors, and the copula function is used to construct their joint distribution.

[0088] Step 1: Analyze the key influencing factors in each task scenario and collect real-world scenario data for each key factor;

[0089] Table 2 Key Influencing Factors

[0090]

[0091]

[0092] Based on the analysis of key factors at each stage of the task scenario, the values ​​of each key factor in region A during time period T are collected to obtain a real-world scenario dataset.

[0093] Step 2: Combine the data to obtain a distribution model of the influencing factors;

[0094] The data of each key factor collected in step 1 are normalized, and the distribution of each key factor is fitted to obtain the distribution model of each factor.

[0095] Step 3: Factor Correlation Analysis

[0096] Based on the normalized key factor data, the correlation between factors is analyzed pairwise using the Pearson correlation coefficient, the formula of which is:

[0097]

[0098] Where r represents the correlation between the two factors, X i Y i This represents the value of the i-th sample among factors X and Y; This represents the average value of factors X and Y. Further, a correlation matrix is ​​constructed based on the correlation calculation results. If the correlation between two factors exceeds 0.6, proceed to step 4 until the joint distribution of all related factors is determined, then proceed to step 3; otherwise, proceed directly to step 3.

[0099] Step 4: Construct a joint distribution for correlated factors using the copula function.

[0100] Based on the correlation matrix obtained in step 3, a copula function is used to construct a joint distribution model of the correlated factors for all correlated factors. Assume there are n key influencing factors in the UAV ground target dynamic tracking scenario, and the distribution of each factor is denoted as Fi. a (x a ), F b (x b )...F n (x n If ), then its joint distribution is denoted as:

[0101] L(x a ,x b ,…x n )=C(F a (x a ),F b (x b ),…,F n (x n )

[0102] Where, L(x) a ,x b ,…x n This represents the joint distribution of all relevant factors. The optimal Copula function needs to be selected as the joint distribution function using the AIC information criterion. The formula for calculating the AIC value is:

[0103] f AIC = -2ln(LF) + 2k

[0104] Among them, f AIC Let AIC represent the value, and LF represent the value of the likelihood function. The values ​​of various parameters of the scene are obtained by sampling from the distributions of each factor and their joint distribution using Monte Carlo sampling.

[0105] [Example] By obtaining weather and traffic flow information for a certain region, data on key factors in the scenario are obtained. The distribution of each factor is shown in Table 3:

[0106] Table 3 Determination of Key Factor Distribution Types

[0107] factor Distribution type factor Distribution type Light intensity W(0.617,0.265) Large vehicles W(0.655,0.116) Cloud cover N(0.635,0.318) medium-sized vehicles N(0.237,0.223) Rainfall W(0.310,0.194) small vehicles W(1.120,0.287) wind speed N(0.308,0.164) Radiation intensity N(0.619,0.254) vehicle speed N(0.737,0.122) temperature N(0.762,0.183)

[0108] Where W() represents the Weibull distribution and N() represents the normal distribution. We also obtained the correlations of the above 10 key factors. Taking light intensity and cloud cover as an example, their correlation is -0.93, therefore we need to construct suitable copula functions for them. The AIC information criterion test results of the distribution fitting results of the five copula functions are shown in Table 4:

[0109] Table 4. AIC Test Results

[0110] Model AIC Gussion copula -56.11 t copula 991.88 Gumbel-copula 2.00 Clayton-copula -42.94 Frank-copula -19.02

[0111] Therefore, we can choose the Gussion copula function as the joint distribution model for sunshine and cloudy.

[0112] Step 3: Construct a factor impact model to simulate the impact of key factors on the UAV target tracking task process;

[0113] Step 1: Building a model of weather influencing factors;

[0114] 1) Fog Impact Modeling: The impact of heavy fog on drones is obtained through a fog formation model. The model is as follows:

[0115] I(x)=J(x)+A(1-t(x))

[0116] Where I(x) represents the image after adding the effect of heavy fog; J(x) represents the image captured by the drone in clear weather; t(x) represents the transmittance, which represents the effect of light intensity on the original image; and A represents the light intensity.

[0117] 2) Rain and Snow Impact Modeling: The impact of rain and snow on images is analyzed using a general rain and snow model. The model is as follows:

[0118]

[0119] Where O represents the image after adding rain and snow effects; B represents the image captured under the working conditions; S t Raindrops that are in the same direction and shape; α represents the cumulative amount of raindrops in layer s; α represents transmittance, which represents the effect of light intensity on the image during rainy weather; A represents light intensity.

[0120] 3) Wind speed impact modeling: The wind speed impact is represented by the force exerted by the wind on the UAV per unit time, and the magnitude of this force is described by momentum.

[0121] F wind =v wind ·m gas

[0122] Among them, Fwind v represents the force exerted by the wind on the drone per unit time. wind Wind speed, m gas The mass of air acting on the drone is expressed by the following formula:

[0123] m gas =ρ gas ·S·v wind

[0124] Where, ρ gas S represents the air density, and S represents the contact area of ​​the drone's windward side.

[0125] Step 2: Occlusion factors affect model construction;

[0126] The obstruction range of a building is calculated using the drone's location and the location and height information of ground buildings. Assuming the road direction is the y-direction, the obstruction range is:

[0127]

[0128] Where w is the area of ​​obstruction by the building on the ground, (x uav y uav h uav Let (x) be the position of the UAV at time t, and (x) be the position of the UAV at time t obj y obj h obj The location and height of the building are shown in the figure. If the target is within the obstruction range, the drone will lose the target.

[0129] Step 3: Construction of the target and interference target impact model;

[0130] Factors influencing the target and interfering targets include target type, target size, and color, with target type and size being affected by target type. The types of targets and interfering targets (t) are determined through sampling. sp Type t t Size t si Color t c .

[0131] [Example] Taking the effect of occlusion as an example, the impact of occlusion is as follows: Figure 4 As shown. Assuming the road direction is y-axis, the coordinates of the top floor are (100, 10, 10), while the drone's current position is (98, 10, 15). Therefore, the obstruction range of the building on the ground can be calculated as follows:

[0132]

[0133] Therefore, it can be concluded that when the drone is in this position, the ground obstruction range is 6m. When a dynamic target is within this range, the drone cannot obtain the target's location information.

[0134] Step 4: Establish a Markov decision process for drones and dynamic targets;

[0135] By constructing a Markov decision process between the UAV and the ground dynamic target, the state space, action space, state transition probability, and reward function of the UAV and the ground dynamic target are defined, and an adversarial environment between the two is generated.

[0136] Step 1: Define the state space of the UAV and the dynamic target;

[0137] By placing the UAV and the dynamic target in a three-dimensional coordinate system, and using a Markov decision process, the state space for UAV and ground dynamic target tracking is established:

[0138] S:(x target x uAv y target y uav v target v uAv φ target φ uAv h uav )

[0139] Where, x target This represents the target's position information in the x-direction of a three-dimensional coordinate system; x uav Represents the drone's position information in the x-direction; y target This represents the target's position information on the y-axis; y uav This represents the drone's position information on the y-axis; v target V represents the speed of the target; uav Represents the speed of the drone; φ target The deflection angle representing the target; φ uav The deflection angle of the drone; h uav The altitude information of the drone.

[0140] Step 2: Construct the state and action space of the UAV and dynamic targets;

[0141] In a scenario where a drone tracks a dynamic target, the action space for both the drone and the dynamic target is:

[0142] A: (a uav a target )

[0143] Among them, a uav It is a three-dimensional vector representing the acceleration of the UAV; a targetIt is a two-dimensional vector representing the acceleration of a dynamic target on the ground.

[0144] Step 3: Define the state transition probabilities between the UAV and the dynamic target's state and actions;

[0145] In the scenario of UAV tracking a dynamic target, the state transition probability function is:

[0146] S t+1 =f(S) t A t )

[0147] Among them, S t A represents the state of the UAV and the dynamic ground target at time t; t This indicates the actions taken by the drone and the dynamic ground target at that moment; S t+1 This indicates the state of the drone and the dynamic ground target at the next moment.

[0148] Step 4: Define the reward function for the Markov decision process of the UAV and the dynamic target;

[0149] The reward for a dynamic target consists of two parts: a positional reward and a tracking penalty. The reward function is determined by the relative position of the UAV and the dynamic target, and is expressed as:

[0150]

[0151] Where, r pos Represents positional reward; ε is a minimal number; h uav θ is the drone's flight altitude; θ is the drone's detection angle; c t The factor used to measure whether a ground target is being tracked has the following values:

[0152]

[0153] c t =1 represents a dynamic target escaping the drone's tracking, while c t =0 indicates that the target is being tracked by a drone.

[0154]

Example

[0155] Table 5 Range of Markov Decision Process Parameters

[0156]

[0157] Step 5: Construct a target behavior model based on reinforcement learning, and combine it with key factors to form a test scenario for UAV target tracking;

[0158] The constructed Markov decision process is solved using the TD3 reinforcement learning algorithm, which is used to solve problems in high-dimensional continuous action spaces.

[0159] Step 1: Initialize the current actor-critic network and the target actor-critic network in the TD3 network, and randomize their network parameters;

[0160] Step 2: In the current actor network and the target actor network, select the drone and the dynamic target in state S respectively. t Given the given conditions, actions A and A′ should be taken. Calculate the expected reward of each action, using the following function:

[0161]

[0162] Where Q(s,a) represents the dynamic target and the UAV in state S. t An estimate of the return or expected profit that can be obtained by taking action A. Represents the expected value r pos t γ represents the reward value of the dynamic objective at time t; γ represents the discount factor, used to measure the importance of the current reward and the future reward.

[0163] Step 3: Place the calculated expected reward and corresponding action into their respective experience pools. Using the current critic network and the target critic network, select the action A with the highest expected reward from the experience pools and selected actions. max and A′ max ;

[0164] Step 4: Compare and select action A max and A′ max Given the reward value, select the action corresponding to the smallest reward and update the current actor-critic network parameters. After updating the current actor-critic network parameters g (g is a positive integer) times, pass its parameters to the target actor-critic network.

[0165] For each state S t Steps 2-4 are executed repeatedly for r (where r is a positive integer) times, ensuring that the TD3 algorithm selects the optimal action A in each state to maximize the expected reward. The model obtained after training r (where r is a positive integer) times is the target behavior model. This model is used to select the optimal target behavior, and combined with the key factor values ​​obtained in the above steps, an edge test scenario for UAV target tracking is generated.

[0166] [Example] This article uses a UAV based on the Kalman filter target tracking algorithm as a test object. Combined with the TD3 reinforcement learning algorithm mentioned in step five, a behavioral model of the target is generated. The parameters of the TD3 algorithm are shown in Table 6.

[0167] Table 6 TD3 Hyperparameters

[0168] parameter value Discount factor γ 0.99 Actor Network Learning Rate 0.005 Critic Network Learning Rate 0.01 Total training stride 18000 Batch size 128 Buffer size 10000

[0169] The above steps are combined to construct a test scenario for UAV dynamic target tracking. The UAV is then placed into this test scenario to obtain the edge test scenario for UAV dynamic target tracking. This method is compared with a Monte Carlo scene generation method to test the UAV tracking success rate. The results show that this method can efficiently find the edge test scenario for UAV dynamic target tracking, and the test results are shown in Table 7.

[0170] Table 7 Test Results

[0171]

Claims

1. A method for generating edge test scenarios for dynamic target tracking of unmanned aerial vehicles based on reinforcement learning, characterized in that... It includes the following steps: Step 1: Construct a task scenario for drone target tracking and define the drone tracking success rate; The UAV dynamic target tracking task is divided into five stages: UAV takeoff, target identification, target tracking, return to base, and landing. The UAV tracking success rate is defined as the proportion of the total task time during which the UAV can accurately acquire the dynamic target's position information. The calculation formula is as follows: Where t=0 represents the time when the dynamic target first appears in the UAV's field of view during the target tracking task; This indicates the number of frames in which the drone successfully tracked the target during the target tracking task; This indicates the total number of frames in the target tracking task; Step 2: Analyze the influencing factors in the task scenario, and combine real data to obtain the distribution type and correlation of each influencing factor, and construct the joint distribution of the influencing factors with correlation; Step 1: Analyze the influencing factors in each mission scenario and collect real-world scenario data for each influencing factor. The influencing factors for each stage of the UAV mission are shown in the table below: Table 1 Influencing Factors Task Phase Influencing factors take off Weather, wind speed Identify target Weather, target characteristics, interfering targets, road conditions Tracking target Weather, wind speed, target characteristics, interfering targets, road conditions Return Weather, wind speed, drone location landing Weather, wind speed, drone location Based on the task scenario analysis, the influencing factors at each stage are analyzed, and the values ​​of each influencing factor in a certain region within the time period T are collected to obtain a real-world scenario dataset. Step 2: Combine the data to obtain a distribution model of the influencing factors; The data of each influencing factor collected in step 1 are normalized, and the distribution of each influencing factor is fitted to obtain the distribution model of each influencing factor. Step 3: Correlation analysis of influencing factors; Based on the normalized data of influencing factors, the correlation between influencing factors is analyzed pairwise using the Pearson correlation coefficient, the formula of which is as follows: Where r represents the correlation between the two influencing factors. , This represents the value of the i-th sample among the influencing factors of X and Y; , This represents the average value of the influencing factors X and Y. Further, a correlation matrix is ​​constructed based on the correlation calculation results. If the correlation between two influencing factors exceeds 0.6, proceed to step 4 until the joint distribution of all relevant influencing factors is determined, then proceed to step 3. Otherwise, proceed directly to step 3. Step 4: Construct a joint distribution for the correlated influencing factors using the copula function; Based on the correlation matrix obtained in step 3, the copula function is used to construct a joint distribution model of the relevant influencing factors for all correlated influencing factors; assuming there are n influencing factors in the UAV ground target dynamic tracking scenario, the distribution of each influencing factor is denoted as follows: , … Then its joint distribution is denoted as: in, To represent the joint distribution of all relevant influencing factors, the optimal Copula function needs to be selected as the joint distribution function using the AIC information criterion. The formula for calculating the AIC value is as follows: in, This represents the value of AIC. The value of the likelihood function is obtained by sampling the values ​​of various parameters of the scene from the distribution and joint distribution of various influencing factors using the Monte Carlo sampling method. Step 3: Construct a factor influence model to simulate the impact of influencing factors on the UAV target tracking task process; Step 1: Building a model of weather influencing factors; 1) Fog Impact Modeling: The impact of heavy fog on drones is obtained through a fog formation model. The model is as follows: in, This represents the image after the fog effect has been added. This refers to images captured by a drone on a sunny day. Transmittance represents the effect of light intensity on the original image; Represents light intensity; 2) Rain and Snow Impact Modeling: The impact of rain and snow on images is analyzed using a general rain and snow model. The model is as follows: Where O represents the image after adding the effects of rain and snow; B represents the image captured under the working conditions; Raindrops that are in the same direction and shape; express The accumulation of layers of raindrops; Transmittance represents the effect of light intensity on the image during rainy weather; Represents light intensity; 3) Wind speed impact modeling: The wind speed impact is represented by the force exerted by the wind on the UAV per unit time, and the magnitude of this force is described by momentum. in, This represents the force exerted by wind speed on the drone per unit time. Indicates wind speed. The formula for calculating the air mass flow rate acting on the drone is: in, Indicates the density of air. This indicates the contact area of ​​the drone's windward side; Step 2: Occlusion factors affect model construction; The obstruction range of a building is calculated using the location of the drone and the location and height information of ground buildings. Assuming the road direction is the y-direction, the obstruction range is: Where w represents the area of ​​obstruction by the building on the ground. Let t be the position of the drone. The location and height of the building are used to determine the target's location; if the target is within the obstruction range, the drone will lose the target. Step 3: Construction of the target and interference target impact model; Factors influencing the target and interfering targets include target type, target subclass, target size, and color. Target type and size are also affected by target type. The types of targets and interfering targets are determined through sampling. Subclasses of target types ,size ,color ; Step 4: Establish a Markov decision process for drones and dynamic targets; Step 1: Define the state space of the UAV and the dynamic target; By placing the UAV and the dynamic target in a three-dimensional coordinate system, and using a Markov decision process, the state space for UAV and ground dynamic target tracking is established: in, This represents the target's position information in the x-direction of the three-dimensional coordinate system; This represents the drone's position information in the x-direction; This represents the target's position information on the y-axis; This represents the drone's position information on the y-axis; Represents the speed of the target; Represents the speed of the drone; The angle representing the target; The deflection angle representing the drone; Information representing the drone's altitude; Step 2: Construct the state and action space of the UAV and dynamic targets; In a scenario where a drone tracks a dynamic target, the action space for both the drone and the dynamic target is: in, It is a three-dimensional vector representing the acceleration of the drone; It is a two-dimensional vector representing the acceleration of a dynamic ground target; Step 3: Define the state transition probabilities between the UAV and the dynamic target's state and actions; In the scenario of UAV tracking a dynamic target, the state transition probability function is: in, This represents the state of the UAV and the dynamic ground target at time t; This indicates the actions taken by the drone and the dynamic ground target at that moment; This indicates the state of the drone and the dynamic ground target at the next moment; Step 4: Define the reward function for the Markov decision process of the UAV and the dynamic target; The reward for a dynamic target consists of two parts: a positional reward and a tracking penalty. The reward function is determined by the relative position of the UAV and the dynamic target, and is expressed as: in, Indicates location-based rewards; It is a very small number; This refers to the flight altitude of the drone; For the detection angle of the drone; The factor used to measure whether a ground target is being tracked has the following values: This represents a dynamic target escaping the drone's tracking, and This indicates that the target is being tracked by a drone; Step 5: Construct a target behavior model based on reinforcement learning, and combine it with influencing factors to form a test scenario for UAV target tracking; Step 1: Initialize the current actor-critic network and the target actor-critic network in the TD3 network, and randomize the network parameters; Step 2: In the current actor network and the target actor network, select the drone and the dynamic target in their respective states. Actions to be taken in the following situations: A and Calculate the expected reward of this action. The reward calculation function is: in, Indicates the dynamic target and the state of the drone. An estimate of the return or expected profit that can be obtained by taking action A. Indicates the expected value. This represents the reward value of the dynamic objective at time t; This represents the discount factor, used to measure the importance of current and future rewards; Step 3: Place the calculated expected reward and corresponding action into their respective experience pools. Use the current critic network and the target critic network to select the action with the highest expected reward from the experience pools and selected actions. and ; Step 4: Compare and select actions and Given the reward value, select the action corresponding to the minimum reward and update the current actor-critic network parameters. After the current actor-critic network parameters are updated g times, pass its parameters to the target actor-critic network, where g is a positive integer. For each state Execute steps 2-4 of step five for r iterations, so that in each state, the TD3 network can select the optimal action to maximize the expected reward, where r is a positive integer. The model obtained after training r times is the target behavior model. The optimal target behavior is selected through this model, and combined with the obtained influencing factor values, an edge test scenario for UAV target tracking is generated.