Unmanned aerial vehicle cognitive electronic reconnaissance system based on scene segmentation

Abstract

The invention relates to the technical field of unmanned aerial vehicle reconnaissance, in particular to an unmanned aerial vehicle cognitive electronic reconnaissance system based on scene segmentation, which comprises an unmanned aerial vehicle, a reconnaissance task area and an unmanned aerial vehicle learning module, the reconnaissance task area comprises a source domain module and a target domain module, and the reconnaissance task area is divided into a plurality of small areas; the source domain module selects a small region as a Souter task interaction environment, the Souter task interaction environment is used as a source domain, and source domain information is collected; the unmanned aerial vehicle learning module comprises a deep learning algorithm and a difference adaptive migration deep reinforcement learning algorithm; the target domain module is provided with a plurality of Scurter2 target domains, deep reinforcement learning is carried out on the Scurter2 target domains, Scurter2 target domain information is collected, a source domain unmanned aerial vehicle control strategy is transferred and learned to the Scurter2 target domains according to the Scurter2 task interaction environment and the information difference of the Scurter2 target domains, strategy network input is adjusted by utilizing the environment difference of the source domain and the Scurter2 target domains, and a cognitive electronic reconnaissance system is established. The method can achieve the quick starting of the unmanned aerial vehicle, and optimizes the convergence result.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) reconnaissance, and in particular to an unmanned aerial vehicle (UAV) cognitive electronic reconnaissance system based on scene segmentation. Background Art

[0002] Electronic reconnaissance is used to intercept, identify and locate radiation source signals, laying the foundation for accurate perception of battlefield electromagnetic situation, and providing guidance information for interference with target air defense systems and firepower destruction.

[0003] Electronic reconnaissance strategies are used to guide the electronic reconnaissance payload to interact with the environment in the time domain, frequency domain, and air domain, and to achieve deep integration with the maneuvering control of the reconnaissance platform, so as to achieve the purpose of quickly and accurately obtaining radiation source information. Traditional electronic reconnaissance equipment is limited by its perception ability and environmental adaptability, and its performance is poor in the battlefield electromagnetic environment without prior knowledge. The introduction of artificial intelligence technology can enable electronic reconnaissance equipment to have learning capabilities and provide solutions for autonomous reconnaissance of intelligent agents under weak prior conditions. In the collaborative reconnaissance and interference tasks of multiple drones, the introduction of artificial intelligence technology can optimize the collaborative reconnaissance strategy of drone groups in uncertain environments, solve the problems of high action space dimensions and difficult strategies, and optimize the time-varying priority of reconnaissance tasks. In addition, in order to avoid target and terrain threats and improve the efficiency of drone reconnaissance interference and battlefield survivability, some scholars focus on drone reconnaissance route planning. However, the battlefield electromagnetic situation is complex and changeable, and the reconnaissance targets are often non-cooperative and lack prior knowledge of target threats. At the same time, drones can only gradually realize the recognition of target threats through interaction with the environment, breaking the limitations caused by weak prior knowledge.

[0004] When the actual mission area and the training mission area have obvious differences in scope, the confrontation situation between the two sides on the actual battlefield changes rapidly, and the combat area, combat mission, and combat unit may change at any time, resulting in the inapplicability of the strategy. It takes a lot of time to train a model that fits the current mission environment from scratch using a deep reinforcement learning algorithm. Summary of the invention

[0005] In view of this, the present invention provides a UAV cognitive electronic reconnaissance system based on scene segmentation to achieve the purpose of rapid startup of the UAV and optimize the convergence results.

[0006] In a first aspect, the present invention provides a UAV cognitive electronic reconnaissance system based on scene segmentation, comprising a UAV, a reconnaissance mission area, and a UAV learning module; the reconnaissance antenna beam of the UAV is aligned with a radar;

[0007] The reconnaissance mission area includes a source domain module and a target domain module, and the reconnaissance mission area is divided into a plurality of similar small areas;

[0008] The source domain module is used to set the Scouter task interaction environment, select one of the small areas as the Scouter task interaction environment, and the drone collects and processes basic information of the Scouter task interaction environment, including but not limited to length, width, and height; the drone uses the deep reinforcement learning algorithm to generate a drone control strategy, and uses the Scouter task interaction environment as the source domain;

[0009] The drone learning module includes a deep learning algorithm and a differential adaptive transfer deep reinforcement learning algorithm;

[0010] The target domain module is used to set multiple Scouter2 target domains, and all the small areas except the source domain are used as Scouter2 target domains respectively, and deep reinforcement learning is performed on the Scouter2 target domains. The drone collects and processes basic information of multiple Scouter2 target domains, including but not limited to length, width, and height. According to the difference between the Scouter task interaction environment and the Scouter2 target domain information, a deep reinforcement learning algorithm is used to perform adaptation training, and the drone control strategy trained in the source domain is transferred to the Scouter2 target domain. The difference between the source domain and the Scouter2 target domain is used to adjust the policy network input, so that the deep reinforcement learning algorithm converges quickly, and the difference adaptive transfer deep reinforcement learning algorithm is used to build a drone cognitive electronic reconnaissance strategy network;

[0011] Multiple trained Scouter2 target domains are spliced ​​into a reconnaissance mission area.

[0012] Through the above technical solution, the reconnaissance mission area is divided into multiple Scouter2 target domains, and the difference adaptive transfer deep reinforcement learning algorithm is used in a single Scouter2 target domain to accelerate the learning speed of the UAV in the Scouter2 target domain. Finally, multiple Scouter2 target domains are spliced ​​into a reconnaissance mission area, so that the UAV can start quickly and optimize the convergence results, which can better meet the needs of cognitive electronic reconnaissance tasks, making the UAV's reconnaissance behavior more diverse. In addition to flying around and reciprocating outside the radar detection power, it also shows behaviors such as escaping radar detection, repeatedly performing reconnaissance tasks, and quickly or smoothly adjusting the flight attitude.

[0013] Secondly, in the Scouter mission interaction environment, the radar is responsible for searching for aerial targets and locking and destroying threatening targets, and the radar signal of the Scouter mission interaction environment area is detected by using the UAV platform carrying the reconnaissance payload; the radar detection power range is approximately the firepower strike range, and the radius is set to D a, the area where the UAV can intercept radar signals and fly safely, the width is set to D c , the two sides of the drone fuselage represent the coverage of the reconnaissance antenna, the azimuth angle is set to α, the pitch angle is set to β, the azimuth angle α and the pitch angle β of the drone's antenna main lobe beam cover the radar, and the radar signal power P received by the drone's reconnaissance payload t .

[0014] The above technical solution is the basis for realizing the cognitive electronic reconnaissance mission of UAV.

[0015] Thirdly, when the azimuth angle α and the pitch angle β of the main lobe beam of the UAV antenna cover the radar, the radar signal power P received by the UAV reconnaissance payload is t When the sensitivity is exceeded, the radar signal can be intercepted by the drone reconnaissance payload, which is the reconnaissance flag position. n,t , the judgment conditions are:

[0016]

[0017] In formula (1), d t is the three-dimensional spatial position of the radar in the geocentric coordinate system s r,t =[x r,t ,y r,t ,z r,t ] and the three-dimensional spatial position of the drone u,t =[x u,t ,y u,t ,z u,t ], denoted as d t =s r,t -s u,t ; d t Projected to the UAV platform coordinate system to obtain l t , and then l t Project onto the three coordinate axes to obtain the distance vector l t,x , l t,y , l t,z ;P r,t is the radar transmission power, G r,t is the radar antenna gain, G u,t is the gain of the drone reconnaissance antenna, λ is the signal wavelength, γ is the antenna polarization loss, and D a is the fire attack distance, D c is the width of the range ring for effective and safe reconnaissance by the drone;

[0018] The vector distance d t The angle between the drone wing is defined as the behavior angle θ BHA , describing the alignment of the reconnaissance antenna to the radar:

[0019] θBHA =deg(arc cos <v mov l t >)-90 (2),

[0020] In formula (2), v mov The unit vector representing the speed of the drone, θ BHA Projection on the xoy plane and yoz plane of the UAV platform coordinate system to obtain the angle |θ BHA | xoy and |θ BHA | yoz , n,t The judgment condition is rewritten as:

[0021]

[0022] The key to completing long-term continuous reconnaissance missions is to continuously intercept radar signals; i is the number of interception cycles, and its judgment condition is:

[0023]

[0024] In formula (4), N e represents the maximum number of steps allowed in each episode, Q represents the error in receiving and processing echoes by the radar;

[0025] According to formulas (1) and (4), the key to completing the UAV cognitive electronic reconnaissance mission is to let the UAV learn the vector distance d t Set Δ;

[0026] In an observation cycle time τ, the UAV platform is set to move with constant acceleration, then the vector distance d t The change in Δd t This is achieved by maneuvering the UAV platform; by controlling the acceleration vector a of the UAV platform at time t u,t Control Δd t The key to the UAV cognitive electronic reconnaissance mission strategy is to optimize the acceleration a u,t , the objective function is:

[0027]

[0028] In formula (5), λ1, λ2, λ3, η1, η2, η3 represent adjustment coefficients, all of which are greater than 0. The UAV cognitive electronic reconnaissance mission controls the acceleration a u,t To minimize the objective function J(a u,t ); u,t Projected onto the azimuth plane and pitch plane of the UAV platform respectively, the azimuth acceleration a is obtained α,t and pitch acceleration aβ,t ; The UAV cognitive electronic reconnaissance mission strategy is to optimize a α,t and a β,t The value of is used to control the maneuverability of the UAV platform.

[0029] By adopting the above technical solution, it can be seen from formulas (1) and (4) that the core of the UAV's reconnaissance mission is to control the relative vector distance dt between it and the radar. Formula (5) reflects the core of the UAV's cognitive electronic reconnaissance mission strategy, building a bridge for the following optimization and improving the accuracy of the calculation results.

[0030] Fourthly, there are three small areas, one of which is 75km×75km in size, and deep reinforcement learning is performed to learn the reconnaissance strategy a. u,t ; In an observation cycle time τ, the reconnaissance strategy a u,t This will produce a vector distance d t The change in Δd t ; Δd t It is determined by the UAV cognitive electronic reconnaissance strategy network and the UAV maneuverability, and has nothing to do with the size of the reconnaissance mission area; the strategy result Δd t In the reconnaissance mission area, the relative change δ t The relationship is shown in formula (6):

[0031]

[0032] In formula (6), X max , X min , Y max , Y min , H max , H min are the maximum and minimum values ​​of the length, width and height of the reconnaissance mission area; l t,x , l t,y , l t,z They are the distance vectors of the three coordinate axes. When the length, width, and height of the reconnaissance mission area change, they become X' max , X' min , Y' max , Y' min , H' max , H' min Then, substitute into formula (6), δ t The same reconnaissance strategy produces different results in different reconnaissance mission areas. The scaling factor is designed according to the change of the mission area, so that the UAV cognitive electronic reconnaissance strategy network can quickly adapt to the reconnaissance mission areas of different sizes, as shown in formula (7):

[0033]

[0034]

[0035]

[0036]

[0037] In formula (7), ρ1, ρ2, and ρ3 are the scaling factors of the three coordinate axes respectively.

[0038] By adopting the above technical means, the transfer learning between the Scouter task interaction environment and the Scouter2 target domain is realized by calculating the scaling factor, so that the UAV can quickly start in the Scouter2 target domain, optimize the convergence results, and achieve the purpose of rapid convergence of the entire reconnaissance mission area.

[0039] In a fifth aspect, according to a scene segmentation-based UAV cognitive electronic reconnaissance system according to claim 2, it is characterized in that part of the UAV cognitive electronic reconnaissance task is an observable Markov strategy process, and the part of the observable Markov strategy process of the UAV cognitive electronic reconnaissance task is described by a six-tuple <∑, Ο, A, F, R, γ>; ∑ is the set of actual states of the UAV and the radar Σ r is the radar position set, Σ r is the set of drone positions, is the set of UAV speeds, Π is the set of radar signal powers intercepted by the UAV, and Ο is the set of states observed by the UAV When the drone processes the intercepted radar signal, it will add a certain amount of system noise, and the observed radar state will deviate from the actual value. is the set of vector distances between the UAV and the radar with deviations, is the set of deviations between the unit vector of the drone’s line of sight and the radar’s line of sight and the unit vector of the drone’s velocity direction. is the biased radar signal power set intercepted by the UAV; A is the UAV reconnaissance strategy action set {A α , A β}, A α is the set of azimuthal accelerations, A β is the set of pitch accelerations; F is the state transfer function, R is the reward function, and γ is the discount factor; at time t, the real environment state s t =[s r,t ,s u,t ,v u,t ,P t ]∈, the state observed by the drone After the UAV control strategy network, the execution strategy π is output and action a is selected t =[aα,t ,a β,t ]∈, with the transfer function Transfer to state s t+1 =[s r,t+1 ,s u,t+1 ,v u,t+1 ,P t+1 ]∈get the corresponding reward R(s t+1 ,a t ,s t );During the reconnaissance mission time period T, the expected cumulative reward obtained by the execution strategy π is used to evaluate its quality:

[0040]

[0041] Formula (8) is used to design the reward function of deep reinforcement learning, and the optimal execution strategy π is selected by maximizing the cumulative reward. * , so as to obtain J(a u,t )Minimal movement

[0042]

[0043] In a sixth aspect, according to the UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 5, it is characterized in that the purpose of the transfer learning is to transfer the s and learning tasks s The knowledge gained in is transferred to the target domain Δ of Scouter2 g , helping Scouter2 learn task Τ in the target domain g To obtain better learning effect; in the Scouter task interaction environment, the strategy process of UAV cognitive electronic reconnaissance task is expressed by six-tuple <∑ sσ ,Ο sσ ,A sσ ,F sσ ,R sσ ,γ sσ > to describe; in Scouter2 target domain, the sextuplet <∑gσ,Ο gσ ,A gσ ,F gσ ,R gσ ,γ gσ > to describe;∑ s *A s The source domain Δ s , F s and R s Determines the learning task T sThe goal of deep reinforcement learning is to divide the transfer problem into two cases: "the source domain and the target domain of Scouter2 are different, but the learning tasks are the same" and "the source domain and the target domain of Scouter2 are the same, but the learning tasks are different".

[0044] By adopting the above technical means and controlling the variables, two problems of transfer deep reinforcement learning are discussed to improve the applicability of this method.

[0045] In a seventh aspect, the deep reinforcement learning is a difference adaptive transfer deep reinforcement learning algorithm, and the deep reinforcement learning includes reward modeling and difference adaptive normalized observation state, wherein the reward function formula is:

[0046]

[0047] In formula (10), is the vector distance between the UAV and the radar with observation error; |θ BHA | xoy,t and |θ BHA | yoz,t is the time t that θ BHA Project the angles on the xoy plane and yoz plane of the UAV platform coordinate system respectively;

[0048] The variables that have a direct impact on the UAV cognitive electronic reconnaissance mission are selected as the input of the policy network. The observation state of the UAV includes five sets {S u , u ,~,~,~}, respectively are the drone positions Drone speed Vector distance between drone and radar Deviation between the drone's speed direction and the radar's line of sight direction Intercepted radar signal power The observation state set contains thirteen-dimensional vectors, which have different dimensions and different sparsity, so the degree of numerical change in one strategy time is also different. The thirteen-dimensional vectors are normalized to obtain the difference adaptive normalized observation state at time t

[0049] By adopting the above technical means, the difference adaptive transfer deep reinforcement learning algorithm has obvious advantages over the deep reinforcement learning algorithm in terms of fast startup and optimized convergence results. The difference adaptive transfer deep reinforcement learning algorithm is used to accelerate the learning speed of drones in the target domain.

[0050] In an eighth aspect, the difference adaptive normalized observation state includes a change in the mission area range of the target domain and the source domain, a change in the speed range of the drone of the target domain and the source domain, and a change in the difficulty of the reconnaissance mission of the target domain and the source domain;

[0051] Among them, the changes in the task area range of the target domain and the source domain are mainly the changes in the length and width of the task area: the height of the drone reconnaissance area mainly depends on the drone ceiling. When the drone ceiling remains unchanged, the height of the reconnaissance area remains unchanged. The difference between the target domain and the source domain task area is used to adjust the observation state, and the observation state of the task area difference adaptive normalization is obtained.

[0052]

[0053] In formula (11), Δx = X max -X min ,Δy=Y max -Y min ,Δh=H max -H min ,Δv=V max -V min , X max , X min , Y max , Y min , H max , H min are the maximum and minimum values ​​of the length, width, and height of the task area; V max , V min are the maximum and minimum values ​​of the UAV maneuvering speed respectively; v mov,x , v mov,y , v mov,z are the projections of the UAV speed direction vector on the three coordinates; l t,x , l t,y , l t,z are the distance vectors of the three coordinate axes; s,t is the radar signal power received by the reconnaissance payload, P u,min is the sensitivity of the reconnaissance payload, Δx'=X' max -X' min ,Δy'=Y' max -Y' min , X' max , X' min , Y' max , Y' min are the maximum and minimum values ​​of the length, width, and height of the task area in the target domain, respectively;

[0054] The speed range of the drone in the Scouter2 target domain and the source domain changes. When the speed range of the drone changes, the change in the vector distance between the drone and the radar in the Scouter2 target domain and the source domain will change, causing the transfer function of the next state to change, making the policy network migrated to the Scouter2 target domain inapplicable; the difference in the speed range of the drone in the Scouter2 target domain and the source domain is used to adjust the observation state, and the observation state with adaptive normalization of the speed difference is obtained.

[0055]

[0056] In formula (12), Δv'=V' max -V' min The maximum value of the UAV speed in the Scouter2 target domain is V' max , the minimum value is V' min ;

[0057] The difficulty of the reconnaissance mission of Scouter2 target domain and source domain varies. The length of the drone flight time is not limited in the reconnaissance mission, so the change of the drone speed value has no direct impact on the task completion criterion. In order to reduce the input dimension and accelerate convergence, the observation state does not consider the drone speed; at this time, the observation state formula (13) is:

[0058]

[0059] The ninth aspect also includes simulation experiments. The simulation experiments set up three groups of experiments to test the learning effect of the deep reinforcement learning algorithm of difference adaptive transfer by changing the task area range, drone maneuvering speed range, and task difficulty in the Scouter task interaction environment.

[0060] By adopting the above-mentioned technical means, the advantages of the test algorithm in terms of fast startup and optimized convergence results are tested, thereby reducing mission risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a schematic diagram of the division of the Scouter task interaction environment area;

[0062] Figure 2 It is a schematic diagram of the division of electronic reconnaissance mission areas;

[0063] Figure 3 It is a network framework for differential adaptive transfer deep reinforcement learning;

[0064] Figure 4 It is to use the reward to evaluate the convergence performance of the algorithm;

[0065] Figure 5It is to use BHA to evaluate the convergence performance of the algorithm;

[0066] Figure 6 It is the track of drone reconnaissance in the 50km×50km mission area;

[0067] Figure 7 It is the drone reconnaissance track of Episode 4711 in the 150km×150km mission area;

[0068] Figure 8 It is to use the reward to evaluate the convergence performance of the algorithm;

[0069] Fig. 9 It is to use BHA to evaluate the convergence performance of the algorithm;

[0070] Fig.10 The maximum speed of the drone is 150m / s. The reconnaissance track of the drone in episode 4427;

[0071] Fig.11 The maximum speed of the drone is 400m / s. The reconnaissance track of the drone in episode 4813;

[0072] Fig.12 It is to use the reward to evaluate the convergence performance of the algorithm;

[0073] Fig.13 BHA is used to evaluate the convergence performance of the algorithm. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0075] This application proposes a UAV cognitive electronic reconnaissance system based on scene segmentation, which uses a differential adaptive transfer deep reinforcement learning algorithm to accelerate the learning speed of the UAV in the target domain. Compared with the deep reinforcement learning algorithm, the differential adaptive transfer deep reinforcement learning algorithm has obvious advantages in fast startup and optimized convergence results. It can better meet the needs of cognitive electronic reconnaissance tasks, and the reconnaissance behavior of the UAV is more abundant. In addition to flying around and flying back and forth outside the radar detection power, it also shows behaviors such as escaping radar detection, repeating reconnaissance tasks, and quickly or smoothly adjusting the flight attitude. It shows that the UAV not only recognizes the range of radar detection power, but also recognizes how to ensure its own survival in the confrontation with the radar. It recognizes the standard for completing the reconnaissance mission and can repeat the mission until the standard is met. It recognizes what kind of flight attitude is beneficial to completing the reconnaissance mission and can choose to adjust quickly or smoothly according to the specific mission scenario.

[0076] A UAV cognitive electronic reconnaissance system based on scene segmentation includes a UAV, a reconnaissance mission area and a UAV learning module; the reconnaissance mission area includes a source domain module and a target domain module, and the reconnaissance mission area is divided into multiple similar small areas;

[0077] The source domain module is used to set up the Scouter task interaction environment. A small area is selected as the Scouter task interaction environment. The drone collects and processes the basic information of the Scouter task interaction environment, including but not limited to length, width, and height. The drone uses a deep reinforcement learning algorithm to generate a drone control strategy and uses the Scouter task interaction environment as the source domain.

[0078] The drone learning module includes deep learning algorithms and differential adaptive transfer deep reinforcement learning algorithms; in the Scouter mission interaction environment, the radar is responsible for searching for aerial targets and locking and destroying threatening targets, and the radar signal in the Scouter mission interaction environment area is detected by using the drone platform carrying a reconnaissance payload. Although the drone can know the approximate deployment location of the radar in advance through intelligence data such as early satellite reconnaissance, the detection power of the radar is not a priori for the drone, which will pose a great threat to the survival of the drone. Figure 1 In the example, the radar detection power range is approximately equal to the firepower strike range, which is represented by the red area and the radius is set to D. a ; The area where the drone can intercept radar signals and fly safely, the width is set to D c ; The two sides of the drone fuselage represent the coverage of the reconnaissance antenna, the azimuth angle is set to α, and the pitch angle is set to β.

[0079] Among them, the azimuth angle α and pitch angle β of the main lobe beam of the UAV antenna cover the radar, and the radar signal power P received by the UAV reconnaissance payload tWhen the azimuth angle α and pitch angle β of the main lobe beam of the UAV antenna cover the radar and meet the radar signal power P received by the UAV reconnaissance payload t When the sensitivity is exceeded, the radar signal can be intercepted by the drone reconnaissance payload, which is the reconnaissance flag position. nt , the judgment conditions are:

[0080]

[0081] In formula (1), d t is the three-dimensional spatial position of the radar in the geocentric coordinate system s r,t =[x r,t ,y r,t ,z r,t ] and the three-dimensional spatial position of the drone u,t =[x u,t ,y u,t ,z u,t ], denoted as d t =s r,t -s u,t . t Projected to the UAV platform coordinate system to obtain l t , and then l t Project onto the three coordinate axes to obtain the distance vector l t,x , l t,y , l t,z ;P r,t is the radar transmission power, G r,t is the radar antenna gain, G u,t is the gain of the drone reconnaissance antenna, λ is the signal wavelength, γ is the antenna polarization loss, and D a is the fire attack distance, D c It is the width of the range ring for effective and safe reconnaissance by the drone.

[0082] The vector distance d t The angle between the drone wing is defined as the behavior angle θ BHA , describing the alignment of the reconnaissance antenna to the radar:

[0083] θ BHA =deg(arc cos <v mov l t >)-90 (2),

[0084] In formula (2), v mov The unit vector representing the speed of the drone, θ BHA Projection on the xoy plane and yoz plane of the UAV platform coordinate system to obtain the angle |θ BHA | xoy and |θ BHA |yoz , n,t The judgment condition is rewritten as:

[0085]

[0086] The key to completing long-term continuous reconnaissance missions is to continuously intercept radar signals; i is the number of interception cycles, and its judgment condition is:

[0087]

[0088] In formula (4), N e represents the maximum number of steps allowed in each episode, Q represents the error of radar receiving and processing echoes; when the UAV successfully intercepts the radar signal, and the radar tracks the distance error of the UAV Q||d t ||Greater than the fire attack radius D a It is most beneficial for the drone to complete the mission.

[0089] From formulas (1) and (4), it can be concluded that the key to the UAV's reconnaissance mission is to control the relative vector distance d between it and the radar. t Therefore, the key to completing the UAV cognitive electronic reconnaissance mission is to let the UAV learn the vector distance d t Set Δ.

[0090] In an observation cycle time τ, the UAV platform is set to move with constant acceleration, then the vector distance d t The change in Δd t This is achieved by maneuvering the UAV platform. By controlling the acceleration vector a of the UAV platform at time t u,t Can control Δd t The key to the UAV cognitive electronic reconnaissance mission strategy is to optimize the acceleration a u,t , the objective function is:

[0091]

[0092] In formula (5), λ1, λ2, λ3, η1, η2, and η3 represent adjustment coefficients, and their values ​​are all greater than 0.

[0093] UAV cognitive electronic reconnaissance mission by controlling the acceleration a u,t To minimize the objective function J(a u,t ), a u,t Projected onto the azimuth plane and pitch plane of the UAV platform respectively, the azimuth acceleration a is obtained α,t and pitch acceleration a β,t ; The UAV cognitive electronic reconnaissance mission strategy is to optimize a α,t and a β,tThe value of is used to control the maneuverability of the UAV platform.

[0094] In the reconnaissance mission, the detection performance of the radar remains unchanged, and the performance of the drone reconnaissance payload remains unchanged. Only the reconnaissance area, mission difficulty, and drone maneuverability are changed, and Δ does not change. Therefore, it is feasible to migrate the policy network trained in Scouter to Scouter2 to accelerate learning.

[0095] The actual battlefield is very large, and the reconnaissance mission area after modeling is very large. It is very difficult to find an optimization method that is completely equivalent to the mission area. First, an overly large mission area will cause a surge in the state space of the reinforcement learning algorithm, and the sparsity of rewards and high-quality samples will become more obvious, increasing the difficulty of algorithm optimization and making it difficult for the algorithm to converge. Second, the modeling calculation of a mission area that is too large will take up a huge amount of computing resources. However, the computing power of the drone payload is limited, and a long calculation time will affect the timeliness of reconnaissance.

[0096] Although the mission area is large, the core area that needs to be reconnaissance in a reconnaissance operation is relatively sparse and concentrated. We divide a large reconnaissance mission area into multiple similar small areas for learning, and then use transfer learning and region splicing methods to enable the strategy to quickly match reconnaissance mission areas of any size, such as Figure 2 shown.

[0097] Figure 2 In the experiment, the mission area is divided into three small areas. One of the small areas is 75 km × 75 km. Deep reinforcement learning is performed to learn the reconnaissance strategy a. u,t ; In an observation cycle time τ, the reconnaissance strategy a u,t This will produce a vector distance d t The change in Δd t . Δd t It is determined by the UAV’s cognitive electronic reconnaissance strategy network and the UAV’s maneuverability, and has nothing to do with the size of the reconnaissance mission area; however, the UAV accurately observes the changes in the environment caused by each step of the reconnaissance strategy, which is conducive to the adjustment of the reconnaissance strategy. Strategy result Δd t In the reconnaissance mission area, the relative change δ t The relationship is shown in formula (6).

[0098]

[0099] In formula (6), X max , X min , Y max , Y min , H max , H min are the maximum and minimum values ​​of the length, width and height of the reconnaissance mission area; lt,x , l t,y , l t,z They are the distance vectors of the three coordinate axes. When the length, width, and height of the reconnaissance mission area change, they become X' max , X' min , Y' max , Y' min , H' max , H' min Then, substitute into formula (6), δ t The same reconnaissance strategy produces different results in different reconnaissance mission areas. Therefore, after the mission area changes, the originally trained strategy network will no longer be applicable. It is necessary to design a scaling factor according to the change of the mission area so that the UAV cognitive electronic reconnaissance strategy network can quickly adapt to reconnaissance mission areas of different sizes, as shown in formula (7):

[0100]

[0101] In formula (7), ρ1, ρ2, and ρ3 are the scaling factors of the three coordinate axes respectively.

[0102] The target domain module is used to set multiple Scouter2 target domains, and all small areas except the source domain are used as Scouter2 target domains. Deep reinforcement learning is performed on the Scouter2 target domains. The drone collects and processes basic information of multiple Scouter2 target domains, including but not limited to length, width, and height. The drone's reconnaissance antenna beam is aimed at the radar. According to the difference between the Scouter task interaction environment and the Scouter2 target domain information, the deep reinforcement learning algorithm is used for adaptation training. The drone control strategy trained in the source domain is transferred to the Scouter2 target domain. The difference between the source domain and the Scouter2 target domain is used to adjust the policy network input so that the deep reinforcement learning algorithm converges quickly. The difference adaptive transfer deep reinforcement learning algorithm is used to build a drone cognitive electronic reconnaissance strategy network; multiple trained Scouter2 target domains are spliced ​​into a reconnaissance mission area;

[0103] The UAV’s cognitive electronic reconnaissance mission is partly an observable Markov strategy process. The UAV does not know the exact deployment location and detection power range of the radar, and does not fully understand the environment of the reconnaissance area. It can only observe part of the environment through the reconnaissance payload it carries, so it is partially observable. In order to optimize the radar signal sorting and recognition results, the UAV needs to maintain effective reconnaissance for a long time, which makes the reconnaissance mission have a significant sequential characteristic and increases the difficulty of the reconnaissance strategy.

[0104] The partially observable Markov strategy process of the UAV cognitive electronic reconnaissance mission is described by a six-tuple <Σ,Ο,Α,F,R,γ>; Σ is the set of actual states of the UAV and radar Σ r is the radar position set, Σ u is the set of drone positions, is the set of UAV speeds, Π is the set of radar signal powers intercepted by the UAV, and Ο is the set of states observed by the UAV When the drone processes the intercepted radar signal, it will add a certain amount of system noise, and the observed radar state will deviate from the actual value. is the set of vector distances between the UAV and the radar with deviations, is the set of deviations between the unit vector of the drone’s line of sight and the radar’s line of sight and the unit vector of the drone’s velocity direction. is the biased radar signal power set intercepted by the UAV; Α is the UAV reconnaissance strategy action set {Α α ,Α β}, Α α is the set of azimuthal accelerations, Α β is the set of pitch accelerations; F is the state transfer function, R is the reward function, and γ is the discount factor; at time t, the real environment state s t =[s r,t ,s u,t ,v u,t ,P t ]∈Σ, the state observed by the drone After the drone control strategy network, output the execution strategy π and select action a t =[a α,t ,a β,t ]∈Α, with the transfer function Transfer to state s t+1 =[s r,t+1 ,s u,t+1 ,v u,t+1 ,P t+1 ]∈Σget the corresponding reward R(s t+1 ,a t ,s t ); During the reconnaissance mission time period T, the expected cumulative reward obtained by executing strategy π is used to evaluate its quality:

[0105]

[0106] Formula (8) is used to design the reward function of deep reinforcement learning, and the optimal execution strategy π is selected by maximizing the cumulative reward. * , so as to obtain J(a u,t )Minimal movement

[0107]

[0108] The purpose of transfer learning is to transfer s and learning tasks s The knowledge gained in is transferred to the target domain Δ of Scouter2 g , helping Scouter2 learn task Τ in the target domain g To achieve better learning results, transfer learning is introduced in the UAV cognitive electronic reconnaissance mission. The purpose is to transfer the knowledge learned by the UAV in the Scouter mission interaction environment to the Scouter2 target domain and improve the generalization performance of the algorithm. In the Scouter mission interaction environment, the strategy process of the UAV cognitive electronic reconnaissance mission is represented by a six-tuple <∑ sσ ,Ο sσ ,A sσ ,F sσ ,R sσ ,γ sσ > to describe. In Scouter2 target domain, the six-tuple <Σ gσ ,Ο gσ ,Α gσ ,F gσ ,R gσ ,γ gσ > to describe. s ×Α s The source domain Δ s , F s and R s Determines the learning task s According to whether the domain and learning task are the same before and after the change, the transfer deep reinforcement learning problem is divided into two cases: "the source domain is different from the target domain of Scouter2, but the learning task is the same" and "the source domain is the same as the target domain of Scouter2, but the learning task is different".

[0109] When the source domain is different from the target domain of Scouter2 and the learning task is the same, the source domain is different from the target domain of Scouter2 by Δ s ≠Δ g , but the learning task is the same s =T g The displacement of the drone in three-dimensional space is achieved by adjusting the azimuth acceleration and pitch acceleration. When the spatial dimension remains unchanged, the action space will not change. Assuming that the Scouter task interaction environment and Scouter2 target domain are both three-dimensional spaces, s =Α g , Scouter2 target domain changes by Σ s ≠Σ g cause.

[0110] The source domain is denoted as The target domain of Scouter2 is denoted as When the length and width of the drone's reconnaissance area change, the drone's three-dimensional position Σ su and radar 3D position Σ sr The quantity and value will change, resulting in Σ su ≠Σ gu ,Σ sr ≠Σ gr , destroying the mapping relationship between observation state and output action in the UAV cognitive electronic reconnaissance strategy network. Obviously, if the policy network of the source domain is directly transferred to the Scouter2 target domain, the UAV needs to spend time learning the changes in the reconnaissance area and adjusting the mapping relationship between observation state and output action. If the difference between the source domain and the Scouter2 target domain can be used to reduce the impact on the UAV's observation state, learning time will be saved. In the Scouter2 target domain learning, the transfer deep reinforcement learning algorithm learns the strategy πt at time t, and the gap between its value function and the optimal strategy π* can be expressed as:

[0111]

[0112] In formula (A), is the approximation error, which is caused by the approximation value of the function. As the number of learning times increases, the approximation error will gradually decrease and approach the optimal strategy function value. is the estimation error, using a finite sample N t The value function of is estimated. As the number of samples in the target domain increases, the estimation error decreases and converges. ε is the optimization error caused by the function approximating the non-global optimal strategy.

[0113] In the Scouter2 target domain, after migrating the policy network, due to the existence of the above three errors, it is necessary to use deep reinforcement learning methods to reduce the errors. Reducing the gap in drone observation status between the Scouter2 target domain and the source domain is conducive to improving the utilization efficiency of deep reinforcement learning, obtaining high-quality samples, and reducing approximation errors. This will quickly start learning. As the number of high-quality samples increases, the estimation error High-quality samples are also helpful in guiding deep reinforcement learning to fully "explore" in the target domain, further enriching samples and reducing the optimization error ε.

[0114] When the source domain is the same as the Scouter2 target domain, the learning tasks are different, and the target domain is the same Δ s =Δ g , learning tasks are different s ≠Τ gThere are two cases: one is that the reconnaissance mission cycle changes, and the UAV maneuverability remains unchanged. In this case, the reward function changes R s ≠R g , the transfer function remains unchanged F s =F g Second, the UAV maneuvering speed range changes, and the reconnaissance mission cycle remains unchanged. At this time, the transfer function changes F s ≠F g , the reward function remains unchanged R s =R g .

[0115] For the first case, “the reward function changes, but the transfer function remains unchanged”, in the Scouter2 target domain, when the reconnaissance mission cycle changes, the standard M in formula (4) for measuring whether the drone completes the mission i will change, resulting in a change in the reward function. Although the high-quality actions selected by the policy network can complete the task in the source domain, they cannot obtain higher rewards in the target domain, resulting in mission failure. If the drone can observe the decisive parameters that affect the completion criteria of the reconnaissance mission, it will be beneficial to the fitting of the deep neural network and accelerate the convergence of the algorithm.

[0116] For the second case, “the transfer function changes, but the reward function remains unchanged”, in the Scouter2 target domain, when the UAV maneuvering speed range changes, the vector distance d between the UAV and the radar t The change in Δd t will change, resulting in a change in the transfer function to the next state. At this time, under the same strategy, the state transfer between the source domain and the Scouter2 target domain will be different. After the optimal strategy in the source domain is transferred to the Scouter2 target domain, it is no longer the optimal strategy, and the three errors in formula (A) will be generated. If the drone can observe the impact of its maneuvering speed on the task, it will be beneficial to the convergence of the algorithm.

[0117] Deep reinforcement learning is a differential adaptive transfer deep reinforcement learning algorithm. Deep reinforcement learning includes reward modeling and differential adaptive normalized observation state. The reward function formula is:

[0118]

[0119] In formula (10), is the vector distance between the UAV and the radar with observation error; |θ BHA | xoy,t and |θ BHA | yoz,t is the time t that θ BHA The angles are obtained by projecting on the xoy plane and yoz plane of the UAV platform coordinate system respectively.

[0120] The key to the UAV's completion of cognitive electronic reconnaissance missions is to learn the vector distance set Δ. However, the vector distance set Δ is an abstract feature fitted by a deep neural network, which is expressed as a mapping relationship between input variables and output strategies. When the reconnaissance mission area and mission environment change, the input variables change accordingly, resulting in deviations in the output strategy. Therefore, if the policy network is directly migrated to the Scouter2 target domain, the UAV cannot achieve a high mission completion rate or even cannot complete the mission, and a deep reinforcement learning algorithm is required for adaptation training. Although deep neural networks have strong representation capabilities, the fitting process takes time. The large difference in input variables between the Scouter2 target domain and the source domain will inevitably lead to increased learning costs, poor convergence and other effects. The scope of the reconnaissance area and the speed range of the UAV are generally determined before the mission is executed, and can be regarded as known information in the target domain. Use this known information to derive the gap between the target domain and the source domain, and adjust the input variables of the policy network. By reducing the difference in input variables between the target domain and the source domain, the fitting process of the deep neural network is accelerated and learning time is saved. The deep reinforcement learning network framework for differential adaptive migration is as follows: Figure 3 shown.

[0121] In the interactive environment of Scouter mission, the reward function is set from three aspects: distance, angle, and power. In Scouter2, the mission area, UAV maneuvering speed range, and mission difficulty will change, but the essence of the reconnaissance mission has not changed, and the focus is still on the three aspects of distance, angle, and power. The reward function is shown in formula (B).

[0122]

[0123] In formula (11), is the vector distance between the UAV and the radar with observation error. |θ BHA | xoy,t and |θ BHA | yoz,t is the time t that θ BHA The angles are obtained by projecting on the xoy plane and yoz plane of the UAV platform coordinate system respectively.

[0124] The variables that have a direct impact on the UAV's cognitive electronic reconnaissance mission are selected as the input of the policy network. The observation state of the UAV includes five sets The drone locations are Drone speed Vector distance between drone and radar Deviation between the drone's speed direction and the radar's line of sight direction Intercepted radar signal power The observation state set contains thirteen-dimensional vectors, which have different dimensions and different sparsity, so the degree of numerical change in one strategy time is also different. The thirteen-dimensional vectors are normalized to obtain the difference adaptive normalized observation state at time t

[0125] The difference adaptive normalization observation states include the change of the mission area range of Scouter2 target domain and source domain, the change of the UAV speed range of target domain and source domain, and the change of the reconnaissance mission difficulty of target domain and source domain;

[0126] Among them, the changes in the task area range between the target domain and the source domain are mainly the changes in the length and width of the task area: the height of the drone reconnaissance area mainly depends on the drone ceiling. When the drone ceiling remains unchanged, the height of the reconnaissance area remains unchanged. The difference between the target domain and the source domain task area of ​​Scouter2 is used to adjust the observation state, and the observation state of the task area difference adaptive normalization is obtained.

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133] In formula (11), Δx = X max -X min ,Δy=Y max -Y min ,Δh=H max -H min ,Δv=V max -V min .X max , X min , Y max , Y min , H max , H min are the maximum and minimum values ​​of the length, width, and height of the task area; V max , V min are the maximum and minimum values ​​of the UAV maneuvering speed respectively; v mov,x , v mov,y , v mov,z are the projections of the UAV speed direction vector on the three coordinates; l t,x , lt,y , l t,z are the distance vectors of the three coordinate axes; s,t is the radar signal power received by the reconnaissance payload, P u,min is the sensitivity of the reconnaissance payload, Δx'=X' max -X' min ,Δy'=Y' max -Y' min , X' max , X' min , Y' max , Y' min They are the maximum and minimum values ​​of the length, width, and height of the task area in the target domain of Scouter2, respectively;

[0134] The speed range of the drone in the target domain and the source domain of Scouter2 changes. When the speed range of the drone changes, the change in the vector distance between the drone and the radar in the target domain and the source domain will change, causing the transfer function of the next state to change, making the policy network migrated to the target domain inapplicable; the difference in the speed range of the drone in the target domain and the source domain is used to adjust the observation state, and the observation state with adaptive normalization of the speed difference is obtained.

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141] In formula (12), Δv'=V' max -V' min The maximum value of the UAV speed in the Scouter2 target domain is V' max , the minimum value is V' min ;

[0142] The relationship between the difficulty of reconnaissance tasks in the target domain and source domain of Scouter2. In the source domain, the criterion for the drone to complete the reconnaissance task is that the number of reconnaissance cycles of continuously intercepting radar signals reaches a certain value. The difficulty of reconnaissance tasks in the target domain of Scouter2 can be changed by changing the criterion for the drone to complete the reconnaissance task, that is, the number of reconnaissance cycles. The increase in the number of reconnaissance cycles means the increase in the difficulty of the task.

[0143] From formula (2), we can see that the angle between the drone speed direction and the radar line of sight direction is arccos <v mov l t >, which is closely related to the behavior angle. Therefore, the difference between the speed direction of the drone and the radar line of sight is observed. Will help fit the angle reward. Also, observe the vector distance between the drone and the radar Radar signal power intercepted by the drone They help fit the distance and power rewards, respectively. Since the precise position of the radar is not known in advance, the observation of the vector distance between the drone and the radar It is error-prone and affects the reward function fitting. Therefore, the precise position of the drone needs to be added to the observation state. This is helpful for neural network fitting. The length of the drone flight time is not limited in the reconnaissance mission, so the change in the drone speed value has no direct impact on the mission completion criterion. In order to reduce the input dimension and accelerate convergence, the observation state does not consider the drone speed; at this time, the observation state formula (13) is:

[0144]

[0145]

[0146]

[0147]

[0148]

[0149] Network structure:

[0150] The DDPG, TD3, and SAC algorithms are transferred and learned to obtain three difference adaptive transfer deep reinforcement learning algorithms TL-DDPG, TL-TD3, and TL-SAC, and the hyper parameters of the algorithms are shown in Table 1. In Table 1, 2×3FC-(13 / 10,256,256,2) means that there are two networks, each network consists of three fully connected (FC) layers, thirteen or ten input units, two hundred and fifty-six connection units, two hundred and fifty-six connection units, and two output units.

[0151] Table1.Detailed network configurations for the three algorithms

[0152]

[0153]

[0154] A simulation experiment was conducted to test the learning effect of the deep reinforcement learning algorithm with differential adaptive transfer by changing the task area range, UAV maneuvering speed range, and task difficulty in the Scouter task interaction environment.

[0155] In order to save pre-training time, the task area in the Scouter task interaction environment should not be too large, and the task difficulty should not be too high. Therefore, the length, width, and height of the task area are set to 75km×75km×6km, the task difficulty is 200 observation cycles, and the maximum speed of the drone is 250m / s. When the task area, task difficulty, and drone speed range in the Scouter task interaction environment are changed, the state space, reward function, and state transition probability of the deep reinforcement learning algorithm for differential adaptive migration will be affected respectively. In order to test the convergence ability of the TL-DDPG, TL-TD3, and TL-SAC algorithms, the changes in the Scouter2 target domain environment are divided into three cases:

[0156] The first case: the mission area changes. The length and width of the mission area in the Scouter2 target domain are set to 50km×50km, 100km×100km, 125km×125km, and 150km×150km, respectively, corresponding to scenarios A1, A2, A3, and A4. The height range of the reconnaissance area mainly depends on the UAV's ceiling. The UAV's maneuverability is set unchanged, so the height range remains unchanged, and only the changes in the length and width of the mission area are considered.

[0157] The second case: the speed range of the drone changes. The speed range of the drone is the difference between the maximum speed and the minimum speed. For fixed-wing drones, the minimum speed is generally the critical speed to ensure that the drone does not stall. The minimum speed is set to 50m / s. Therefore, the change in the speed range of the drone is mainly caused by the change in the maximum speed. The maximum speed of the drone in the Scouter2 target domain is set to 150m / s, 300m / s, 350m / s, and 400m / s, corresponding to scenarios B1, B2, B3, and B4.

[0158] The third case: the mission difficulty changes. The difficulty of the reconnaissance mission is mainly determined by the continuous reconnaissance cycle M of the UAV to the radar. i OK, M i The larger the value, the more difficult the task is. The reconnaissance period in the target domain of Scouter2 is set to 150, 250, 300, and 400, corresponding to scenarios C1, C2, C3, and C4.

[0159] Other environment configuration parameters in the Scouter2 target domain are shown in Table 2.

[0160] Table 2 Other configuration parameters of Scouter2 environment

[0161]

[0162] One reconnaissance mission corresponds to one episode executed by the drone. The learning is set to 5,000 episodes, and the maximum number of steps in each episode is 1,000. The criteria for mission termination are divided into three categories:

[0163] Category 1: Determine if the target is in the feature library and continuously capture the target for a period of M i If the mission difficulty requirements are met, the drone wins;

[0164] Category 2: The target is judged to be in the feature library, and the tracking error of the radar tracking the drone is less than 500m, which meets the strike conditions, and the radar wins;

[0165] Category 3: Neither of the above two events occurs until the maximum number of steps is reached and the episode ends.

[0166] The status information of the UAV and radar, such as position and velocity, is reinitialized before each episode begins.

[0167] The evaluation metrics are as follows:

[0168] (1) Percentage of Mission Completion (PMC)

[0169] One thousand episodes were tested, and the percentage of episodes in which drones successfully completed their missions was counted to evaluate the effectiveness of the confrontation between drones and radars.

[0170] (2) Reward

[0171] Set 5,000 episodes, count the reward values ​​for each episode, and evaluate the convergence of the drone during training. There may be multiple local optimal solutions in the continuous state space. In order to prevent large jitters in the reward value of each episode, the average method of every 100 episodes is used for display.

[0172] (3) Behavior Angle (BHA)

[0173] Count the θ of each episode BHA , evaluates the alignment of the reconnaissance antenna to the radar. In order to prevent the θ BHA There is a large jitter, so the average of every 100 episodes is used to display the data.

[0174] (4) Mean Decision Travel (MDV)

[0175] Statistics on the average strategic flight range during drone training can reflect the drone’s fuel consumption and algorithm optimization capabilities.

[0176] (5) Mean Decision Time (MDT)

[0177] Statistics on the average strategy time during drone training can reflect the drone’s strategy speed and computational complexity.

[0178] (6) Training Time (TT)

[0179] Statistics on the training time of drones can reflect the computational complexity and algorithm convergence speed.

[0180] The simulation results include the following sections:

[0181] Simulation results of different task areas: The simulation results of different task areas in Scouter2 target domain are shown in Table 3. The reward results are shown in Figure 4 As shown, the behavior angle results are as follows Figure 5 shown.

[0182] Table 3 Simulation results of Scouter2 in different task areas

[0183]

[0184] Note: DDPG * 、TD3 * 、SAC * The algorithm only migrates the policy network trained in Scouter without training. DDPG, TD3, and SAC algorithms are trained from scratch without migrating the policy network. TL-DDPG, TL-TD3, and TL-SAC algorithms both migrate the policy network and perform training. The boldface in the table only indicates the optimal value, and the boldface in brackets indicates the change in value compared with the corresponding deep reinforcement learning algorithm.

[0185] From Table 3, Figure 4 and Figure 5 It can be seen that the TL-DDPG, TL-TD3 and TL-SAC algorithms have obvious advantages in fast startup and optimized convergence results compared with the corresponding deep reinforcement learning algorithms. The closer the target domain of Scouter2 is to the interactive environment of the Scouter task, the higher the PMC value after applying transfer learning. The greater the difference between the target domain of Scouter2 and the interactive environment of the Scouter task, the more obvious the generalization advantage of transfer learning. Among them, the PMC value of the TL-SAC algorithm is relatively more stable and can complete the task better.

[0186] The MDV value increases with the expansion of the mission area, indicating that the flight range of the UAV increases within a strategy time. However, the MDV values ​​of the three differential adaptive transfer reinforcement learning algorithms are generally lower than those of the three reinforcement learning algorithms, indicating that the application of transfer learning can reduce the range of invalid UAV flights and significantly reduce the training time TT value.

[0187] When the difference between Scouter2 and Scouter increases, the MDT values ​​of the three difference adaptive transfer reinforcement learning algorithms are higher than those of the three reinforcement learning algorithms, indicating that the difference adaptive transfer reinforcement learning algorithm has high complexity.

[0188] Analysis of behavior in different task areas:

[0189] When the reconnaissance mission area is small (50km×50km), the distance between the UAV and the radar is relatively close. The UAV needs to quickly adjust its attitude in a short period of time to ensure that the reconnaissance mission area coverage meets the mission requirements. Therefore, the UAV reciprocates more often in a relatively small angle range centered on the radar. When the UAV flies in a circle outside the radar detection power, the edge of the trajectory fluctuates greatly due to the need to constantly adjust its attitude in a short period of time, such as Figure 6 , Figure 7 As shown, Figure 6 The first row shows the drone flying back and forth outside the radar detection range in episode 4386. (a) to (d) correspond to steps 84, 116, 172, and 261 respectively. The second row shows the drone flying in a circle outside the radar detection range in episode 4552. (e) to (h) correspond to steps 75, 108, 196, and 259 respectively. Figure 7 , (a) to (d) correspond to steps 110, 317, 515, and 693, respectively.

[0190] When the reconnaissance mission area is large (150km×150km), the distance between the UAV and the radar is relatively long. The UAV has sufficient time to adjust its attitude when approaching the radar, so the UAV often flies around the radar detection power, and the edge of the trajectory has small fluctuations. However, due to the long distance between the UAV and the radar, the observation error is large, which will cause the UAV to search for radar signals for a long time.

[0191] Simulation results of different maximum speeds of drones: The simulation results of different maximum speeds of drones in the Scouter2 target domain are shown in Table 4. The reward convergence is as follows Figure 8 As shown, the behavior angle converges as Fig. 9 As shown, Figure 8 The first row is DDPG and TL-DDPG algorithms, the second row is TD3 and TL-TD3 algorithms, and the third row is SAC and TL-SAC algorithms. Fig. 9The first row is DDPG and TL-DDPG algorithms, the second row is TD3 and TL-TD3 algorithms, and the third row is SAC and TL-SAC algorithms.

[0192] Table 4 Simulation results of Scouter2 UAV at different maximum speeds

[0193]

[0194]

[0195] Note: DDPG * 、TD3 * 、SAC * The algorithm only migrates the policy network trained in Scouter without training. The DDPG, TD3, and SAC algorithms are trained from scratch without migrating the policy network. The TL-DDPG, TL-TD3, and TL-SAC algorithms both migrate the policy network and perform training.

[0196] From Table 4, Figure 8 and Fig. 9 It can be seen that the TL-DDPG, TL-TD3 and TL-SAC algorithms have obvious advantages in fast startup and optimized convergence results compared with the corresponding deep reinforcement learning algorithms. When the drone speed is high (≥300m / s), the reward value is high and the behavior angle can converge to a smaller angle. When the drone speed is low (≤150m / s), the convergence performance of the reward and behavior angle will deteriorate. This shows that a high drone speed is more beneficial to completing the reconnaissance mission. Among them, the PMC value of TL-TD3 is relatively more stable and the task is better completed.

[0197] The MDV value increases with the increase of the maximum speed of the UAV, indicating that the flight range of the UAV increases within a strategy time. However, the MDV values ​​of the three differential adaptive transfer reinforcement learning algorithms are generally lower than those of the three reinforcement learning algorithms, indicating that the application of transfer learning can reduce the range of invalid flight of the UAV and reduce the training time TT value.

[0198] The MDT values ​​of the three difference adaptive transfer reinforcement learning algorithms are higher than those of the three reinforcement learning algorithms, indicating that the difference adaptive transfer reinforcement learning algorithms are highly complex.

[0199] Analysis of the drone's behavior at different maximum speeds:

[0200] When the drone is moving at a slower speed, it is easier to adjust its posture and evade radar detection flexibly. Fig.10As shown in the figure, (a) to (e) correspond to steps 510, 529, 540, 567, and 712, respectively. At step 529, the drone senses that it may be detected by the radar (from a bird's eye view, the blue line in the figure indicates that the radar has detected the drone). In order to escape radar detection, the drone quickly turns its nose and flies to a safe area. After the drone flies to a safe distance, it continues to monitor the radar until the reconnaissance mission is completed.

[0201] Combination Fig.11 , (a) to (e) correspond to steps 156, 416, 877, and 986, respectively. When the drone is moving at a high speed, the strategy time becomes mismatched, causing the drone to fly more invalid distances. In order to adapt to the strategy at the next moment, the drone needs to adjust its attitude, resulting in frequent attitude adjustments. The excessive speed and frequent attitude adjustments make it difficult for the drone to complete 200 cycles of reconnaissance missions at one time. It can only repeat the reconnaissance mission continuously, reducing the efficiency of mission completion.

[0202] The simulation results of different task difficulties are as follows: The simulation results of different task difficulties in the Scouter2 target domain are shown in Table 5, and the reward results are as follows Fig.12 As shown, the first row is DDPG and TL-DDPG algorithms, the second row is TD3 and TL-TD3 algorithms, and the third row is SAC and TL-SAC algorithms; the behavior angle results are shown in Fig.13 As shown, the first row is DDPG and TL-DDPG algorithms, the second row is TD3 and TL-TD3 algorithms, and the third row is SAC and TL-SAC algorithms.

[0203] Table 5 Simulation results of different task difficulties in Scouter2

[0204]

[0205] Note: DDPG * 、TD3 * 、SAC * The algorithm only migrates the policy network trained in Scouter without training. The DDPG, TD3, and SAC algorithms are trained from scratch without migrating the policy network. The TL-DDPG, TL-TD3, and TL-SAC algorithms both migrate the policy network and perform training.

[0206] From Table 5, Fig.12 and Fig.13It can be seen that the TL-DDPG, TL-TD3 and TL-SAC algorithms have obvious advantages in fast startup and optimized convergence results compared with the corresponding deep reinforcement learning algorithms. Among them, the PMC values ​​of the TL-TD3 and TL-SAC algorithms are both above 96%, and the convergence performance is good and stable. The closer the number of reconnaissance cycles in Scouter2 is to the number of reconnaissance cycles in Scouter, the better the quality of the drone's mission completion.

[0207] The MDV value increases with the increase of the reconnaissance cycle, indicating that the UAV's flight range has increased within a strategy time. The three differential adaptive transfer reinforcement learning algorithms generally have higher MDV values ​​than the three reinforcement learning algorithms. This is because transfer learning improves the task completion rate and increases the reconnaissance cycle, resulting in an increase in the UAV's flight range.

[0208] Compared with the three reinforcement learning algorithms, the MDT values ​​of the three difference adaptive transfer reinforcement learning algorithms are higher, indicating that the complexity of the transfer learning algorithm is high. However, the TT values ​​are lower, indicating that the application of transfer learning can reduce the training time.

[0209] Analysis of behaviors of different task difficulties: The cognitive reconnaissance behaviors of drones can basically be divided into two categories: one is that when the drone approaches the radar at a long distance, there is sufficient time to adjust the flight attitude. Therefore, it is easy to meet the reconnaissance antenna coverage requirements, and the drone tends to fly around in a large range outside the radar detection power; the other is that when the drone approaches the radar at a close distance, the time to adjust the flight attitude is very short. In order to meet the reconnaissance antenna coverage requirements, the drone frequently adjusts the flight attitude within a small range, so the drone tends to fly back and forth in a small range outside the radar detection power. In addition, the length of the drone's track flying around the radar is different when the reconnaissance cycle is different. Although the reward function has changed, the reward design has not changed, and the goal drive is consistent, so the reconnaissance behavior learned by the drone will not change significantly, which is consistent with our previous research results.

[0210] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A UAV cognitive electronic reconnaissance system based on scene segmentation, comprising a UAV, characterized in that: Also included is a reconnaissance mission area and a UAV learning module; the UAV's reconnaissance antenna beam is aimed at the radar; The reconnaissance mission area includes a source domain module and a target domain module, and the reconnaissance mission area is divided into a plurality of similar small areas; The source domain module is used to set the Scouter task interaction environment, select one of the small areas as the Scouter task interaction environment, and the drone collects and processes basic information of the Scouter task interaction environment, including but not limited to length, width, and height; the drone uses the deep reinforcement learning algorithm to generate a drone control strategy, and uses the Scouter task interaction environment as the source domain; The drone learning module includes a deep learning algorithm and a differential adaptive transfer deep reinforcement learning algorithm; The target domain module is used to set multiple Scouter2 target domains, and all the small areas except the source domain are used as Scouter2 target domains respectively, and deep reinforcement learning is performed on the Scouter2 target domains. The drone collects and processes basic information of multiple Scouter2 target domains, including but not limited to length, width, and height. According to the difference between the Scouter task interaction environment and the Scouter2 target domain information, a deep reinforcement learning algorithm is used to perform adaptation training, and the drone control strategy trained in the source domain is transferred to the Scouter2 target domain. The difference between the source domain and the Scouter2 target domain is used to adjust the policy network input, so that the deep reinforcement learning algorithm converges quickly, and the difference adaptive transfer deep reinforcement learning algorithm is used to build a drone cognitive electronic reconnaissance strategy network; Multiple trained Scouter2 target domains are spliced ​​into a reconnaissance mission area.

2. The UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 1 is characterized in that: In the Scouter mission interaction environment, the radar is responsible for searching for aerial targets and locking and destroying threatening targets. The radar signal in the Scouter mission interaction environment area is detected by using the UAV platform carrying the reconnaissance payload; the radar detection power range is approximately the firepower strike range, and the radius is set to D a , the area where the UAV can intercept radar signals and fly safely, the width is set to D c , the two sides of the drone fuselage represent the coverage of the reconnaissance antenna, the azimuth angle is set to α, the pitch angle is set to β, the azimuth angle α and the pitch angle β of the drone's antenna main lobe beam cover the radar, and the radar signal power P received by the drone's reconnaissance payload t .

3. The UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 2 is characterized in that: When the azimuth angle α and the pitch angle β of the main lobe beam of the UAV antenna cover the radar, the radar signal power P received by the UAV reconnaissance payload t When the sensitivity is exceeded, the radar signal can be intercepted by the drone reconnaissance payload, which is the reconnaissance flag position. n,t , the judgment conditions are: In formula (1), d t is the three-dimensional spatial position of the radar in the geocentric coordinate system s r,t =[x r,t ,y r,t ,z r,t ] and the three-dimensional spatial position of the drone u,t =[x u,t ,y u,t ,z u,t ], denoted as d t =s r,t -s u,t ; d t Projected to the UAV platform coordinate system to obtain l t , and then l t Project onto the three coordinate axes to obtain the distance vector l t,x , l t,y , l t,z ;P r,t is the radar transmission power, G r,t is the radar antenna gain, G u,t is the gain of the drone reconnaissance antenna, λ is the signal wavelength, γ is the antenna polarization loss, and D a is the fire attack distance, D c is the width of the range ring for effective and safe reconnaissance by the drone; The vector distance d t The angle between the drone wing is defined as the behavior angle θ BHA , describing the alignment of the reconnaissance antenna to the radar: θ BHA =deg(arcs <v mov l t >)-90 (2), In formula (2), v mov The unit vector representing the speed of the drone, θ BHA Projection on the xoy plane and yoz plane of the UAV platform coordinate system respectively obtains the angle |θ BHA | xoy and |θ BHA | yoz , n,t The judgment condition is rewritten as: The key to completing long-term continuous reconnaissance missions is to continuously intercept radar signals; i is the number of interception cycles, and its judgment condition is: In formula (4), N e represents the maximum number of steps allowed in each episode, Q represents the error in receiving and processing echoes by the radar; According to formulas (1) and (4), the key to completing the UAV cognitive electronic reconnaissance mission is to let the UAV learn the vector distance d t Set Δ; In an observation cycle time τ, the UAV platform is set to move with constant acceleration, then Vector distance d t The change in Δd t This is achieved by maneuvering the UAV platform; by controlling the acceleration vector a of the UAV platform at time t u,t Control Δd t The key to the UAV cognitive electronic reconnaissance mission strategy is to optimize the acceleration a u,t , the objective function is: In formula (5), λ1, λ2, λ3, η1, η2, η3 represent adjustment coefficients, and their values ​​are all greater than 0; UAV cognitive electronic reconnaissance mission by controlling the acceleration a u,t To minimize the objective function J(a u,t ); u,t Projected onto the azimuth plane and pitch plane of the UAV platform respectively, the azimuth acceleration a is obtained α,t and pitch acceleration a β,t ; The UAV cognitive electronic reconnaissance mission strategy is to optimize a α,t and a β,t The value of is used to control the maneuverability of the UAV platform.

4. The UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 3 is characterized in that: There are three small areas, one of which is 75km×75km in size, and deep reinforcement learning is performed to learn the reconnaissance strategy a. u,t ; In an observation cycle time τ, the reconnaissance strategy a u, t will produce the vector distance d t The change in Δd t ; Δd t It is determined by the UAV cognitive electronic reconnaissance strategy network and the UAV maneuverability, and has nothing to do with the size of the reconnaissance mission area; Strategy result Δd t In the reconnaissance mission area, the relative change δ t The relationship is shown in formula (6): In formula (6), X max , X min , Y max , Y min , H max , H min are the maximum and minimum values ​​of the length, width and height of the reconnaissance mission area; l t,x , l t,y , l t,z They are the distance vectors of the three coordinate axes. When the length, width and height of the reconnaissance mission area change, becomes X' max , X' min , Y' max , Y' min , H' max , H' min Then, substitute into formula (6), δ t The same reconnaissance strategy produces different results in different reconnaissance mission areas. The scaling factor is designed according to the change of the mission area, so that the UAV cognitive electronic reconnaissance strategy network can quickly adapt to the reconnaissance mission areas of different sizes, as shown in formula (7): In formula (7), ρ1, ρ2, and ρ3 are the scaling factors of the three coordinate axes respectively.

5. The UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 2 is characterized in that: The part of the UAV cognitive electronic reconnaissance mission is an observable Markov strategy process. The part of the observable Markov strategy process of the UAV cognitive electronic reconnaissance mission is described by a six-tuple <∑, Ο, A, F, R, γ>; □ is the set of actual states of the UAV and radar Σ r is the radar position set, Σ r is the set of drone positions, is the set of UAV speeds, Π is the set of radar signal powers intercepted by the UAV, and Ο is the set of states observed by the UAV When the drone processes the intercepted radar signal, it will add a certain amount of system noise, and the observed radar state will deviate from the actual value. is the set of vector distances between the UAV and the radar with deviations, is the set of deviations between the unit vector of the drone’s line of sight and the radar’s line of sight and the unit vector of the drone’s velocity direction. is the biased radar signal power set intercepted by the UAV; A is the UAV reconnaissance strategy action set A α is the set of azimuthal accelerations, A β is the set of pitch accelerations; F is the state transfer function, R is the reward function, γ is the discount factor; at time t, the real environment state Status observed by drone After the UAV control strategy network, the execution strategy π is output and the action is selected Transfer function Transfer to state Get the corresponding reward R(s t+1 ,a t ,s t );During the reconnaissance mission time period T, the expected cumulative reward obtained by the execution strategy π is used to evaluate its quality: Formula (8) is used to design the reward function of deep reinforcement learning, and the optimal execution strategy π is selected by maximizing the cumulative reward. * , so as to obtain J(a u,t )Minimal movement 6. The UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 5 is characterized in that: The purpose of transfer learning is to transfer the s and learning task T s The knowledge gained in is transferred to the target domain Δ of Scouter2 s Help Scouter2 learn task T in the target domain g To obtain better learning effect; in the Scouter task interaction environment, the strategy process of UAV cognitive electronic reconnaissance task is expressed by six-tuple ∑ sσ ,Ο sσ ,A sσ ,F sσ ,R sσ ,γ sσ > to describe; in Scouter2 target domain, six tuples ∑gσ, Ο gσ ,A gσ ,F gσ ,R gσ ,γ gσ > to describe;∑ s *A s The source domain Δ s , F s and R s Determines the learning task T s The goal of deep reinforcement learning is to divide the transfer problem into two cases: "the source domain and the target domain of Scouter2 are different, but the learning tasks are the same" and "the source domain and the target domain of Scouter2 are the same, but the learning tasks are different".

7. The UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 6 is characterized in that: The deep reinforcement learning is a difference adaptive transfer deep reinforcement learning algorithm, which includes reward modeling and difference adaptive normalized observation state, wherein the reward function formula is: In formula (10), is the vector distance between the UAV and the radar with observation error; |θ BHA | xoy,t and |θ BHA | yoz,t is the time t that θ BHA Project the angles on the xoy plane and yoz plane of the UAV platform coordinate system respectively; The variables that have a direct impact on the UAV’s cognitive electronic reconnaissance mission are selected as the input of the policy network. The observation state of the UAV includes five sets: The drone locations are Drone speed Vector distance between drone and radar Deviation between the drone's speed direction and the radar's line of sight direction Intercepted radar signal power The observation state set contains thirteen-dimensional vectors, which have different dimensions and different sparsity, so the degree of numerical change in one strategy time is also different. The thirteen-dimensional vectors are normalized to obtain the difference adaptive normalized observation state at time t 8. The UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 7 is characterized in that: The difference adaptive normalized observation state includes the change of the mission area range of the target domain and the source domain, the change of the speed range of the drone of the target domain and the source domain, and the change of the difficulty of the reconnaissance mission of the target domain and the source domain; Among them, the changes in the task area range of the target domain and the source domain are mainly the changes in the length and width of the task area: the height of the drone reconnaissance area mainly depends on the drone ceiling. When the drone ceiling remains unchanged, the height of the reconnaissance area remains unchanged. The difference between the target domain and the source domain task area is used to adjust the observation state, and the observation state of the task area difference adaptive normalization is obtained. In formula (11), Δx = X max -X min ,Δy=Y max -Y min ,Δh=H max -H min ,Δv=V max -V min .X max , X min , Y max , Y min , H max , H min are the maximum and minimum values ​​of the length, width, and height of the task area; V max , V min are the maximum and minimum values ​​of the UAV maneuvering speed respectively; v mov,x , v mov,y , v mov,z are the projections of the UAV speed direction vector on the three coordinates; l t,x , l t,y , l t,z are the distance vectors of the three coordinate axes; s,t is the radar signal power received by the reconnaissance payload, P u,min is the sensitivity of the reconnaissance payload, Δx'=X' max -X' min ,Δy' ＝ Y' max -Y' min , X' max , X' min , Y' max , Y' min are the maximum and minimum values ​​of the length, width, and height of the task area in the target domain, respectively; The speed range of the drone in the Scouter2 target domain and the source domain changes. When the speed range of the drone changes, the change in the vector distance between the drone and the radar in the Scouter2 target domain and the source domain will change, causing the transfer function of the next state to change, making the policy network migrated to the Scouter2 target domain inapplicable; the difference in the speed range of the drone in the Scouter2 target domain and the source domain is used to adjust the observation state, and the observation state with adaptive normalization of the speed difference is obtained. In formula (12), Δv'=V' max -V' min , the maximum value of the UAV speed in the Scouter2 target domain is V' max , the minimum value is V' min ; The difficulty of the reconnaissance mission of Scouter2 target domain and source domain varies. The length of the drone flight time is not limited in the reconnaissance mission, so the change of the drone speed value has no direct impact on the task completion criterion. In order to reduce the input dimension and accelerate convergence, the observation state does not consider the drone speed; at this time, the observation state formula (13) is:

9. The UAV cognitive electronic reconnaissance system based on scene segmentation according to claim 8 is characterized in that: It also includes a simulation experiment module, which sets up three groups of experiments to test the learning effect of the deep reinforcement learning algorithm of difference adaptive transfer by changing the task area range, the UAV maneuvering speed range, and the task difficulty in the Scouter task interaction environment.

Images