Unmanned aerial vehicle multi-target cognitive electronic reconnaissance system

Abstract

The invention relates to the technical field of unmanned aerial vehicle reconnaissance systems, in particular to an unmanned aerial vehicle multi-target cognitive electronic reconnaissance system which comprises an unmanned aerial vehicle platform, an unmanned aerial vehicle, a ScurterN task simulation module used for intercepting radar signals and a simulator used for modeling unmanned aerial vehicle multi-target cognitive electronic reconnaissance task scenes. The radar is arranged in the reconnaissance task area, a plurality of pseudo target modules are arranged in the reconnaissance task area, a plurality of pseudo targets are obtained from the position of the radar according to a nearest neighbor method, the arc centers of the plurality of pseudo targets are solved and are not updated and iterated continuously, and finally a pseudo target is formed; and the unmanned aerial vehicle multi-target cognitive electronic reconnaissance strategy learning module is used for convergence of a radar reconnaissance strategy and building an unmanned aerial vehicle multi-target cognitive electronic reconnaissance strategy network. According to the method, convergence of more than four radar reconnaissance strategies can be simplified, and the convergence speed of the more than four radar reconnaissance strategies is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) reconnaissance systems, and in particular to an unmanned aerial vehicle (UAV) multi-target cognitive electronic reconnaissance system. Background Art

[0002] UAV cognitive electronic reconnaissance is used to guide unmanned platforms to accurately perceive the electromagnetic situation on the battlefield, thereby guiding subsequent electronic interference and anti-radiation strikes. Using UAV platforms equipped with cognitive electronic reconnaissance equipment to conduct passive reconnaissance of non-cooperative ground radar networks can efficiently obtain radiation source information.

[0003] The research on cognitive reconnaissance of UAVs mainly includes two aspects: ① How to adjust the flight posture of the UAV in physical space so that the radiation source signal falls within the reconnaissance beam range. ② How to control the distance between the UAV and the radar to meet the requirements of signal interception and risk avoidance. For cognitive electronic reconnaissance missions of non-cooperative radar targets, existing studies have achieved long-term continuous interception of a single radar signal by a single UAV. However, high-value air defense positions often use networked radars for joint detection to enhance detection power, which greatly reduces the space for safe reconnaissance of UAVs and increases their battlefield survival risks.

[0004] At present, scholars focus more on task allocation when multiple UAVs perform multi-target tasks rather than the complex information acquisition process, ignoring the impact of information coupling between multiple targets. However, during the UAV reconnaissance process, it is necessary to guard against detection by both target radar and adjacent radars.

[0005] The convergence of the reconnaissance strategy for less than three radars was achieved, but it was difficult to converge for more than four radars. The reason is that the increase in the number of radars will lead to the coupling of their detection power, and the weights of the reward function for seeking benefits and avoiding harm are difficult to determine, making it difficult to find the point of global benefit balance, making the reconnaissance strategy prone to fall into local optimality. Summary of the invention

[0006] In view of this, the present invention provides a UAV multi-target cognitive electronic reconnaissance system to solve the problem that the reconnaissance strategies of more than four radars are difficult to converge.

[0007] Optionally, the present invention provides a UAV multi-target cognitive electronic reconnaissance system, including a UAV platform and a UAV, for intercepting radar signals;

[0008] Scouter_N mission simulation module, a simulator for modeling UAV multi-target cognitive electronic reconnaissance mission scenarios;

[0009] A reconnaissance mission area, where the radar is set in the reconnaissance mission area, wherein the reconnaissance mission area is provided with a plurality of pseudo target modules, wherein the radar position is obtained according to the nearest neighbor method to obtain a plurality of pseudo targets, and the centers of the plurality of pseudo target arcs are solved and updated and iterated continuously, and finally a pseudo target is formed;

[0010] The UAV multi-target cognitive electronic reconnaissance strategy learning module is used to converge radar reconnaissance strategies and build a UAV multi-target cognitive electronic reconnaissance strategy network.

[0011] By adopting the above technical solution, when the UAV conducts reconnaissance on a networked radar with more than four units, it can achieve the balance point of electronic reconnaissance of multiple radar targets by setting a pseudo target, that is, shaping a reconnaissance pseudo target and giving a corresponding reconnaissance maneuver strategy to reconcile the contradiction between the two behaviors of approaching and avoiding the UAV. In addition, by defining the pseudo target state, the input state dimension reduction during the reconnaissance strategy training is achieved, which is very effective when facing a networked radar with an arc layout, improving the convergence speed of the reconnaissance strategy of more than four radars and simplifying the convergence difficulty of the reconnaissance strategy of more than four radars.

[0012] Optionally, the radar positions are arranged in an arc with the center of the position. Since the z coordinates of each radar are the same, the spatial coordinates of the radar can be projected onto the xoy plane to calculate the pseudo target, including the following three cases:

[0013] ① When there are two radars, the positions of the two radars (x a1 ,y a1 )、(x a2 ,y a2 ) as the starting point and end point to draw a line segment, the midpoint of the line segment Set as a false target;

[0014] ② When there are three radars, the positions of the three radars (x b1 ,y b1 )、(x b2 ,y b2 )、(x b3 ,y b3 ) as the vertex to make a triangle, and the center of its circumscribed circle κ is set as a pseudo target, κ=(x',y')

[0015]

[0016]

[0017] ③ When the number of radars is M (M>3), first group the M radars into several groups with two or three radars as the smallest unit, and use methods ① and ② to obtain the pseudo targets of each group of radars respectively; repeat the above process for the pseudo targets obtained after the above grouping, and finally obtain a unique pseudo target.

[0018] By adopting the above technical solution, the setting of pseudo targets is realized, and the UAV can conduct reconnaissance on the pseudo targets, thereby achieving the purpose of rapid convergence of the reconnaissance strategy network.

[0019] Optionally, the process of building the UAV multi-target cognitive electronic reconnaissance strategy network includes building a mixed target observation state, building a mixed target reward function, and building a mixed scenario training network. The reward setting that combines the pseudo target and the real radar target is used to overcome the difficulty in decoupling the detection power of the networked radar and the difficulty in determining the reward weight, thereby avoiding the algorithm from falling into a local optimum.

[0020] By adopting the above technical solution and using the reward setting that combines pseudo targets and real radar targets, we can overcome the difficulties in decoupling the detection power of networked radars and determining the reward weights, thus preventing the algorithm from falling into a local optimum.

[0021] Optionally, the mixed target observation state uses a mixed observation state of a pseudo target and a real radar target as the input of the strategy network. Normalized observation state of the drone at time t As shown in the following formula:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] In the above formula, It is the drone location. is the vector displacement between the pseudo target and the UAV, is the difference between the speed direction of the drone and the line of sight of the pseudo target. Corresponding to the power of the 1st to Mth radar signals reaching the drone; x u,t ,y u,t ,h u,t are the three-dimensional coordinates of the drone’s position, Δx = X max -X min ,Δy=Y max -Y min ,Δh=H max -H min ;X max , X min , Y max , Y min , H max , Hmin They are the maximum and minimum length, width, and height of the task area, respectively. is the displacement vector between the pseudo target and the UAV (including observation error), l c,t is the unit vector of the line of sight between the pseudo target and the drone, projecting it onto the x-axis, y-axis, and z-axis to obtain l c,t,x , l c,t,y , l c,t,z , v mov,t,x , v mov,t,y , v mov,t,z They are the projections of the UAV’s velocity direction vector on the three coordinate axes. is the Mth radar signal power reaching the UAV (including observation error).

[0028] By adopting the above technical solution, the description of the observation state of the mixed target is completed, and the mixed observation state of the combination of the pseudo target and the real radar target is used as the input of the strategy network to quantify the input of the strategy network.

[0029] Optionally, the mixed reward function includes distance reward, angle reward, power reward and reconnaissance mission completion reward. The pseudo target is used to guide the drone to approach the key reconnaissance area, and its distance reward is r 1,t :

[0030]

[0031] In the above formula, is the vector distance between the pseudo target and the UAV (including observation error), They correspond to the power of the first to Mth radar signals arriving at the UAV (including observation error), is the power of the mth radar signal intercepted by the UAV (including observation error), P u,min is the sensitivity of the drone’s reconnaissance payload;

[0032] The angle reward is calculated by using the angle relationship between the drone and each radar. It is beneficial for the reconnaissance antenna to align with the radar. The angle reward of the drone for the pseudo target is r 2,t :

[0033]

[0034] In the above formula, θ c,t|xoy and θ c,t|yoz They are θ c,t The projection of the xoy plane and yoz plane in the UAV platform coordinate system, θ c,t It is the angle between the line of sight direction vector and the wing direction vector between the UAV and the pseudo target, which can also be defined as the behavior angle BHA;

[0035] The power reward is used to improve reconnaissance efficiency, shorten mission time, and encourage drones to observe multiple radars and intercept their signals at the same time. The power reward is calculated using the power relationship between the drone and each radar. The power reward of the drone for the mth radar is r 3,m,t :

[0036]

[0037] The combined power bonus of M radars is r 3,t :

[0038]

[0039] The reconnaissance mission completion reward is a one-time mission completion reward r when the drone completes all radar target reconnaissance missions. 4,t :

[0040]

[0041] In the above formula, I 1～M is the number of reconnaissance cycles of the UAV for the first to mth radars, I is the mission completion criterion, when the UAV completes all radar target reconnaissance missions, it is 300, if the UAV is destroyed by air defense firepower, it is -300;

[0042] In summary, the reward function at time t is recorded as r t :

[0043] r t =χ1r 1,t +χ2r 2,t +χ3r 3,t +χ4r 4,t ,

[0044] χ1, χ2, χ3, and χ4 are the assigned weights, and their values ​​are all set to 1 here.

[0045] By adopting the above technical solution, the optimization of the mixed reward function is completed and the purpose of mixed target observation is achieved.

[0046] Optionally, the hybrid scene training network is a hybrid scene training based on an Actor-Critic (AC) framework, and the hybrid scene training network includes pre-training and reinforcement training; the pre-training reduces the difficulty of the overall task by setting a smaller number of radar targets and reconnaissance cycles, ensuring that there are sufficient excellent samples in the pre-training process, so that the pre-training network converges quickly;

[0047] The enhanced training is based on pre-training, increasing the number of radar targets and the reconnaissance cycle, and conducting enhanced training to ensure that the MTR strategy network meets the requirements of the reconnaissance mission.

[0048] By adopting the above technical solution, pre-training and reinforcement training are mixed to solve the problem of difficulty in convergence of strategy network caused by sparse excellent samples, and significantly accelerate the convergence of reconnaissance network.

[0049] Optionally, a simulation experiment module is also included, which includes a simulation environment, indicator analysis, result analysis and behavior analysis. The simulation experiment module changes the number of radar targets, sets multiple groups of reconnaissance tasks with different difficulties, tests the convergence of the learning framework, and analyzes the cognitive reconnaissance behavior of the UAV.

[0050] By adopting the above technical solution, the collection of experimental data is completed.

[0051] In summary, the present application includes at least one of the following beneficial technical effects:

[0052] 1. This application sets a pseudo target, finds the balance point of multi-target electronic reconnaissance, shapes a reconnaissance pseudo target and gives a corresponding reconnaissance maneuver strategy to reconcile the contradiction between the two behaviors of approaching and avoiding the drone. In addition, by defining the pseudo target state, the input state dimension reduction during the reconnaissance strategy training is achieved;

[0053] 2. This application studies the mixed target reward function and uses the reward setting that combines pseudo targets and real radar targets to overcome the difficulty in decoupling the detection power of networked radars and the difficulty in determining the reward weights, thus avoiding the algorithm from falling into the local optimum;

[0054] 3. This application proposes a hybrid scenario training method to conduct multi-target cognitive electronic reconnaissance strategy training. By mixing simple scenario pre-training and complex scenario reinforcement training, the problem of difficulty in convergence of strategy network caused by sparse excellent samples is solved, and the convergence of reconnaissance network is significantly accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of the UAV multi-target cognitive electronic reconnaissance mission scenario.

[0056] Figure 2 This is a schematic diagram of the pseudo target calculation process.

[0057] Figure 3 It is a mixed scenario training network structure based on the AC framework.

[0058] Figure 4 It is the network structure of DDPG, TD3, and SAC algorithms.

[0059] Figure 5 It is the reward result graph.

[0060] Figure 6 It is the behavior angle result diagram.

[0061] Figure 7 It is a drone track in the third test episode of five radar reconnaissance missions. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0063] At present, in order to improve the convergence speed of the reconnaissance strategy of more than four radars, an unmanned aerial vehicle multi-target cognitive electronic reconnaissance system is disclosed in an embodiment of the present application.

[0064] A multi-target cognitive electronic reconnaissance system for unmanned aerial vehicles includes an unmanned aerial vehicle platform, an unmanned aerial vehicle, a Scouter_N task simulation module, a reconnaissance task area, and a multi-target cognitive electronic reconnaissance strategy learning module for unmanned aerial vehicles, wherein the unmanned aerial vehicle and the unmanned aerial vehicle platform are used to intercept radar signals, the Scouter_N task simulation module is a simulator for modeling the multi-target cognitive electronic reconnaissance mission scenario for unmanned aerial vehicles; in the reconnaissance task area, a radar is set in the reconnaissance task area, and the reconnaissance task area is provided with multiple pseudo target modules, and the radar position is obtained according to the nearest neighbor method to obtain multiple pseudo targets, and the arc centers of multiple pseudo targets are solved and updated and iterated, and finally a pseudo target is formed; the multi-target cognitive electronic reconnaissance strategy learning module for unmanned aerial vehicles is used for the convergence of radar reconnaissance strategies and the establishment of a multi-target cognitive electronic reconnaissance strategy network for unmanned aerial vehicles. When the unmanned aerial vehicle conducts reconnaissance on a networked radar greater than four, the balance point of electronic reconnaissance of multiple radar targets is achieved by setting pseudo targets, that is, a reconnaissance pseudo target is formed and a corresponding reconnaissance maneuver strategy is given to reconcile the contradiction between the two behaviors of approaching and avoiding the unmanned aerial vehicle. In addition, by defining the pseudo-target state, the input state dimensionality reduction is achieved during the reconnaissance strategy training, which is very effective when facing arc-shaped networked radars. It improves the convergence speed of the reconnaissance strategy of more than four radars and simplifies the convergence difficulty of the reconnaissance strategy of more than four radars.

[0065] Among them, the Scouter_N mission simulation module is a simulator that can be used to model the multi-target cognitive electronic reconnaissance mission scenario of UAVs. Figure 1 As shown in the figure, M (M>3) radars deployed at the edge of the position jointly perform air defense tasks, search for air targets and destroy locked targets. The deployment position of M radars meets the position air defense coverage requirements, that is, the detection areas of two adjacent radars overlap. Figure 1 The yellow circle in the middle indicates the detection range of each radar (radius Da,m ). The drone carrying the electronic reconnaissance payload is responsible for detecting the signals of M radars in the area and recognizing the joint detection power of the networked radars in order to guide precise electronic interference or firepower destruction. Through the previous intelligence data, the drone can know the approximate location of these radars, but the detection power of the radars is unknown. In order to intercept the signal without being tracked by any radar, the drone must fly outside the joint detection power of M radars and maintain continuous reconnaissance of each radar for a certain period of time in order to optimize its clustering and recognition results. Figure 1 In the figure, the green areas on both sides of the drone fuselage are the detection range of the reconnaissance antenna. The reconnaissance beam can only receive signals within a limited range (azimuth angle is α, pitch angle is β).

[0066] Mathematical model analysis of the above Scouter_N task simulation module:

[0067] ① Electromagnetic environment of reconnaissance payload:

[0068] The reconnaissance payload carried by the UAV is in a complex electromagnetic environment where M radars radiate together, i.e., the signals of M radars are superimposed, expressed as K t,t+Dt :

[0069]

[0070] In formula (1), k m,n,t Represents the electromagnetic signature of the nth signal generated by the mth radar at time t.

[0071] ②Reconnaissance payload intercepts radar signals:

[0072] The interception capability of the reconnaissance payload is limited. It can only be intercepted when the signal falls into the reconnaissance beam range and the energy is higher than its sensitivity. m,t Describe the alignment of the reconnaissance antenna to the mth radar.

[0073] θ m,t =deg(arccos <v mov,t d m,t >)-90 (2),

[0074] In formula (2), d m,t is the relative vector displacement between the mth radar and the UAV in the geocentric coordinate system, v mov,t The unit vector representing the speed of the drone. The reconnaissance flag of the nth signal of the mth radar is ζ m,n,t The judgment conditions are:

[0075]

[0076] In formula (3), θ m,t|xoy and θ m,t|yozThey are θ m,t Projection of the xoy plane and yoz plane in the UAV platform coordinate system. u,m,t is the power of the mth radar signal intercepted by the UAV. u,min is the sensitivity of the UAV reconnaissance payload. Therefore, within Δt, the radar signal intercepted by the reconnaissance payload is expressed as

[0077]

[0078] In formula (4), p m is the probability that the mth radar signal enters the reconnaissance payload without being lost. After processing, these intercepted radar signals will form M signal feature cluster centers, which can be optimized by increasing the continuous reconnaissance cycle for each radar. For the mth radar, the number of cycles in which its signal is continuously intercepted is I m The judgment conditions are:

[0079]

[0080] In formula (5), N e represents the maximum number of interaction steps allowed in a reconnaissance mission, Q(d m,t ) is used to describe the tracking error of the mth radar to the UAV. When the error is less than the threshold δ m , will trigger the guidance condition. When the UAV successfully intercepts the signal of the mth radar and the radar does not enter the guidance state, it is most beneficial for the UAV to complete the mission. The distance between the UAV and the M radars is the key to completing the reconnaissance mission, which can be achieved through the maneuvering vector a of the UAV platform. u,t To maximize I m , the objective function of the multi-target cognitive electronic reconnaissance mission at time t can be rewritten as:

[0081]

[0082] In formula (6), λ m , η m represents the adjustment coefficient, all values ​​are greater than 0, and λ m ·η m =1.

[0083] In summary, the coupling of detection power between radars will greatly increase the complexity of UAV reconnaissance missions. Therefore, enabling UAVs to master timely evaluation methods for the coupling of detection power between networked radars is the key to optimizing reconnaissance strategies.

[0084] Application of deep reinforcement learning algorithm in the UAV multi-target cognitive electronic reconnaissance strategy learning module:

[0085] The basic principle and state transition of the deep reinforcement learning algorithm are described as follows: Limited by the interception and processing capabilities of the reconnaissance payload and the maneuverability of the platform, the drone can only perceive part of the environmental information, which belongs to a partially observable Markov process. The POMDP of the drone's cognitive electronic reconnaissance mission is described by a six-tuple <Σ,Ο,Α,F,R,γ>: Σ is the set of the actual states of the drone and the radar {Σ r ,Σ u ,Π},Σ r is a set of M radar positions {s r,1 ,…,s r,M},Σ u is the set of UAV positions, Π is the set of M radar signal powers intercepted by the UAV {P u,1,t ,…,P u,M,t}. Ο is the set of states observed by the drone is the vector distance set between the drone and M radars (including observation errors), is the set of deviations between the M radar sight direction unit vectors and the drone speed direction unit vectors is the power set of M radar signals intercepted by the drone Α is the set of acceleration actions of the drone in azimuth and pitch. F is the state transfer function, R is the reward function, and γ is the discount coefficient. At time t, the real environment state Status observed by drone According to the strategy π, choose action Transfer function Transfer to state Get the corresponding reward R(s t+1 ,a t ,s t ). During the reconnaissance mission time period T, the optimal strategy π* is selected by maximizing the cumulative reward:

[0086]

[0087] The principle of shaping the pseudo target is as follows: considering the huge threat posed by the joint detection of networked radars to the survival of UAVs, a reward for avoiding harm is designed. Under the condition of ensuring the survival of UAVs, a reward for seeking profit is used to encourage UAVs to complete reconnaissance tasks. However, it is difficult to converge for reconnaissance tasks with a large number of radars (more than four radars). There are mainly two reasons: ① The weights of avoiding harm and seeking profit in the overall reward are difficult to control; ② The coupled detection power of networked radars has a superposition and offsetting effect on the UAV's seeking profit and avoiding harm, resulting in an oscillation phenomenon similar to that in the artificial potential field algorithm, making it difficult for UAVs to learn effective reconnaissance behavior.

[0088] In summary, the detection power of networked radar cannot be simply decoupled. It should be considered as a whole, that is, to define the comprehensive detection power generated by a pseudo target. Figure 1 As shown in the figure, the white dotted radar target is a pseudo target, and the rest are real radar targets. The pseudo target is designed to directly guide the UAV to quickly reach the key reconnaissance area without establishing a reconnaissance sequence. Considering that the joint detection range of the networked radar is to meet the position air defense needs, it is generally deployed at the edge of the position, and the position distribution of the networked radar is arranged in an arc with the center of the position, as shown in Figure 1. Figure 2 (a) is shown. The following is a method for finding the center of a pseudo target: divide the radar position into two or three groups according to the nearest neighbor method, and meet the requirements of parity splitting. After that, several pseudo target points are obtained, and the arc centers of these pseudo targets are solved and continuously updated and iterated until they are finally transformed into a pseudo target.

[0089] Since the z coordinates of each radar are the same, their spatial coordinates can be projected onto the xoy plane to calculate the pseudo target, including the following three cases:

[0090] ① Two radars, with their positions (x a1 ,y a1 )、(x a2 ,y a2 ) as the starting point and end point to draw a line segment, the midpoint of the line segment Set as a false target.

[0091] ②Three radars, with their positions (x b1 ,y b1 )、(x b2 ,y b2 )、(x b3 ,y b3 ) as the vertices to make a triangle, and the center of its circumscribed circle κ is set as the pseudo target.

[0092] κ=(x',y')

[0093]

[0094]

[0095] ③M (M>3) radars are first grouped into several groups with two or three radars as the smallest unit. Figure 2 In (b), the red, yellow, and blue boxes represent different radar groups, respectively. The pseudo targets of each radar group are obtained using the above methods ① and ②. Figure 2 In (c), the red triangle, yellow triangle, and blue triangle represent the pseudo targets of the corresponding groups, respectively; the above process is repeated for the obtained pseudo targets, and finally a unique pseudo target is obtained. Figure 2In (d), the white five-pointed star represents the final pseudo target.

[0096] The state of mixed target observation of pseudo targets and real radar targets is: Different from the single target reconnaissance task, the MTR task needs to quantify the threat state of M radars at the same time. If the observation information of M radars is input into the policy network at the same time, the mapping relationship between input and output will be complicated, making the network fitting difficult. Therefore, the mixed observation state of pseudo targets and real targets is used as the input of the policy network. The normalized observation state of the drone at time t As shown in formula (9):

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] In formula (15), It is the drone location. is the vector displacement between the pseudo target and the UAV. It is the difference between the speed direction of the drone and the line of sight direction of the pseudo target. They correspond to the power of the first to Mth radar signals reaching the drone. u,t ,y u,t ,h u,t are the three-dimensional coordinates of the drone’s position. max -X min ,Δy=Y max -Y min ,Δh=H max -H min .X max , X min , Y max , Y min , H max , H min They are the maximum and minimum values ​​of the length, width, and height of the task area respectively. is the displacement vector between the pseudo target and the UAV (including observation error). c,t is the unit vector of the line of sight between the pseudo target and the drone, projecting it onto the x-axis, y-axis, and z-axis to obtain l c,t,x , l c,t,y , l c,t,z .v mov,t,x , v mov,t,y , vmov,t,z They are the projections of the UAV speed direction vector on the three coordinate axes. is the Mth radar signal power reaching the UAV (including observation error).

[0103] The mixed reward function includes distance reward, angle reward, power reward and reconnaissance mission completion reward. The mixed target reward function is: Scouter shapes the reward function from three aspects: distance, angle and power. Although the number of radar targets in the Scouter_N task simulation module has increased compared with the former, it is still a reconnaissance mission in essence, and the reward function can still be shaped from three aspects: distance, angle and power.

[0104] ① Distance Reward:

[0105] The pseudo target is used to guide the drone to approach the key reconnaissance area, and its distance reward is r 1,t :

[0106]

[0107] In formula (10), is the vector distance between the pseudo target and the UAV (including observation error). They correspond to the power of the first to Mth radar signals arriving at the UAV (including observation errors). is the power of the mth radar signal intercepted by the UAV (including observation error). u,min is the sensitivity of the UAV reconnaissance payload.

[0108] ② Angle Rewards:

[0109] Calculating the reward using the angle relationship between the drone and each radar is more conducive to aligning the reconnaissance antenna with the radar, but when the drone observes more than two radars at the same time, the weight of the angle reward is difficult to reasonably distribute. From a global perspective, although using pseudo targets to calculate the angle reward may affect the alignment of the drone reconnaissance antenna with each radar, it can ensure the completion of the overall mission. The angle reward of the drone for the pseudo target is r 2,t :

[0110]

[0111] In formula (11), θ c,t|xoy and θ c,t|yoz They are θ c,t Projection of the xoy plane and yoz plane in the UAV platform coordinate system. θ c,t It is the angle between the line of sight direction vector and the wing direction vector between the UAV and the pseudo target, and can also be defined as the behavior angle BHA.

[0112] ③Power Rewards:

[0113] In order to improve reconnaissance efficiency and shorten mission time, drones are encouraged to observe multiple radars and intercept their signals at the same time. The power reward is calculated using the power relationship between the drone and each radar. The power reward of the drone for the mth radar is r 3,m,t :

[0114]

[0115] The combined power bonus of M radars is r 3,t :

[0116]

[0117] ④Rewards for completing reconnaissance missions:

[0118] In the MTR mission, when the UAV focuses on some targets, it is easy to fall into the local optimum. To solve this problem, a reconnaissance mission completion reward is set. When the UAV completes all radar target reconnaissance missions, it will receive a one-time mission completion reward r. 4,t :

[0119]

[0120] In formula (13), I 1～M is the number of reconnaissance cycles of the UAV for the first to Mth radars, and I is the mission completion criterion. When the UAV completes all radar target reconnaissance missions, it gets 300, and if the UAV is destroyed by air defense firepower, it gets -300.

[0121] In summary, the reward function at time t is recorded as r t :

[0122] r t =χ1r 1,t +χ2r 2,t +χ3r 3,t +χ4r 4,t (15)

[0123] χ1, χ2, χ3, and χ4 are the assigned weights, and their values ​​are all set to 1 here.

[0124] The mixed scenario training network is: There are three main reasons why the MTR strategy network has difficulty in convergence: ① The shape of the joint detection area of ​​multiple radars is irregular, which increases the difficulty of algorithm exploration; ② The drone needs to maintain a certain number of continuous reconnaissance cycles for each radar to obtain a large reward, and the sparsity of excellent samples makes it difficult for the network to converge; ③ The setting of pseudo-targets can encourage drones to recognize the joint detection power of multiple radars from a global reconnaissance perspective, but weakens the local reconnaissance perspective of each radar, causing the drone to easily "miss the target" and fall into a local optimum.

[0125] In response to the above problems, a hybrid scene training based on the Actor-Critic (AC) framework is proposed. The hybrid scene training of the Actor-Critic (AC) framework includes pre-training and intensive training: ① Use simple scenes for pre-training first. Reduce the difficulty of the overall task by reducing the number of radar targets and reconnaissance cycles, and ensure that there are enough excellent samples in the pre-training process so that the pre-training network converges quickly. ② Use complex scenes for intensive training. On the basis of pre-training, increase the number of radar targets and reconnaissance cycles, and perform intensive training to ensure that the MTR strategy network meets the requirements of the reconnaissance mission. Through simple scene pre-training, help drones master typical reconnaissance behaviors. These reconnaissance behaviors will not change significantly due to the increase in task difficulty, so they can be used to improve the convergence of high-difficulty tasks in complex scenes. Its network structure is as follows Figure 3 shown.

[0126] The simulation experiment module includes simulation environment, indicator analysis, result analysis and behavior analysis. The simulation experiment module changes the number of radar targets, sets four groups of reconnaissance tasks with different difficulties, tests the convergence of the learning framework, and deeply analyzes the cognitive reconnaissance behavior of the UAV.

[0127] First, conduct a simulation environment. The number of radar radiation sources will directly affect the complexity of the electromagnetic environment. The more radars there are, the more complex the electromagnetic environment is, and the more difficult the reconnaissance mission is. Divide the mission scenarios according to the number of radar targets. Each mission corresponds to an episode executed by the drone, and the maximum number of steps allowed for each episode is 1,500. The criteria for mission termination are divided into three categories:

[0128] ① Determine that M targets are in the feature library, and the number of cycles I in which each target is continuously intercepted 1～M ≥I, drone wins;

[0129] ② If the target is in the feature library, and the tracking error of any radar tracking the drone at time t satisfies the fire strike condition, the radar wins;

[0130] ③Neither of the above two events occurs until the maximum number of steps is reached and the episode ends.

[0131] Before formal training, each learning network needs to be pre-trained for 3,000 episodes for a simple task with three radar targets and a reconnaissance cycle of 30 for each radar target. Then, intensive training is performed for 2,000 episodes for task scenarios with different numbers of targets and a reconnaissance cycle of 100. Before each episode, the position, speed and other status information of the UAV and radar are reinitialized.

[0132] The maximum speed of the UAV in the Scouter_N mission simulation module is 150m / s, and other configuration parameters remain unchanged. The software and hardware environment parameters of the Scouter_N mission simulation module are shown in Table 1.

[0133] Table 1 Software and hardware environment parameters of Scouter_N

[0134]

[0135] Three typical AC framework reinforcement learning algorithms, DDPG, TD3, and SAC, are used to optimize the drone reconnaissance strategy. The network structure of the algorithm is as follows: Figure 4 shown.

[0136] Indicator analysis: (1) Percentage of Mission Completion (PMC)

[0137] Test 1,000 episodes, count the percentage of episodes the drone wins, and evaluate whether the drone understands the MTR task.

[0138] (2) Reward

[0139] The reward values ​​obtained by the drone in the training episodes are counted to evaluate the convergence of the reconnaissance strategy, and the average value is taken every 100 episodes.

[0140] (3) Pseudo target behavior angle (BehaviorAngle, BHA)

[0141] The behavior angle values ​​of the drone towards the pseudo target in the training episodes are counted to evaluate the alignment between the reconnaissance antenna and the pseudo target, and the average value is taken every 100 episodes for display.

[0142] (4) Mean Decision Time (MDT)

[0143] The average decision-making time of the drone in the training episodes was counted to evaluate the decision-making speed of the drone.

[0144] (5) Training Time (TT)

[0145] Count the training time spent by the drone in the training episode and evaluate the time cost of the algorithm.

[0146] Results analysis: The simulation results of different mission scenarios in the Scouter-N mission simulation module are shown in Table 2. PMC value is the most important indicator of the MTR task and is the focus of analysis. Table 2 only counts the MDT and TT values ​​when the PMC value is greater than 60%. It can be seen from Table 2 that the method of combining reconnaissance pseudo-targets with scene pre-training proposed by us has greatly improved the performance of the DRL network. Compared with the literature, in the mission environment of 4 targets, the performance is improved by more than 40%, and the overall training time is significantly shortened. Figure 5 The convergence results of the most difficult six-target MTR task are shown, and PMC can still be maintained above 60%. In addition, by analyzing the convergence trend of the reward curve, the algorithm can also learn and enhance performance in more episodes. For the three different algorithms, the learning frameworks used all show good convergence, and within a given training episode, Pre-DDPG has the best overall performance.

[0147] The following is combined Figure 6 The behavior angle convergence curve shown in Figure 2 is used to analyze the reasons for the reduction of MDT and TT. It can be seen that for different numbers of radars, the behavior angles of all algorithms show a trend close to 0, which shows that the reconnaissance pseudo-targets can effectively guide the UAV to reconnaissance key areas, thereby overcoming the influence of the coupling of multiple radar detection power. Pre-training successfully allows the UAV to master the purpose of cognitive reconnaissance in advance, that is, to reduce the behavior angle on the basis of approaching the target.

[0148] Table 2 Simulation results of Scouter_N in different mission scenarios

[0149]

[0150] Behavior Analysis: Finally, taking a successful five-radar reconnaissance mission as an example, a test episode was selected to analyze the typical reconnaissance behavior of the drone, such as Figure 7 As shown, Figure 7 (a1) to (b3) correspond to steps 176, 254, 601, 903, 1061, and 1357 respectively. The red curve in the figure is the trajectory of the drone. Therefore, we summarize the typical behaviors of intelligent drones, which are mainly divided into the following three types:

[0151] ① Start at the nearest location: When the UAV is far away from the radar target, it will accelerate to the key reconnaissance area and start reconnaissance from the nearest radar target;

[0152] ②Continuous hovering: In order to avoid the detection coverage of the networked radar, the drone mainly flies steadily without any maneuvers that produce large overloads, which is different from single-target reconnaissance behavior. This shows that after the drone's safe reconnaissance airspace is further compressed, stability is the primary factor in completing cognitive reconnaissance;

[0153] ③ Backtracking and reentry. When the safe reconnaissance airspace is limited but the reconnaissance cycle is required to be long, it is difficult for drones to complete the reconnaissance of all targets at one time. Observing a large number of behaviors, it is found that drones put battlefield survival first, mainly focus on safe reconnaissance, and use reentry-style reconnaissance behaviors to meet the mission of continuous interception of radar signals.

[0154] Experiments have shown that this method can achieve electronic reconnaissance of six radar targets under partially observable conditions, and the UAV has mastered typical maneuvers such as stable circling.

[0155] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A UAV multi-target cognitive electronic reconnaissance system, comprising a UAV platform and a UAV, for intercepting radar signals, characterized in that: It also includes the Scouter_N mission simulation module, a simulator for modeling UAV multi-target cognitive electronic reconnaissance mission scenarios; A reconnaissance mission area, where the radar is set in the reconnaissance mission area, wherein the reconnaissance mission area is provided with a plurality of pseudo target modules, wherein the radar position is obtained according to the nearest neighbor method to obtain a plurality of pseudo targets, and the centers of the plurality of pseudo target arcs are solved and updated and iterated continuously, and finally a pseudo target is formed; The UAV multi-target cognitive electronic reconnaissance strategy learning module is used to converge radar reconnaissance strategies and build a UAV multi-target cognitive electronic reconnaissance strategy network.

2. The multi-target cognitive electronic reconnaissance system for unmanned aerial vehicles according to claim 1 is characterized in that: The radar positions are distributed in an arc with the center of the position. Since the z coordinates of each radar are the same, the spatial coordinates of the radar can be projected onto the xoy plane to calculate the pseudo target, including the following three cases: ① When there are two radars, the positions of the two radars (x a1 ,y a1 )、(x a2 ,y a2 ) as the starting point and end point to draw a line segment, the midpoint of the line segment Set as a false target; ② When there are three radars, the positions of the three radars (x b1 ,y b1 )、(x b2 ,y b2 )、(x b3 ,y b3 ) is used as the vertex to make a triangle, and the center of its circumscribed circle κ is set as a pseudo target; κ=(x',y') ③ When the number of radars is M (M>3), first group the M radars into several groups with two or three radars as the smallest unit, and use methods ① and ② to obtain the pseudo targets of each group of radars respectively; repeat the above process for the pseudo targets obtained after the above grouping, and finally obtain a unique pseudo target.

3. The multi-target cognitive electronic reconnaissance system for unmanned aerial vehicles according to claim 2 is characterized in that: The process of building the UAV multi-target cognitive electronic reconnaissance strategy network includes building a mixed target observation state, building a mixed target reward function, and building a mixed scenario training network. The reward setting that combines the pseudo target and the real radar target is used to overcome the difficulties in decoupling the detection power of the networked radar and determining the reward weight, thereby avoiding the algorithm from falling into the local optimum.

4. The UAV multi-target cognitive electronic reconnaissance system according to claim 3 is characterized in that: The mixed target observation state uses the mixed observation state of the pseudo target and the real radar target as the input of the strategy network. The normalized observation state of the UAV at time t As shown in the following formula: In the above formula, It is the drone location. is the vector displacement between the pseudo target and the UAV, is the difference between the speed direction of the drone and the line of sight of the pseudo target. Corresponding to the power of the 1st to Mth radar signals reaching the UAV; x u,t ,y u,t ,h u,t are the three-dimensional coordinates of the drone’s position, Δx = X max -X min ,Δy=Y max -Y min ,Δh=H max -H min ;X max , X min , Y max , Y min , H max , H min They are the maximum and minimum length, width, and height of the task area, respectively. is the displacement vector between the pseudo target and the UAV (including observation error), l c,t is the unit vector of the line of sight between the pseudo target and the drone, projecting it onto the x-axis, y-axis, and z-axis to obtain l c,t,x , l c,t,y , l c,t,z ;v mov,t,x , v mov, t ,y , v mov,t,z They are the projections of the UAV speed direction vector on the three coordinate axes. is the power of the Mth radar signal reaching the UAV (including observation error).

5. The UAV multi-target cognitive electronic reconnaissance system according to claim 4 is characterized in that: The mixed reward function includes distance reward, angle reward, power reward and reconnaissance mission completion reward. The pseudo target is used to guide the UAV to approach the key reconnaissance area, and its distance reward is r 1,t : In the above formula, is the vector distance between the pseudo target and the UAV (including observation error), They correspond to the power of the first to Mth radar signals arriving at the UAV (including observation error), is the power of the mth radar signal intercepted by the UAV (including observation error), P u,min is the sensitivity of the drone’s reconnaissance payload; The angle reward is calculated by using the angle relationship between the drone and each radar. It is beneficial for the reconnaissance antenna to align with the radar. The angle reward of the drone for the pseudo target is r 2,t : In the above formula, θ c,t|xoy and θ c,t|yoz They are θ c,t The projection of the xoy plane and yoz plane in the UAV platform coordinate system, θ c,t It is the angle between the line of sight direction vector and the wing direction vector between the UAV and the pseudo target, which can also be defined as the behavior angle BHA; The power reward is used to improve reconnaissance efficiency, shorten mission time, and encourage drones to observe multiple radars and intercept their signals at the same time. The power reward is calculated using the power relationship between the drone and each radar. The power reward of the drone for the mth radar is r 3,m,t : The combined power bonus of M radars is r 3,t : The reconnaissance mission completion reward is a one-time mission completion reward r when the drone completes all radar target reconnaissance missions. 4,t : In the above formula, I 1～M is the number of reconnaissance cycles of the UAV for the first to mth radars, I is the mission completion criterion, when the UAV completes all radar target reconnaissance missions, it is 300, if the UAV is destroyed by air defense firepower, it is -300; In summary, the reward function at time t is recorded as r t : r t =χ1r 1,t +x2r 2,t +x3r 3,t +x4r 4,t , χ1, χ2, χ3, and χ4 are the assigned weights, and their values ​​are all set to 1 here.

6. The UAV multi-target cognitive electronic reconnaissance system according to claim 5 is characterized in that: The hybrid scene training network is a hybrid scene training based on the Actor-Critic (AC) framework, and the hybrid scene training network includes pre-training and reinforcement training; the pre-training reduces the difficulty of the overall task by setting a smaller number of radar targets and reconnaissance cycles, ensuring that there are enough excellent samples in the pre-training process, so that the pre-training network converges quickly; The enhanced training is based on pre-training, increasing the number of radar targets and the reconnaissance cycle, and conducting enhanced training to ensure that the MTR strategy network meets the requirements of the reconnaissance mission.

7. The UAV multi-target cognitive electronic reconnaissance system according to claim 6 is characterized in that: It also includes a simulation experiment module, which includes a simulation environment, indicator analysis, result analysis and behavior analysis. The simulation experiment module changes the number of radar targets, sets multiple groups of reconnaissance tasks with different difficulties, tests the convergence of the learning framework, and analyzes the cognitive reconnaissance behavior of the UAV.

Images