Reinforcement learning based multi-uav radar cross section optimization method and system

Abstract

The application discloses a multi-unmanned aerial vehicle radar cross section optimization method and system based on reinforcement learning, and relates to the technical field of multi-unmanned aerial vehicle cooperative control. An unmanned aerial vehicle is selected as a master agent, and a radar cross section (RCS) value is taken as a reference; the rest unmanned aerial vehicles are slave agents; state information of the unmanned aerial vehicle group is acquired in real time and input into an action policy network constructed based on a TD3 algorithm, a continuous action vector is output, real-time RCS adjustment instructions are generated, a cooperative flight trajectory of the unmanned aerial vehicle group in a three-dimensional space is dynamically simulated and generated based on Lie group SO (3) theory, a multi-dimensional reward function is designed, a reward value reflecting the RCS cooperative effect is calculated in real time, the policy network is cyclically updated through a reinforcement learning mechanism, and a cooperative control strategy is trained. Through cooperative RCS adjustment of the master unmanned aerial vehicle and the slave unmanned aerial vehicle, the RCS characteristic of the slave unmanned aerial vehicle is dynamically adjusted to a similar level of the master unmanned aerial vehicle, and a plurality of unmanned aerial vehicles present a highly similar RCS characteristic distribution in radar detection.

Description

Technical Field

[0001] This invention relates to the field of multi-UAV cooperative control and radar cross section optimization technology, and more specifically, to a method and system for multi-UAV radar cross section optimization based on reinforcement learning. Background Technology

[0002] In recent years, unmanned aerial vehicle (UAV) technology has developed rapidly in both civilian and special fields, particularly in applications such as UAV formation performances, collaborative operations, electronic warfare drills, and radar system testing. The collaborative control technology of multiple UAV systems has become a research hotspot. In these scenarios, it is often necessary to precisely control the radar characteristics of a UAV swarm, ensuring it exhibits a specific scattering pattern distribution in radar detection. This meets the low observability (stealth) requirements of formation flight, the realism requirements of electronic warfare drills, or the standardization requirements of radar system testing. The radar cross section (RCS) is a key physical quantity describing a target's ability to scatter radar waves; it directly determines the target's detectability, identifiable characteristics, and signal strength within a radar system. In multi-UAV collaborative applications, by intelligently controlling the RCS characteristics of each UAV to achieve highly similar or specific distribution patterns in radar detection, various tactical and technical effects can be realized, such as swarm electromagnetic stealth, target simulation, and decoy generation.

[0003] Currently, the main technical solutions for realizing RCS control and multi-drone collaboration of UAVs are as follows: passive RCS control technology, active RCS adjustment technology, formation flight control technology, and traditional multi-objective optimization methods. However, the existing technologies have the following shortcomings: (1) Lack of intelligent collaborative strategies: Existing methods mostly adopt preset fixed control rules and lack intelligent collaborative decision-making mechanisms that can adaptively adjust according to the real-time environment and task status. (2) Insufficient dynamic optimization and real-time capabilities: Traditional optimization algorithms have high computational complexity and slow convergence speed, making it difficult to handle the real-time RCS collaborative control requirements of multiple UAVs during dynamic flight. (3) Low RCS feature similarity control accuracy: Existing technologies cannot accurately and quickly converge the RCS features of multiple UAVs and maintain them within a highly similar range, resulting in inconsistent features of the group on radar, making them easy to identify and distinguish. (4) Lack of systematic evaluation standards: There is a lack of an effective, multi-dimensional quantitative index to comprehensively evaluate the overall effect of multi-UAV RCS collaborative control, such as similarity, concentration, uniformity, and smoothness. Therefore, there is an urgent need for a new method that can achieve online, intelligent, and high-precision collaborative optimization of RCS among multiple UAVs in order to overcome the shortcomings of the existing technologies. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a method and system for optimizing the radar cross section (RCS) of multiple UAVs based on reinforcement learning. By coordinating the RCS adjustment between the master UAV and the slave UAVs, the RCS characteristics of the slave UAVs are dynamically adjusted to a level similar to that of the master UAV, thereby enabling multiple UAVs to exhibit highly similar RCS feature distributions in radar detection.

[0005] The first aspect of this invention provides a method for optimizing the radar cross section of multiple unmanned aerial vehicles (UAVs) based on reinforcement learning, comprising the following steps:

[0006] S101: Construct a master-slave intelligent agent architecture, set one drone as the master drone, use the radar cross section (RCS) value of the master drone as the benchmark, and the other drones as slave drones, initialize the state and RCS parameters of all drones.

[0007] S102: The state information of the UAV swarm is obtained in real time through the state space encoder and input into the action policy network based on the TD3 algorithm. The action policy network is used to output continuous action vectors to generate the real-time RCS adjustment of the master UAV and each slave UAV.

[0008] S103: The attitude of the UAV is modeled based on the Lie group SO(3) theory, and based on the action vector, the cooperative flight trajectory of the UAV swarm in three-dimensional space is dynamically simulated and generated by combining the Euler integral method with environmental disturbances.

[0009] S104: Design a multidimensional reward function to calculate the reward value that reflects the RCS cooperative effect in real time. Using the reward value, update the action policy network cyclically through a reinforcement learning mechanism. Use the updated action policy network to iteratively optimize the RCS cooperative control strategy of the master-slave agent.

[0010] In this scheme, in step S101, a master-slave intelligent agent architecture is constructed, one drone is designated as the master drone, the radar cross section (RCS) value of the master drone is used as the reference, and the other drones are slave drones. The states and RCS parameters of all drones are initialized, including:

[0011] Construct a master-slave UAV cooperative control architecture consisting of a master intelligent agent controller, slave intelligent agent controllers and a cooperative optimization coordinator. Initialize the UAV swarm, set the UAV swarm to contain N UAVs, designate one UAV as the master UAV and the remaining N-1 UAVs as slave intelligent agents.

[0012] Load the predefined RCS lookup table, use the main intelligent agent controller to obtain the initial state data of the main UAV. The initial state data includes three-dimensional position coordinates, three-dimensional velocity components, and three attitude angles: roll, pitch, and yaw. Use the initial state data to query the RCS lookup table to obtain the initial reference RCS value of the main UAV. Use the initial reference RCS value as the RCS reference standard for the entire cluster and send it to the collaborative optimization coordinator.

[0013] The system uses a parallel acquisition of real-time status data from the drones by the agent controllers. Each agent controller independently queries the RCS lookup table to obtain the current RCS value of the corresponding drone.

[0014] In this scheme, step S102 involves a collaborative optimization decision-making process for state coding and RCS based on the TD3 algorithm, including:

[0015] The flight trajectory data of the entire UAV swarm at the current moment is obtained from the simulation environment. The global state vector corresponding to the flight trajectory data is obtained using a state space encoder, and the global state vector is standardized.

[0016] The standardized global state vector is fed into the action policy network, which is then used for in-depth processing and analysis to understand the cooperative situation of the UAV swarm and output an N-dimensional continuous action vector. The first element of the action vector corresponds to the adjustment command of the master UAV, and the remaining N-1 elements correspond to the adjustment commands of each slave UAV.

[0017] The motion vectors are linearly mapped to obtain the RCS adjustment values ​​of the master UAV and the slave UAV, respectively, and the RCS adjustment values ​​of all UAVs are applied to the simulation environment.

[0018] In this scheme, step S103 involves generating and updating the three-dimensional dynamic trajectory based on Lie group theory, including:

[0019] The attitude of the UAV is modeled based on the Lie group SO(3) theory. The attitude representation device receives the action vector output from the action policy network. According to the action vector, the target attitude change command of each UAV is given for pitch angle and yaw angle. According to the target attitude change command, the target three-dimensional attitude matrix is ​​calculated for each UAV based on the Lie group SO(3) theory.

[0020] Based on the target three-dimensional attitude matrix, the difference between the current attitude and the target attitude is obtained using a dynamics updater. According to the preset flight mission profile, the required body acceleration and body angular velocity of the UAV are calculated. Using the three-dimensional attitude matrix at the current moment, the acceleration and angular velocity in the body coordinate system are transformed into the world coordinate system. Using the Euler integral method, combined with the transformed world coordinate system acceleration and angular velocity, the speed, position and attitude of the UAV are updated at a fixed time step.

[0021] The disturbance compensator generates a random noise vector that conforms to a Gaussian distribution. This random noise vector is then superimposed on the ideal world acceleration calculated by the dynamics updater. The position and velocity are updated again using the Euler integral with the acceleration after environmental disturbance, generating the cooperative flight trajectory of the UAV swarm in three-dimensional space and deducing the global state vector of the UAV swarm at the next moment.

[0022] In this scheme, step S104 involves designing a multi-dimensional reward function to calculate the reward value reflecting the RCS collaborative effect in real time. This includes: acquiring the RCS data of the master UAV and slave UAVs at the current moment; designing a multi-dimensional reward function by comprehensively considering similarity, concentration, uniformity, and smoothness indicators; and calculating the arithmetic mean of the RCS values ​​of all slave UAVs in the similarity evaluation. Calculate the RCS value of the main UAV Compared with the average RCS value of drones absolute error Calculate the reward based on the absolute error. ;

[0023] In the centralized assessment, the standard deviation of all RCS values ​​from the drone is calculated. The 3-sigma rule is applied to determine the reward; if the RCS value of the main drone is... If the RCS value is less than the upper boundary of the distribution of drone RCS values, a positive reward is given. ,

[0024] Additionally, constraints are placed on the variance of the drone swarm, such as the standard deviation of the RCS value of the drones. Punishment Otherwise, a reward will be given. ;

[0025] In the uniformity assessment, the range of the UAV's RCS value at the current moment is calculated. Penalties are set according to the range. Historical data Compared with the current RCS value of the drone merge Calculate the overall range and apply a penalty. ; Calculate the skewness of the merged data and kurtosis To measure the asymmetry and steepness of the distribution, a penalty is generated. and , Obtain the total uniformity reward ;

[0026] In the smoothness assessment, the differences between the average RCS values ​​of the master drone and the slave drone over the time series are calculated separately, and the standard deviations of the differences between the master drone and the slave drone are also calculated separately. and The smoothness of RCS changes for both the master and slave drones is penalized based on the differential standard deviation. .

[0027] Bonus main drone RCS value Below all situations where drones offer a tactical advantage, if ,but ,otherwise ;

[0028] The reward values ​​for similarity, concentration, uniformity, and smoothness indicators, along with tactical advantages, are weighted and summed to obtain the reward value for the current time step: .

[0029] In this scheme, in step S108, the reward value is used to iteratively update the action policy network through a reinforcement learning mechanism. The updated action policy network is then used to iteratively optimize the RCS cooperative control strategy of the master-slave agent, including:

[0030] The reward value, the current global state vector, the action vector, and the next global state vector are stored as experience data in the replay buffer.

[0031] When the action policy network is updated, empirical data is randomly sampled from the replay buffer for optimization. The action policy network is used to calculate the target action based on the global state vector at the next moment, and random noise is added during the calculation process.

[0032] Two target critic networks are used to calculate two target Q values ​​based on the global state vector and target action at the next time step, and the minimum of the two values ​​is taken as the final target Q value.

[0033] Two online critic networks are used to estimate the current Q value based on the current empirical data. The difference between the target Q value and the current Q value is used as the loss function, and the two online critic networks are updated simultaneously using gradient descent.

[0034] After the critic network is updated, the performance gradient of the action policy network is calculated, and the parameters of the action policy network are fine-tuned along the direction that can improve the Q value given by the critic network.

[0035] Through iterative processing of random experience data, the action policy network learns to generate the highest-reward RCS adjustment instruction defined by a multidimensional reward function, thereby optimizing the RCS cooperative control strategy of the master-slave agents.

[0036] The second aspect of this invention provides a multi-UAV radar cross section optimization system based on reinforcement learning. The system includes: a master-slave UAV cooperative control module, an RCS optimization decision module, a three-dimensional dynamic trajectory modeling module, and a multi-dimensional RCS feature similarity optimization reward module.

[0037] The master-slave drone collaborative control module: constructs a master-slave intelligent agent architecture, sets one drone as the master drone, uses the radar cross section (RCS) value of the master drone as the benchmark, and the other drones as slave drones, and initializes the state and RCS parameters of all drones.

[0038] The RCS optimization decision module consists of three parts: a state space encoder, an action policy network, and a dual critic network. It receives raw state data from the UAV swarm and encodes it into state vectors using the state space encoder. The action policy network calculates continuous action vectors and parses them into RCS adjustment actions. Finally, the dual critic network evaluates the value of the decision and trains and updates the network using the TD3 algorithm.

[0039] The three-dimensional dynamic trajectory modeling module consists of three parts: attitude representation, dynamic updater and disturbance compensator. It models the attitude of the UAV based on the Lie group SO(3) theory, and dynamically simulates and generates the cooperative flight trajectory of the UAV swarm in three-dimensional space by combining the Euler integral method with environmental disturbance.

[0040] The multidimensional RCS feature similarity optimization reward module designs a multidimensional reward function by integrating similarity, concentration, uniformity and smoothness indicators, calculates the reward value that reflects the RCS cooperative effect, and uses the reward value to cyclically update the policy network through a reinforcement learning mechanism, and finally trains a cooperative control policy that can keep the RCS features of the UAV swarm consistent.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] 1. Highly efficient group RCS collaborative optimization effect: Through the design of master-slave intelligent agent collaborative control architecture and TD3 algorithm, this invention can achieve highly consistent optimization of the RCS characteristics of UAV groups. Compared with traditional single-entity control methods, the group stealth effect is significantly improved, and the RCS feature similarity can reach more than 95%, effectively reducing the probability of radar detection and thus improving the stealth performance of UAV groups.

[0043] 2. Strong adaptability of three-dimensional spatial cooperation: Since the present invention adopts a three-dimensional trajectory modeling module based on Lie group theory, it can accurately simulate the cooperative flight of UAV groups in complex three-dimensional environments. It takes into account multiple factors such as gravity and disturbance, and the trajectory prediction accuracy is improved by more than 30% compared with traditional methods. It can maintain a stable cooperative flight state in complex environments, thereby achieving precise control of RCS features.

[0044] 3. Precise multi-dimensional reward optimization mechanism: Due to the design of the multi-dimensional RCS feature similarity optimization reward module, this invention achieves comprehensive optimization of similarity, concentration, uniformity and smoothness, improves network convergence speed by 40%, and significantly enhances training stability. This enables the UAV swarm to maintain stealth performance while possessing good flight stability and coordination, and can achieve long-term stable cooperative stealth flight. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments or examples of the present invention, the drawings used in the embodiments or examples will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained according to these drawings without creative effort.

[0046] Figure 1 A flowchart of a multi-UAV radar cross section optimization method based on reinforcement learning is shown;

[0047] Figure 2 The diagram shows the overall framework of the simulation environment initialization and reinforcement learning.

[0048] Figure 3 A block diagram of a multi-UAV radar cross section optimization system based on reinforcement learning is shown. Detailed Implementation

[0049] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0051] Figure 1 A flowchart of a multi-UAV radar cross section optimization method based on reinforcement learning is shown.

[0052] like Figure 1 As shown, this embodiment provides a multi-UAV radar cross section optimization method based on reinforcement learning, including:

[0053] S101: Construct a master-slave intelligent agent architecture, set one drone as the master drone, use the radar cross section (RCS) value of the master drone as the benchmark, and the other drones as slave drones, initialize the state and RCS parameters of all drones.

[0054] It should be noted that a master-slave UAV cooperative control architecture is constructed, consisting of a master agent controller, slave agent controllers, and a cooperative optimization coordinator. The UAV swarm is initialized, and the swarm is set to contain N UAVs. One UAV is designated as the master UAV, and the remaining N-1 UAVs are slave agents. The master agent controller is responsible for managing the RCS characteristics of the baseline UAV, which serves as the RCS benchmark reference for the entire UAV swarm. The slave agent controllers are responsible for adjusting the RCS characteristics of the other UAVs to dynamically adjust them to a level similar to that of the master agent. The cooperative optimization coordinator coordinates the RCS adjustment strategies between the master and slave agents, and achieves a high degree of consistency in the group's RCS characteristics through intelligent algorithms, thereby improving the overall stealth effect.

[0055] Load the predefined RCS lookup table and use the main intelligent agent controller to obtain the initial state data of the main UAV, which includes three-dimensional position coordinates. Three-dimensional velocity components And the three attitude angles of roll, pitch, and yaw. ),in This represents the three-dimensional spatial coordinates of the main UAV. This represents the velocity component of the main UAV. This represents the rotational attitude angle of the main UAV. The initial reference RCS value of the main UAV is obtained by querying the RCS lookup table using the initial state data. ,in This represents the radar cross-section (RCS) of the main UAV. The initial baseline RCS value is used as the RCS reference standard for the entire cluster and broadcast to the collaborative optimization coordinator.

[0056] like Figure 2As shown, real-time status data from the drones is acquired in parallel from the intelligent agent controller. For each drone... , arrive ,in This indicates that the location coordinates are obtained from the number of drones. velocity components and attitude angle subscript Indicates the first Each drone and its corresponding intelligent agent controller independently queries the RCS lookup table to obtain the current RCS value of the drone. ,in Let represent the radar cross-section of the i-th drone. The intelligent agent controller compares... With the received benchmark Calculate the adjustment deviation ,in Indicates the first The RCS deviation of the drone is used to generate preliminary adjustment action suggestions. ,in This represents the action values ​​from the drone (range: [0,1], subsequently mapped to [-3,13] dBsm). The collaborative optimization coordinator receives the baseline from the main agent controller. The TD3 algorithm is used for global coordination to address deviations from the agent controller.

[0057] S102: The state information of the UAV swarm is obtained in real time through the state space encoder and input into the action policy network based on the TD3 algorithm. The action policy network is used to output continuous action vectors to generate the real-time RCS adjustment of the master UAV and each slave UAV.

[0058] It should be noted that the RCS collaborative optimization module based on the TD3 algorithm consists of a state-space encoder, an action policy network, and a dual critic network. The state-space encoder stores the UAV's position coordinates. velocity components and attitude angle As state variables, they constitute a 9×N dimensional state space, where N is the total number of drones. The action policy network is designed with a continuous action space [0,1]. Action values ​​are converted into RCS adjustment parameters through linear mapping. The action mapping range of the master agent controller is [-13,13] dBsm, and the action mapping range of the slave agent controller is [-3,13] dBsm, allowing for finer RCS feature adjustment. The dual critic network adopts a dual Q-network structure to avoid Q-value overestimation. ,in and For two critic networks, For state, This is the action vector.

[0059] Obtain the flight trajectory data of the entire drone swarm at the current moment from the simulation environment. , shape is Each line contains: The subscript j represents the j-th UAV. A state-space encoder is used to obtain the global state vector corresponding to the flight trajectory data; the data is flattened into an 81-dimensional vector. ,in To comprehensively characterize the instantaneous state of the entire UAV swarm, a global state vector is generated, and the global state vector is standardized. ,in Let be the mean and standard deviation of the global state vector. Then, standardize the global state vector. The data is fed into an action policy network, which performs in-depth processing and analysis to understand the cooperative situation of the UAV swarm and outputs an N-dimensional continuous action vector. , For policy networks, The parameter is defined as follows: the first element of the action vector corresponds to the adjustment command of the main UAV. The remaining N-1 elements correspond to the adjustment commands from each drone. The commands are all continuous values ​​within the range of [0, 1], representing the intensity of the adjustment. The RCS adjustment amount of the main UAV is obtained by linearly mapping the action vector. and the RCS adjustment amount from the drone, ; ， Adjusted RCS value Input simulation environment and generate the next state. and rewards Stored in the playback buffer.

[0060] S103: The attitude of the UAV is modeled based on the Lie group SO(3) theory, and based on the action vector, the cooperative flight trajectory of the UAV swarm in three-dimensional space is dynamically simulated and generated by combining the Euler integral method with environmental disturbances.

[0061] It should be noted that the attitude of the UAV is modeled based on the Lie group SO(3) theory, and the initial state of the UAV (initial position, initial velocity, initial attitude angle range) and flight mission profile command (defining the body acceleration and angular velocity of different flight stages) are initialized to simulate the cooperative flight of the UAV swarm in complex three-dimensional space. The three-dimensional rotation is represented by the Lie group SO(3), and the pitch angle and yaw angle are randomly generated. The dynamic updater uses Euler integral to update the position and velocity, and the disturbance compensator considers gravity and Gaussian noise.

[0062] The attitude representation receives motion vectors output from the action policy network, and each UAV receives target attitude change commands based on the pitch and yaw angles of these motion vectors. , For drones Generate SO(3) elements for the roll, pitch, and yaw attitude angles:

[0063] ,

[0064] ,

[0065] ,

[0066] in , Indicates rotation around the Y-axis Angle, rotation around the Z-axis angle;

[0067] Based on the target attitude change command, the three-dimensional attitude matrix of the target is calculated for each UAV according to the Lie group SO(3) theory. ;

[0068] Based on the target's three-dimensional attitude matrix, a dynamics updater is used to obtain the difference between the current attitude and the target attitude. According to the preset flight mission profile, the required body acceleration of the UAV is calculated. and body angular velocity Using the current 3D attitude matrix, the acceleration and angular velocity in the body coordinate system are transformed to the world coordinate system. Using the Euler integral method, combined with the transformed world coordinate system acceleration and angular velocity, the UAV's velocity, position, and attitude are updated at fixed time steps; the target's 3D attitude matrix is ​​received. The velocity vector measured in the body coordinate system The position of the drone in the world coordinate system Each step Updated in seconds.

[0069] First 400 steps: ,

[0070] The last 100 steps: ,

[0071] The Lie algebra representation of tiny rotations occurring within a time step: The Lie algebra representation that updates the current pose. ;

[0072] Calculate world acceleration ,in ,in To perform the exponential mapping operation, It is represented in the form of a standard 3x3 rotation matrix.

[0073] The perturbation compensator generates a random noise vector that conforms to a Gaussian distribution. The random noise vector is superimposed onto the ideal world acceleration calculated by the dynamics updater. The position and velocity are then updated again using Euler integrals based on the acceleration after environmental perturbation, generating the cooperative flight trajectory of the UAV swarm in three-dimensional space and predicting the global state vector of the UAV swarm at the next moment. To adjust the RCS, the UAVs may need to change their attitude. Attitude change commands are transmitted as real-time input to the dynamics updater. The dynamics updater calculates the corresponding body acceleration and angular velocity based on the new target attitude or angular velocity, thereby updating the trajectory.

[0074] S104: Design a multidimensional reward function to calculate the reward value that reflects the RCS cooperative effect in real time. Using the reward value, update the action policy network cyclically through a reinforcement learning mechanism. Use the updated action policy network to iteratively optimize the RCS cooperative control strategy of the master-slave agent.

[0075] It should be noted that, by acquiring the RCS data of the master and slave drones at the current moment, a multi-dimensional reward function is designed by comprehensively considering similarity, concentration, uniformity, and smoothness indicators. In the similarity evaluation, the arithmetic mean of the RCS values ​​of all slave drones is calculated. , ; Calculate the RCS value of the main UAV Compared with the average RCS value of drones absolute error Calculate the reward based on the absolute error. ;

[0076] In the centralized assessment, the standard deviation of all RCS values ​​from the drone is calculated. The 3-sigma rule is applied to determine the reward; if the RCS value of the main drone is... Less than the upper boundary of the RCS value distribution of the UAV. A positive reward will be given. Additionally, constraints are placed on the variance of the drone swarm, such as the standard deviation of the RCS value of the drones. Punishment Otherwise, a reward will be given. ;

[0077] In the uniformity assessment, the range of the UAV's RCS value at the current moment is calculated. Penalties are set according to the range. Historical data Compared with the current RCS value of the drone merge Calculate the overall range and apply a penalty. ; Calculate the standard deviation of the merged data ; Calculate the skewness of the merged data and kurtosis To measure the asymmetry and steepness of the distribution, , ;

[0078] Skewness measures the asymmetry of a distribution. A larger absolute value of skewness indicates a more severe left or right skewness in the distribution, resulting in a penalty. , Kurtosis measures the steepness of a distribution. Excessively high kurtosis indicates an overly concentrated distribution, potentially leading to outliers and generating penalties. , Obtain the total uniformity reward ;

[0079] In the smoothness assessment, the difference between the average RCS values ​​of the master UAV and the slave UAV over the time series is calculated separately. Calculate the standard deviation of the difference between the master UAV and the slave UAV respectively. and The smoothness of RCS changes for both the master and slave drones is penalized based on the differential standard deviation. .

[0080] Bonus main drone RCS value Below all situations where drones offer a tactical advantage, if ,but ,otherwise ;

[0081] The reward values ​​for similarity, concentration, uniformity, and smoothness indicators, along with tactical advantages, are weighted and summed to obtain the reward value for the current time step: .

[0082] The reward value, the current global state vector, the action vector, and the next global state vector are stored as experience data in the replay buffer. When the action policy network is updated, experience data is randomly sampled from the replay buffer for optimization. The action policy network is used to calculate the target action based on the next global state vector, and random noise is added during the calculation process.

[0083] Two target critic networks are used, each based on the global state vector at the next time step. and target action Calculate two target Q values, and take the minimum of the two values ​​as the final target Q value; , This is the current reward value. As a discount factor, For network parameters;

[0084] The current Q value was estimated using two online networks of critics based on current empirical data. The loss function is based on the difference between the target Q-value and the current Q-value. Two online critic networks are updated simultaneously using gradient descent.

[0085] The current Q value was estimated using two online networks of critics based on current empirical data. The loss function is based on the difference between the target Q-value and the current Q-value. Two online critic networks are updated simultaneously using gradient descent.

[0086] After the critic network is updated, the performance gradient of the action policy network is calculated. , The performance gradient of the action policy network. , For gradient operators, This indicates that the action policy network is in state The specific actions to be output are as follows: The parameters of the action policy network are defined as follows: The parameters of the action policy network are fine-tuned along directions that improve the Q-value given by the critic network; through iterative processing of random experience data, the action policy network learns to generate RCS adjustment instructions with the highest reward defined by a multidimensional reward function, thus optimizing the RCS cooperative control strategy of the master-slave agents.

[0087] Figure 3 The architecture diagram of a multi-UAV radar cross section optimization system based on reinforcement learning is shown.

[0088] The second embodiment of the present invention provides a multi-UAV radar cross section optimization system based on reinforcement learning. The system includes: a master-slave UAV cooperative control module, an RCS optimization decision module, a three-dimensional dynamic trajectory modeling module, and a multi-dimensional RCS feature similarity optimization reward module.

[0089] The master-slave UAV cooperative control module constructs a master-slave agent architecture, designating one UAV as the master UAV, using its radar cross section (RCS) value as the benchmark, and the remaining UAVs as slave UAVs. It initializes the states and RCS parameters of all UAVs. The master agent controller is responsible for maintaining the RCS value as the group benchmark; the slave agent controllers are responsible for adjusting their respective UAVs to align their RCS with the master UAV. A cooperative optimization coordinator aggregates all information, uses reinforcement learning algorithms to calculate and generate globally optimal cooperative control commands, ensuring a high degree of consistency in the RCS characteristics of the entire UAV swarm.

[0090] The RCS optimization decision module consists of three parts: a state-space encoder, an action policy network, and a dual critic network. It receives raw state data from the UAV swarm and encodes it into state vectors using the state-space encoder. The action policy network calculates continuous action vectors and parses them into RCS adjustment actions. Finally, the dual critic network evaluates the value of the decisions, and the network is trained and updated using the TD3 algorithm. This solves the problems of poor real-time performance and inaccurate decision-making in traditional optimization methods in dynamic environments, achieving efficient and stable intelligent optimization.

[0091] The three-dimensional dynamic trajectory modeling module consists of three parts: attitude representation, dynamic updater and disturbance compensator. It models the attitude of the UAV based on the Lie group SO(3) theory, avoids the "universal deadlock" problem that may occur in traditional Euler angles, and thus simulates flight mechanics more realistically. Based on the action vector, it dynamically simulates and generates the cooperative flight trajectory of the UAV swarm in three-dimensional space by combining the Euler integral method with environmental disturbances.

[0092] The multi-dimensional RCS feature similarity optimization reward module designs a multi-dimensional reward function by integrating similarity, concentration, uniformity, and smoothness indices. It calculates reward values ​​reflecting the collaborative effect of RCS. Similarity ensures that the mean RCS values ​​of slave and master drones are close; concentration ensures that the RCS values ​​among slave drones are as concentrated as possible, avoiding dispersion; uniformity ensures a good RCS distribution pattern among slave drones; and smoothness requires that the RCS values ​​change smoothly over time, avoiding abrupt changes. Using these reward values, the policy network is iteratively updated through a reinforcement learning mechanism, ultimately training a collaborative control strategy that maintains consistent RCS features in the drone swarm.

[0093] The third embodiment of the present invention provides a computer-readable storage medium, which includes a program for a multi-UAV radar cross section optimization method based on reinforcement learning. When the program for the multi-UAV radar cross section optimization method based on reinforcement learning is executed by a processor, it implements the steps of the multi-UAV radar cross section optimization method based on reinforcement learning.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms. Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0095] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing the radar cross section of multiple unmanned aerial vehicles (UAVs) based on reinforcement learning, characterized in that, Includes the following steps: S101: Construct a master-slave intelligent agent architecture, set one drone as the master drone, use the radar cross section (RCS) value of the master drone as the benchmark, and the other drones as slave drones, initialize the state and RCS parameters of all drones. S102: The state information of the UAV swarm is obtained in real time through the state space encoder and input into the action policy network based on the TD3 algorithm. The action policy network is used to output continuous action vectors to generate the real-time RCS adjustment of the master UAV and each slave UAV. S103: The attitude of the UAV is modeled based on the Lie group SO(3) theory, and based on the action vector, the cooperative flight trajectory of the UAV swarm in three-dimensional space is dynamically simulated and generated by combining the Euler integral method with environmental disturbances. S104: Design a multidimensional reward function to calculate the reward value that reflects the RCS cooperative effect in real time. Using the reward value, update the action policy network cyclically through a reinforcement learning mechanism. Use the updated action policy network to iteratively optimize the RCS cooperative control strategy of the master-slave agent. In step S102, state coding and RCS collaborative optimization decision-making are performed based on the TD3 algorithm, including: The flight trajectory data of the entire UAV swarm at the current moment is obtained from the simulation environment. The global state vector corresponding to the flight trajectory data is obtained using a state space encoder, and the global state vector is standardized. The standardized global state vector is fed into the action policy network, which is then used for in-depth processing and analysis to understand the cooperative situation of the UAV swarm and output an N-dimensional continuous action vector. The first element of the action vector corresponds to the adjustment command of the master UAV, and the remaining N-1 elements correspond to the adjustment commands of each slave UAV. The motion vectors are linearly mapped to obtain the RCS adjustment values ​​of the master UAV and the slave UAV respectively, and the RCS adjustment values ​​of all UAVs are applied to the simulation environment. In step S103, the three-dimensional dynamic trajectory is generated and updated based on Lie group theory, including: The attitude of the UAV is modeled based on the Lie group SO(3) theory. The attitude representation device receives the action vector output from the action policy network. According to the action vector, the target attitude change command of each UAV is given for pitch angle and yaw angle. According to the target attitude change command, the target three-dimensional attitude matrix is ​​calculated for each UAV based on the Lie group SO(3) theory. Based on the target three-dimensional attitude matrix, the difference between the current attitude and the target attitude is obtained using a dynamics updater. According to the preset flight mission profile, the required body acceleration and body angular velocity of the UAV are calculated. Using the three-dimensional attitude matrix at the current moment, the acceleration and angular velocity in the body coordinate system are transformed into the world coordinate system. Using the Euler integral method, combined with the transformed world coordinate system acceleration and angular velocity, the speed, position and attitude of the UAV are updated at a fixed time step. The disturbance compensator generates a random noise vector that conforms to a Gaussian distribution. This random noise vector is then superimposed on the ideal world acceleration calculated by the dynamics updater. The position and velocity are updated again using the Euler integral with the acceleration after environmental disturbance, generating the cooperative flight trajectory of the UAV swarm in three-dimensional space and deducing the global state vector of the UAV swarm at the next moment.

2. The method for optimizing the radar cross section of multiple unmanned aerial vehicles based on reinforcement learning according to claim 1, characterized in that, In step S101, a master-slave agent architecture is constructed, with one drone designated as the master drone and its radar cross section (RCS) value used as a reference. The remaining drones are designated as slave drones. The states and RCS parameters of all drones are initialized, including: Construct a master-slave UAV cooperative control architecture consisting of a master intelligent agent controller, slave intelligent agent controllers and a cooperative optimization coordinator. Initialize the UAV swarm, set the UAV swarm to contain N UAVs, designate one UAV as the master UAV and the remaining N-1 UAVs as slave intelligent agents. Load the predefined RCS lookup table, use the main intelligent agent controller to obtain the initial state data of the main UAV. The initial state data includes three-dimensional position coordinates, three-dimensional velocity components, and three attitude angles: roll, pitch, and yaw. Use the initial state data to query the RCS lookup table to obtain the initial reference RCS value of the main UAV. Use the initial reference RCS value as the RCS reference standard for the entire cluster and send it to the collaborative optimization coordinator. The system uses a parallel acquisition of real-time status data from the drones by the agent controllers. Each agent controller independently queries the RCS lookup table to obtain the current RCS value of the corresponding drone.

3. The method for optimizing the radar cross section of multiple unmanned aerial vehicles based on reinforcement learning according to claim 1, characterized in that, In step S104, a multidimensional reward function is designed to calculate the reward value reflecting the RCS synergy effect in real time, including: Obtain the RCS data of the master and slave drones at the current moment, and design a multi-dimensional reward function that integrates similarity, concentration, uniformity, and smoothness indicators. In the similarity evaluation, calculate the arithmetic mean of the RCS values ​​of all slave drones. Calculate the RCS value of the main UAV Compared with the average RCS value of drones absolute error Calculate the reward based on the absolute error. ; In the centralized assessment, the standard deviation of all RCS values ​​from the drone is calculated. The 3-sigma rule is applied to determine the reward; if the RCS value of the main drone is... If the value is less than the upper boundary of the drone's RCS value distribution, a positive reward is given. Additionally, constraints are placed on the variance of the drone swarm, such as the standard deviation of the RCS value of the drones. Punishment Otherwise, a reward will be given. ; In the uniformity assessment, the range of the UAV's RCS value at the current moment is calculated. Penalties are set according to the range. Historical data Compared with the current RCS value of the drone merge Calculate the overall range and apply a penalty. ; Calculate the skewness of the merged data and kurtosis To measure the asymmetry and steepness of the distribution, a penalty is generated. and , Obtain the total uniformity reward ; In the smoothness assessment, the differences between the average RCS values ​​of the master drone and the slave drone over the time series are calculated separately, and the standard deviations of the differences between the master drone and the slave drone are also calculated separately. and The smoothness of RCS changes for both the master and slave drones is penalized based on the differential standard deviation. .

4. The method for optimizing the radar cross section of multiple unmanned aerial vehicles based on reinforcement learning according to claim 3, characterized in that, Bonus main drone RCS value Below all situations where drones offer a tactical advantage, if ,but ,otherwise ; The reward values ​​for similarity, concentration, uniformity, and smoothness indicators, along with tactical advantages, are weighted and summed to obtain the reward value for the current time step: .

5. The method for optimizing the radar cross section of multiple unmanned aerial vehicles based on reinforcement learning according to claim 1, characterized in that, In step S104, the reward value is used to iteratively update the action policy network through a reinforcement learning mechanism. The updated action policy network is then used to iteratively optimize the RCS cooperative control policy of the master-slave agent, including: The reward value, the current global state vector, the action vector, and the next global state vector are stored as experience data in the replay buffer. When the action policy network is updated, empirical data is randomly sampled from the replay buffer for optimization. The action policy network is used to calculate the target action based on the global state vector at the next moment, and random noise is added during the calculation process. Two target critic networks are used to calculate two target Q values ​​based on the global state vector and target action at the next time step, and the minimum of the two values ​​is taken as the final target Q value. Two online critic networks are used to estimate the current Q value based on the current empirical data. The difference between the target Q value and the current Q value is used as the loss function, and the two online critic networks are updated simultaneously using gradient descent. After the critic network is updated, the performance gradient of the action policy network is calculated, and the parameters of the action policy network are fine-tuned along the direction that can improve the Q value given by the critic network. Through iterative processing of random experience data, the action policy network learns to generate the highest-reward RCS adjustment instruction defined by a multidimensional reward function, thereby optimizing the RCS cooperative control strategy of the master-slave agents.

6. A multi-UAV radar cross section optimization system based on reinforcement learning, characterized in that, To implement the reinforcement learning-based multi-UAV radar cross section optimization method as described in any one of claims 1-5, the system includes: a master-slave UAV cooperative control module, an RCS optimization decision module, a three-dimensional dynamic trajectory modeling module, and a multi-dimensional RCS feature similarity optimization reward module; The master-slave drone collaborative control module: constructs a master-slave intelligent agent architecture, sets one drone as the master drone, uses the radar cross section (RCS) value of the master drone as the benchmark, and the other drones as slave drones, and initializes the state and RCS parameters of all drones. The RCS optimization decision module consists of three parts: a state space encoder, an action policy network, and a dual critic network. It receives raw state data from the UAV swarm and encodes it into state vectors using the state space encoder. The action policy network calculates continuous action vectors and parses them into RCS adjustment actions. Finally, the dual critic network evaluates the value of the decision and trains and updates the network using the TD3 algorithm. The three-dimensional dynamic trajectory modeling module consists of three parts: attitude representation, dynamic updater and disturbance compensator. It models the attitude of the UAV based on the Lie group SO(3) theory, and dynamically simulates and generates the cooperative flight trajectory of the UAV swarm in three-dimensional space by combining the Euler integral method with environmental disturbance. The multidimensional RCS feature similarity optimization reward module designs a multidimensional reward function by integrating similarity, concentration, uniformity and smoothness indicators, calculates the reward value that reflects the RCS cooperative effect, and uses the reward value to cyclically update the policy network through a reinforcement learning mechanism, and finally trains a cooperative control policy that can keep the RCS features of the UAV swarm consistent.

Images