Multi-Agent Reinforcement Learning Based Anti-Drone Control System and Method Thereof

Abstract

One embodiment of the present invention provides a multi-agent reinforcement learning-based anti-drone control system comprising: a plurality of interceptor drones, each of which observes state information including enemy drone location, friendly drone location, and remaining energy within a limited sensor range and performs control actions according to a local Q network distributed from a central learning server; a communication relay node that relays communication between the plurality of interceptor drones and the central learning server; and a central learning server that trains the local Q network of each drone using experience data collected from the plurality of interceptor drones and a preset anti-drone specialized reward function, trains a mixing network that receives individual Q values ​​output by the local Q networks during the training process and calculates a global Q value, and distributes the trained local Q network to each drone.

Description

Technology Field

[0001] The present invention relates to a multi-agent reinforcement learning-based anti-drone system, and more specifically, to an anti-drone control system and method for neutralizing enemy drones using a plurality of interceptor drones. Background Technology

[0003] With the recent rapid advancement and widespread adoption of drone technology, various application fields utilizing drones are expanding. Along with this, the importance of anti-drone technology to counter misuse and unauthorized intrusion is increasing, and various anti-drone systems have been developed.

[0004] Conventional anti-drone systems primarily utilize countermeasures based on a single interception method. For example, jamming technology, net launching, and laser-based neutralization methods are known. Furthermore, classical control theories such as PID (Proportional-Integral-Derivative) control and MPC (Model Predictive Control) are widely used for drone flight control.

[0005] Meanwhile, alongside the advancement of artificial intelligence technology, research is underway to apply reinforcement learning to drone control. In particular, Multi-Agent Reinforcement Learning (MARL), a technique in which multiple agents interact with the environment to learn, is being researched in various fields such as robotics, game AI, and autonomous driving.

[0006] In the field of anti-drone warfare, it is common practice to counter multiple enemy drones with a single interceptor; even when multiple interceptors are operated, centralized control or pre-programmed tactics are primarily adopted. Furthermore, in actual battlefield environments, various environmental factors such as communication constraints, limited sensor range, and electronic warfare situations are known to be significant considerations. The problem to be solved

[0008] The technical problem that the present invention aims to solve is to provide an anti-drone control system in which multiple interceptor drones can effectively cooperate to neutralize multiple target drones even in partial observation environments and limited communication conditions.

[0009] In addition, the present invention aims to provide a distributed control technology that enables each interceptor drone to perform optimized actions at the overall system level using only local information within a limited sensor range.

[0010] In addition, the present invention aims to provide a scalable anti-drone system capable of real-time response while maintaining stable performance even in situations involving an increase in the number of drones or communication failures.

[0011] In addition, the present invention aims to provide an anti-drone control method capable of improving overall mission performance by simultaneously considering energy efficiency and multi-target capture performance.

[0012] The technical problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0014] To achieve the above technical objectives, one embodiment of the present invention provides a multi-agent reinforcement learning-based anti-drone control system comprising: a plurality of interceptor drones, each of which observes state information including enemy drone location, friendly drone location, and remaining energy within a limited sensor range and performs control actions according to a local Q network distributed from a central learning server; a communication relay node that relays communication between the plurality of interceptor drones and the central learning server; and a central learning server that trains the local Q network of each drone using experience data collected from the plurality of interceptor drones and a preset anti-drone specialized reward function, trains a mixing network that receives individual Q values ​​output by the local Q networks during the training process and calculates a global Q value, and distributes the trained local Q network to each drone.

[0015] In an embodiment of the present invention, the mixing network is configured to satisfy a monotonicity constraint such that the global Q value does not decrease when the individual Q value increases, and the monotonicity constraint can be implemented by applying an activation function to the weight parameters of the mixing network so that all weights become non-negative.

[0016] In an embodiment of the present invention, each interceptor drone can independently determine a 3-axis control action including pitch, yaw, and roll according to the local Q network using only partial observation information within its sensor range.

[0017] In an embodiment of the present invention, the anti-drone specialized reward function includes an individual reward based on distance to an enemy drone and a common reward based on the team-wide neutralization performance, wherein

[0018] The above individual rewards are designed to have higher values ​​as the distance to enemy drones increases and as the remaining energy increases, and the above common rewards can be configured to be calculated as a value normalized by the number of drones for the neutralization performance of the entire team.

[0019] In an embodiment of the present invention, the central learning server stores a set of experience tuples from the plurality of interceptor drones in a pre-prepared experience replay buffer, constructs training data by randomly sampling the set of experience tuples stored in the memory by a batch size, and can update the mixing network based on the training data and a pre-prepared target network.

[0020] In an embodiment of the present invention, each interceptor drone further includes a net launching device for non-destructively capturing enemy drones, and the control action may further include determining the net launching timing.

[0021] To achieve the above technical objectives, another embodiment of the present invention provides a multi-agent reinforcement learning-based anti-drone control method comprising: a step in which each of a plurality of interceptor drones observes state information including the location of an enemy drone, the location of an allied drone, and remaining energy within a limited sensor range; a step in which a central learning server collects experience data from the plurality of interceptor drones through a communication relay node; a step in which the central learning server trains a local Q network of each drone using the collected experience data and a preset anti-drone specialized reward function, and trains a mixing network that receives individual Q values ​​output by the local Q networks during the training process and calculates a global Q value; a step in which the central learning server distributes the trained local Q networks to each interceptor drone; and a step in which each interceptor drone performs a control action according to the distributed local Q network.

[0022] In an embodiment of the present invention, the training step includes configuring the mixing network to satisfy a monotonic constraint such that the global Q value does not decrease when the individual Q value increases, and the monotonic constraint can be implemented by applying an activation function to the weight parameters of the mixing network so that all weights become non-negative.

[0023] In an embodiment of the present invention, the observing step includes each interceptor drone using only partial observation information within its sensor range, and the step of performing the control action may include each interceptor drone independently determining and performing a 3-axis control action including pitch, yaw, and roll according to the local Q network.

[0024] In an embodiment of the present invention, the learning step comprises pre-setting the anti-drone specialized reward function to include an individual reward based on distance to the enemy drone and a common reward based on the team's overall neutralization performance. The anti-drone reward function is designed such that the individual reward has a higher value as the distance to the enemy drone decreases and as the remaining energy increases, and the common reward is configured to be calculated as a value normalized by the number of drones for the team's overall neutralization performance. Effects of the invention

[0026] According to an embodiment of the present invention, multiple interceptor drones can effectively cooperate to neutralize a target drone even in a partial observation environment and under limited communication conditions.

[0027] In addition, the present invention can secure real-time response capabilities through centralized learning and a distributed execution structure.

[0028] Furthermore, through the neural network structure with monotonic constraints of the present invention, stable learning is possible where the performance improvement of individual drones directly leads to the performance improvement of the entire system. This guarantees convergence even in complex multi-agent environments and enables adaptive learning for new tactical patterns.

[0029] Furthermore, by employing a non-destructive capture mechanism using a net launching method, the present invention can be safely operated even in urban areas. Since the captured drone can be reused and evidence preservation is easy, it can be applied to various civilian and military applications.

[0030] Furthermore, the present invention can maintain stable performance even in situations where some interceptor drones are damaged or communication is lost.

[0031] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description of the invention or the claims. Brief explanation of the drawing

[0033] FIG. 1 is a diagram showing a battlefield environment to which a multi-agent reinforcement learning-based anti-drone control system according to one embodiment of the present invention is applied. FIG. 2 is a block diagram showing the overall configuration of a multi-agent reinforcement learning-based anti-drone control system according to one embodiment of the present invention. FIG. 3 is a diagram showing the yaw axis maneuvering operation of an interceptor drone according to one embodiment of the present invention. FIG. 4 is a diagram showing the pitch axis maneuvering operation of an interceptor drone according to one embodiment of the present invention. FIG. 5 is a diagram showing the roll axis maneuvering operation of an interceptor drone according to one embodiment of the present invention. FIG. 6 is a block diagram showing a detailed connection configuration between a central learning server and an interceptor drone according to one embodiment of the present invention. FIG. 7 is a flowchart illustrating the learning process of a multi-agent reinforcement learning-based anti-drone control method according to one embodiment of the present invention. FIG. 8 is a graph showing the change in the normalized total reward value according to the training episode of an anti-drone system according to one embodiment of the present invention. FIG. 9 is a graph showing the change in compensation values ​​per episode of an individual interceptor drone according to one embodiment of the present invention. FIG. 10 is a graph showing the change in residual energy over time of an interceptor drone according to one embodiment of the present invention. FIG. 11 is a box plot showing the distribution of the number of enemy drones captured by an interceptor drone trained with an embodiment of the present invention and a comparison algorithm. Specific details for implementing the invention

[0034] The present invention is susceptible to various modifications and may take various forms; therefore, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the invention to the specific disclosed forms, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals have been used for similar components in the description of each drawing.

[0035] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0036] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings.

[0037] FIG. 1 is a diagram showing a battlefield environment to which a multi-agent reinforcement learning-based anti-drone control system according to one embodiment of the present invention is applied.

[0038] Referring to FIG. 1, the multi-agent reinforcement learning-based anti-drone control system of the present invention (hereinafter, anti-drone control system) illustrates a scenario in which multiple interceptor drones capture multiple hostile drones in a battlefield environment. The interceptor drones are indicated by blue arrows in the figure, and the hostile drones are indicated by red target marks. Additionally, control drones for system operation are indicated by sky-blue radio wave icons, and the energy status of each drone is indicated by a green battery icon.

[0039] The main objectives of the anti-drone control system according to the present invention are (i) to maximize the number of captured enemy drones and (ii) to utilize energy in an efficient manner. As illustrated in FIG. 1, interceptor drones must use a net to capture as many enemy drones as possible. To this end, the interceptor drones must use energy with maximum efficiency to maximize residual energy.

[0040] For example, if a single interceptor drone exhausts all remaining energy after capturing the first enemy drone, it becomes unable to capture subsequent enemy drones. Therefore, capturing the maximum number of enemy drones while optimally conserving remaining energy is the core objective of the present invention.

[0041] The battlefield environment of Fig. 1 is set against the backdrop of a military base in desert terrain and represents a complex environment with various obstacles such as buildings, communication towers, and military vehicles. In this environment, interceptor drones must effectively cooperate to perform missions even under limited sensor range and communication constraints. In particular, each interceptor drone must be able to make autonomous decisions even in electronic warfare environments or situations involving communication jamming.

[0042] The present invention enables cooperative mission execution among interceptor drones by utilizing Multi-Agent Reinforcement Learning (MARL) in such a complex battlefield environment. As can be seen in FIG. 1, a sophisticated cooperative strategy is required for multiple interceptor drones to respond to enemy drones approaching from multiple directions simultaneously.

[0043] In this scenario, each interceptor drone acts based on individually learned intelligence, yet is designed to achieve optimized cooperation at the overall system level. This approach differentiates itself from existing centralized control methods or pre-programmed tactics, offering the advantage of adaptively responding to the unpredictable maneuvering patterns of enemy drones.

[0044] FIG. 2 is a block diagram showing the overall configuration of a multi-agent reinforcement learning-based anti-drone control system (10) according to one embodiment of the present invention.

[0045] Referring to FIG. 2, the anti-drone control system (10) includes a plurality of interceptor drones (100) that observe state information including enemy drone location, friendly drone location, and remaining energy within a limited sensor range and perform control actions according to a local Q network distributed from a central learning server (300), a communication relay node (200) that relays communication between the plurality of interceptor drones (100) and the central learning server (300), and a central learning server (300) that performs learning using experience data collected from the plurality of interceptor drones (100).

[0046] Each interceptor drone (100) can acquire only partial observation information within its limited sensor range, for example, a radius of 50 m. This partial observation information includes the location of enemy drones (50), the location of friendly drones, and its remaining energy information. The interceptor drone (100) determines control actions through a local Q network so that it can make effective decisions with only this limited information.

[0047] The communication relay node (200) is a component for overcoming communication constraints in a battlefield environment and can operate reliably even in a limited communication environment where communication delay and packet loss occur. For example, the communication relay node (200) guarantees data transmission between the interceptor drone (100) and the central learning server (300) even in an environment with a communication delay of 100ms and a packet loss rate of 10%.

[0048] The central learning server (300) trains the local Q network of each drone using collected experience data and a pre-set anti-drone specialized reward function. For example, the anti-drone specialized reward function may include individual rewards based on distance from enemy drones (50) and common rewards based on the team's overall neutralization performance. In particular, during the process of training the local Q network of each drone, the central learning server (300) trains a mixing network that receives individual Q values ​​output by the local Q networks and calculates a global Q value. The central learning server (300) distributes the trained local Q networks to each drone. In this embodiment, the mixing network is configured to satisfy a monotonicity constraint that prevents the global Q value from decreasing when the individual Q value increases. The monotonicity constraint can be implemented by applying an activation function to the weight parameters of the mixing network so that all weights become non-negative. Specifically, a Softplus activation function w = log(1 + exp(w)) is applied to restrict all weight parameters to have a value greater than or equal to 0. This ensures the mathematical constraint ∂Q_total / ∂Q_i ≥ 0 (for all i), which means that an improvement in the performance of individual drones leads directly to an improvement in the performance of the entire team.

[0049] FIG. 3 is a diagram showing the yaw axis maneuvering operation of an interceptor drone according to one embodiment of the present invention.

[0050] FIG. 4 is a diagram showing the pitch axis maneuvering operation of an interceptor drone according to one embodiment of the present invention.

[0051] FIG. 5 is a diagram showing the roll axis maneuvering operation of an interceptor drone according to one embodiment of the present invention.

[0052] Referring to FIGS. 3 to 5, each interceptor drone (100) can independently determine a 3-axis control action including pitch, yaw, and roll according to a local Q network using only partial observation information within its sensor range.

[0053] As shown in FIG. 3, the yaw axis maneuver is a movement in which the drone rotates around the vertical axis (z-axis), V Ωm It is expressed as (t). This is used when the drone changes direction left or right, and, for example, can be controlled within a range of ±45°. The pitch axis maneuver shown in Fig. 4 is a movement in which the drone rotates around the lateral axis (y-axis), ψ Ωm It is expressed as (t). This is used when the drone tilts up and down, and, for example, can be controlled within a range of ±30°. The roll axis maneuver shown in Fig. 5 is a movement in which the drone rotates around the longitudinal axis (x-axis), φ Ωm It is expressed as (t). This is used when the drone tilts left or right, and, for example, can be controlled within a range of ±45°. The values ​​presented above are for illustrative purposes only, and the present invention is not limited to these numerical ranges.

[0054] This 3-axis control can be performed independently by each drone even in a limited communication environment, and the communication relay node (200) operates stably even in an environment where communication delay and packet loss occur, enabling cooperation between drones.

[0055] The anti-drone specialized reward function used by the central learning server (300) consists of individual rewards and common rewards. The individual reward is designed to have a higher value as the distance to the enemy drone (50) is closer and as the remaining energy is higher. For example, the individual reward can be configured in the form of R_individual = α·(1 / distance) + β·remaining energy, where α and β are weight parameters that control the importance of each.

[0056] The common reward is calculated by normalizing the total team's neutralization performance by the number of drones. Specifically, it can be expressed as R_common = (Number of captured enemy drones) / (Total number of friendly drones). This reward structure incentivizes the simultaneous optimization of individual drone performance and the team's overall cooperative performance.

[0057] The central learning server (300) stores a set of experience tuples from multiple interceptor drones (100) in a pre-prepared experience replay buffer. The experience tuples are configured in the form of (state, action, reward, next state), and, for example, up to 100,000 experience tuples can be stored.

[0058] During the training process, training data is constructed by randomly sampling a set of experience tuples stored in the experience replay buffer in batches of the specified size. For example, the batch size can be set to 32, which improves the stability of the training by reducing the correlation between temporally consecutive data.

[0059] Additionally, the central learning server (300) updates the local Q network and mixing network based on the training data and the pre-prepared target network. The target network is synchronized with the parameters of the main network at regular intervals, for example, every 500 steps, to provide stability to the learning goal.

[0060] Each interceptor drone (100) may further include a net launcher for non-destructively capturing enemy drones (50). In this case, the control action includes not only 3-axis maneuver control but also determination of the net launch timing. The use of a net launcher has the advantage of allowing safe operation in urban areas by capturing enemy drones (50) without destroying them, and enables analysis or reuse of the captured drones.

[0061] As described above, the multi-agent reinforcement learning-based anti-drone control system of the present invention operates effectively even in partial observation environments and under limited communication conditions, and can simultaneously achieve intelligent decision-making by individual drones and cooperation of the entire team.

[0062] FIG. 6 shows a mixing network training architecture for cooperative training between interceptor drones in a multi-agent reinforcement learning-based anti-drone control system according to one embodiment of the present invention.

[0063] Referring to FIG. 6, the training architecture of the present invention is largely composed of an environment, intercept drone networks, a mixing network, an experience replay buffer, and a target network.

[0064] The present invention adopts a Centralized Training Decentralized Execution (CTDE) structure. In the training phase, partial observation information of each drone (o from the environment) m Both (t) and global state information (s(t)) are collected and processed by a central learning server. On the other hand, during the execution phase, each interceptor drone independently decides its action using only its own partial observation information.

[0065] Specifically, the m-th interceptor drone has its observation information o m Receives (t) as input and the Q value through its local Q network Calculate . Here, s(t) is the global state, o m (t) is a partial observation, a m (t) represents the behavior of drone m.

[0066] The individual Q-values ​​output from M interceptor drones are input into the mixing network to form the global Q-value It produces. The mixing network can be expressed as a function like the following mathematical equation 1.

[0067] [Mathematical Formula 1]

[0068]

[0069] Here,

[0070] total(s(t), a(t); θ): joint action-value function

[0071] f(·): A mixing network function that combines individual action value functions under monotonicity constraints

[0072] s(t): Global state at the current time

[0073] a(t): Action vector of all drones, where a(t) = [a1(t), a2(t), ..., a M (t)]

[0074] θ: Parameter of the mixing network

[0075] m : Individual action value function of the m-th interceptor drone

[0076] o m (t): Partial observation of the m-th drone

[0077] a m (t): Action of the m-th drone

[0078] M: Total number of Interceptor Drones

[0079] arg max: Operator that selects the action with the maximum value

[0080] In particular, in this embodiment, the mixing network is configured to satisfy the monotonicity constraint of the following mathematical formula 2.

[0081] [Mathematical Formula 2]

[0082]

[0083] This constraint is the Q value of each individual drone ( m When ) increases, the Q value of the entire team ( It mathematically guarantees that the total does not decrease. To implement this, the weights of the mixing network are restricted to non-negative values. This is a key mechanism that ensures that performance improvement of individual agents in cooperative multi-agent systems does not lead to a degradation of overall system performance.

[0084] As illustrated in FIG. 6, hypernetworks (W1, W2) take the global state s(t) as input and generate weights for the mixing network. Each hypernetwork consists of a single linear layer and uses an absolute value activation function (e.g., ReLU or Softplus) to ensure non-negative weights. This is ∂ total / ∂ m Guarantees satisfaction of the constraint ≥ 0.

[0085] The Experience Replay Buffer is a fixed-size circular data structure that stores the experience tuples of each interceptor drone. Each experience tuple is composed as shown in the following Equation 3.

[0086] [Mathematical Formula 3]

[0087] <O m (t), A m (t), R(s(t), a(t), s'(t+1)), O' m (t+1), A' m (t+1)>

[0088] Here,

[0089] O m (t): Observation vector at the current time point

[0090] A m (t): Total action vector at the current time point

[0091] R(s(t), a(t), s'(t+1)): Compensation function

[0092] O' m (t+1): Observation vector at the next time point

[0093] A' m (t+1): Total action vector at the next time point

[0094] As interceptor drones collect environmental information in a battlefield environment, these transitions are stored in an experience replay buffer. During training, random samples are drawn from this experience replay buffer to break the temporal correlation of sequential data and enable the multiple reuse of previous experiences. When the buffer reaches its capacity, the oldest transition is replaced with a new one. For example, the buffer size can store 150,000 transitions, and training is performed by randomly sampling batch size B (e.g., 64).

[0095] The Target Network has the same structure as the entire training network, including all local Q networks and the mixing network, but is a separate neural network in which parameters are copied from the network currently being trained at specific time intervals K (e.g., 40 steps). If the parameters of the Target Network are denoted as θ', they are periodically synchronized as follows.

[0096] θ' ← θ (every K steps)

[0097] The target network is used to calculate the target value in Temporal Difference Learning to reduce oscillations in the learning process and improve stability. Specifically, the gradient of the loss function is calculated as shown in Equation 4.

[0098] [Mathematical Formula 4]

[0099]

[0100] Here, δθ(t) represents the time difference (TD) error and is defined as shown in the following mathematical equation 5.

[0101] [Mathematical Formula 5]

[0102]

[0103] Here,

[0104] s t (t), a(t) t , s' t (t+1): Target network state, action, next state

[0105] θ t : Neural network parameters of the target network

[0106] γ: Discount factor, adjusts the importance of future rewards

[0107] b: Minibatch index

[0108] B: Mini-batch size of transitions extracted from the replay buffer

[0109] R(·): Immediate reward function

[0110] The target network promotes stable convergence by reducing the variance of action value estimates. Additionally, it can mitigate abrupt oscillations in the learning process by fixing network parameters for a certain period and updating them periodically.

[0111] Through this configuration, the anti-drone system of the present invention can perform effective cooperative interception missions even in complex battlefield environments. In particular, through a mixing network that satisfies monotonicity constraints, it ensures that the performance improvement of individual drones directly leads to the performance improvement of the entire team, and enables stable and efficient learning through experience replay and target networks.

[0112] The architecture of Fig. 6 includes various anti-drone mission-specific elements. First, in terms of multi-target tracking, each drone network independently tracks enemy drones while achieving overall coordination through a mixing network. This enables an effective response even in situations where multiple enemy drones intrude simultaneously.

[0113] Furthermore, for energy-efficient cooperation, the system is designed to incorporate individual drones' remaining energy information into their state, enabling them to learn cooperation strategies that consider energy consumption. This allows drones to efficiently manage energy and complete missions in situations requiring long-duration operations.

[0114] In terms of real-time adaptability, it is designed to enable effective decision-making using only partial observation data, ensuring stable operation even in the event of communication delays or failures. This is a particularly important feature in electronic warfare environments or actual battlefield situations involving communication jamming.

[0115] Furthermore, this architecture has the advantage of being expandable with the same structure even when the number of drones (M) changes. The number of interceptor drones can be flexibly adjusted according to mission requirements, and expansion is possible without additional structural changes.

[0116] Through this configuration, the anti-drone control system of the present invention can perform effective cooperative interception missions even in complex battlefield environments.

[0117] FIG. 7 is a flowchart illustrating the learning process of a multi-agent reinforcement learning-based anti-drone control method according to one embodiment of the present invention.

[0118] Referring to FIG. 7, the anti-drone control method of the present invention is broadly divided into a learning phase and an execution phase.

[0119] First, each of the multiple interceptor drones observes status information including the location of enemy drones (50), the location of friendly drones, and remaining energy within a limited sensor range (S10). The sensor range of each drone may be limited to, for example, a radius of 50m, which reflects the constraints of the actual battlefield environment.

[0120] Subsequently, the central learning server collects experience data from multiple interceptor drones through communication relay nodes (S20). The communication relay nodes are designed to operate stably even in limited communication environments where communication delays and packet loss occur.

[0121] The central learning server trains the local Q network of each drone using an anti-drone specialized reward function that includes individual rewards based on distance from the enemy drone (50) and common rewards based on the team's overall neutralization performance (S30). At the same time, a mixing network that receives individual Q values ​​output by the local Q networks during the learning process and calculates a global Q value is trained together.

[0122] In particular, the mixing network is trained to satisfy the monotonicity constraint that the global Q-value does not decrease as individual Q-values ​​increase. This is because, as explained earlier, ∂h total / ∂h m It means satisfying the condition ≥ 0.

[0123] During the training process, the central training server stores sets of experience tuples from multiple interceptor drones in a pre-prepared experience replay buffer. Training data is constructed by randomly sampling the sets of experience tuples stored in the experience replay buffer by a batch size (e.g., 32 or 64). Then, the local Q network and mixing network are updated based on the training data and the pre-prepared target network.

[0124] The central learning server distributes the trained local Q network to each interceptor drone (S40). This is done via a communication relay node, and each drone receives the latest trained model and stores it in its own experience replay buffer.

[0125] In the execution phase, each interceptor drone performs a control action according to the distributed local Q network (S50). At this time, each drone makes a decision independently using only partial observation information within its sensor range.

[0126] Specifically, each interceptor drone independently determines and performs a 3-axis control action including pitch, yaw, and roll according to a local Q network. Pitch control controls the up-and-down tilt of the drone, yaw control controls left-and-right rotation, and roll control controls left-and-right tilt.

[0127] If each interceptor drone is equipped with a net launcher, the step of performing the control action may further include a step of determining the net launch timing. The net launch timing is determined by a local Q network by taking into account the distance to the enemy drone (50), relative speed, predicted trajectory, etc.

[0128] One of the important features of the present invention is that continuous learning is possible even during execution. New experience data collected by each drone while performing a mission is transmitted back to a central learning server and used to improve the model. Through this, the system can adapt to changing enemy drone tactics or new environmental conditions.

[0129] FIG. 8 is a graph showing the change in the normalized total reward value according to the training episode of a multi-agent reinforcement learning-based anti-drone control system according to one embodiment of the present invention.

[0130] Referring to FIG. 8, interceptor drones trained with the embodiment (Proposed) of the present invention achieve significantly higher reward values ​​compared to comparison algorithms (Comp-1, Comp-2, Comp-3). Specifically, after training 10,000 episodes, the embodiment of the present invention converges to a normalized reward value of approximately 0.8 to 1.0, whereas the comparison algorithms remain at a level of 0.2 to 0.6.

[0131] Of particular note is that the embodiment of the present invention exhibits performance similar to that of a full observation (FO-MDP) environment even in a partial observation (PO-MDP) environment. This demonstrates that the mixing network of the present invention enables effective cooperation even under limited information.

[0132] FIG. 9 is a graph showing the change in compensation values ​​per episode of an individual interceptor drone according to one embodiment of the present invention.

[0133] Referring to Figure 9, it can be seen that all six interceptor drones (1st Drone to 6th Drone) achieve equally high reward values. This means that the burden is not concentrated on any specific drone, and all drones are cooperating effectively. Initially, they start with low reward values, but as learning progresses, all drones converge to high reward values.

[0134] FIG. 10 is a graph showing the change in residual energy over time of an interceptor drone according to one embodiment of the present invention.

[0135] Referring to FIG. 10, interceptor drones trained according to an embodiment of the present invention consume energy efficiently during mission execution. All six drones exhibit similar energy consumption patterns and maintain approximately 20–40% of residual energy even after 60 hours of mission execution. This indicates that the drones have learned a cooperative strategy that takes energy consumption into account.

[0136] In particular, the 3rd Drone (green line) maintains slightly higher residual energy than the other drones, which suggests that a tactical division of roles has been established in which the drone acts as a standby and is deployed when necessary.

[0137] FIG. 11 is a box plot showing the distribution of the number of enemy drones captured by an interceptor drone trained with an embodiment of the present invention and a comparison algorithm.

[0138] Referring to FIG. 11, it can be seen that interceptor drones trained according to an embodiment of the present invention capture the largest number of enemy drones. Specifically,

[0139] - Example of the present invention (Proposed): Average of 8 captured (median approximately 8)

[0140] - Comp-1: Average 5 captures

[0141] - Comp-2: Average of 4 captures

[0142] - Comp-3: Average 1 capture

[0143] Embodiments of the present invention demonstrate up to approximately 800% performance improvement compared to conventional technology, and exhibit stable performance with low variance. This proves that the cooperative multi-agent reinforcement learning-based approach of the present invention is highly effective for anti-drone missions.

[0144] The multi-agent reinforcement learning-based anti-drone control system of the present invention can be utilized in various fields. In the military sector, it can be used for the protection of military facilities, strategic assets, and key infrastructure. In the civilian security sector, it can be used for the protection of critical facilities such as airports, power plants, and government buildings. In the event security sector, it can be used to ensure safety for large-scale sports events and international conferences. It can also be used for border surveillance to detect and block illegal drone intrusions.

[0145] In particular, the present invention employs a non-destructive capture method, allowing for safe operation even in urban areas, and has the advantage of collecting threat information through the analysis of the captured drone.

[0146] As described above, the present invention provides an anti-drone control system that operates effectively even under realistic constraints such as partial observation environments and limited communication conditions. Through the monotonicity constraint of the mixing network, it ensures that the performance improvement of individual drones directly leads to the performance improvement of the entire system, and its performance has been proven through experimental results.

[0147] The core technical features of the present invention are summarized as follows. The present invention adopts a Centralized Learning and Distributed Execution (CTDE) structure, thereby learning an optimal cooperative strategy by utilizing global information during learning, and during execution, each drone independently performs decision-making using only partial observation information. Furthermore, stable cooperative learning is realized by providing a mathematical guarantee through a mixing network with monotonic constraints that prevents an increase in the Q-value of an individual drone from decreasing the Q-value of the entire team. In terms of anti-drone specialized reward design, a reward function is configured to consider the distance from the enemy drone (50), remaining energy, and the team's overall capture performance in a balanced manner, thereby enabling efficient multi-goal achievement. Moreover, the present invention possesses real-time adaptability that operates stably even under practical constraints such as communication delay, packet loss, and partial observation, and provides scalability and flexibility capable of adapting to changes in the number of drones, various battlefield environments, and new tactical patterns.

[0148] This invention does not merely apply AI technology to drones, but rather meticulously designs and optimizes each component of reinforcement learning to meet the specific requirements of anti-drone missions. Through this, it realizes a level of cooperative intelligence that is difficult to achieve with conventional centralized control or pre-programmed tactics.

[0149] Moving forward, the present invention provides a foundation for continuous evolution in accordance with advancements in drone technology and the sophistication of threats. The core architecture of the present invention can flexibly respond even to the introduction of new sensor technologies, communication protocols, or capture mechanisms, which is expected to contribute to securing long-term technological competitiveness.

[0150] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive.

[0151] For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may also be implemented in a combined form. In addition, the figures specifically mentioned in this specification (e.g., sensor range 50 m, batch size 32 / 64, target network synchronization period 40 steps, etc.) represent preferred embodiments and may be appropriately adjusted in actual implementation according to system requirements and environmental conditions.

[0152] The scope of the present invention is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0154] 10: Multi-Agent Reinforcement Learning-Based Anti-Drone Control System 50: Enemy Drone 100: Interceptor Drone 200: Communication relay node 300: Central Learning Server s(t): global state o m (t): Partial observation of the m-th drone a m (t): Action of the m-th drone h m : Individual Q value of the m-th drone htotal: global Q value θ: Current network parameter θ': Target network parameter W1, W2: Hypernetwork weights V Ωm (t): yaw angle ψ Ωm (t): Pitch angle φ Ωm (t): Roll angle S10: State information observation step S20: Experience data collection stage S30: Learning stage S40: Model deployment phase S50: Control action execution step

Claims

Claim 1 A multi-agent reinforcement learning-based anti-drone control system comprising: a plurality of interceptor drones, each observing state information including enemy drone location, friendly drone location, and remaining energy within a limited sensor range, and performing control actions according to a local Q network distributed from a central learning server; and a communication relay node that relays communication between the plurality of interceptor drones and the central learning server. A multi-agent reinforcement learning-based anti-drone control system comprising: a central learning server that trains a local Q network of each drone using experience data collected from the plurality of interceptor drones and a preset anti-drone specialized reward function, and trains a mixing network that receives individual Q values ​​output by the local Q networks during the training process and calculates a global Q value, and distributes the trained local Q networks to each drone; wherein the anti-drone specialized reward function includes an individual reward based on distance to enemy drones and a common reward based on the team's total neutralization performance, wherein the individual reward is designed to have a higher value as the distance to enemy drones is closer and as the remaining energy is higher, and the common reward is configured to be calculated as a value normalized by the number of drones for the total team's neutralization performance. Claim 2 A multi-agent reinforcement learning-based anti-drone control system according to claim 1, wherein the mixing network is configured to satisfy a monotonic constraint such that the global Q value does not decrease when the individual Q value increases, and the monotonic constraint is implemented by applying an activation function to the weight parameters of the mixing network so that all weights become non-negative. Claim 3 A multi-agent reinforcement learning-based anti-drone control system according to claim 1, wherein each interceptor drone independently determines a 3-axis control action including pitch, yaw, and roll according to the local Q network using only partial observation information within its sensor range. Claim 4 delete Claim 5 A multi-agent reinforcement learning-based anti-drone control system according to claim 1, wherein the central learning server stores a set of experience tuples from the plurality of interceptor drones in a pre-prepared experience replay buffer, constructs training data by randomly sampling the set of experience tuples stored in the experience replay buffer by a batch size, and updates the mixing network based on the training data and a pre-prepared target network. Claim 6 A multi-agent reinforcement learning-based anti-drone control system according to claim 1, wherein each interceptor drone further comprises a net launching device for non-destructively capturing enemy drones, and the control action further comprises determining the net launching timing. Claim 7 A multi-agent reinforcement learning-based anti-drone control method comprising: a step in which each of a plurality of interceptor drones observes state information including the location of an enemy drone, the location of a friendly drone, and remaining energy within a limited sensor range; a step in which a central learning server collects experience data from the plurality of interceptor drones through a communication relay node; a step in which the central learning server trains a local Q network of each drone using the collected experience data and a preset anti-drone specialized reward function, and simultaneously trains a mixing network that receives individual Q values ​​output by the local Q networks during the training process and calculates a global Q value; and a step in which the central learning server distributes the trained local Q networks to each interceptor drone. A multi-agent reinforcement learning-based anti-drone control method comprising the step of each interceptor drone performing a control action according to a local Q network distributed thereto, wherein the training step includes pre-setting an anti-drone specialized reward function to include an individual reward based on distance from enemy drones and a common reward based on team-wide neutralization performance, wherein the anti-drone specialized reward function is designed such that the individual reward has a higher value as the distance from enemy drones is closer and has a higher value as the remaining energy is greater, and the common reward is configured to be calculated as a value normalized by the number of drones for the entire team's neutralization performance. Claim 8 A multi-agent reinforcement learning-based anti-drone control method according to claim 7, wherein the training step comprises configuring the mixing network to satisfy a monotonic constraint such that the global Q value does not decrease when the individual Q value increases, and the monotonic constraint is implemented by applying an activation function to the weight parameters of the mixing network so that all weights become non-negative. Claim 9 A multi-agent reinforcement learning-based anti-drone control method according to claim 7, wherein the observing step comprises each interceptor drone using only partial observation information within its sensor range, and the step of performing the control action comprises each interceptor drone independently determining and performing a 3-axis control action including pitch, yaw, and roll according to the local Q network. Claim 10 delete

Images