Reinforcement learning environment simulation platform and method in unmanned aerial vehicle defense

Abstract

The invention provides a reinforcement learning environment simulation platform and method in unmanned aerial vehicle defense, and relates to the technical field of unmanned aerial vehicle interception mechanisms and reinforcement learning. The simulation platform comprises an unmanned aerial vehicle model library, an initialization module, a reset module, an unmanned aerial vehicle defense stepping module and an unmanned aerial vehicle defense man-machine interaction module. Aiming at the problem that a current anti-unmanned aerial vehicle task allocation method based on reinforcement learning lacks a corresponding reinforcement learning simulation environment, a universal reinforcement learning environment model and a simulation platform with high trueness are constructed through modeling and simulation of an unmanned aerial vehicle, various interception equipment, detection equipment, an interception confrontation process and a detection tracking process; the problems that the intelligent agent obtained through existing reinforcement learning training is insufficient in reliability and is only suitable for a specific scene are solved. And the decision-making capability of the intelligent agent in a complex environment is improved.

Description

Technical Field

[0001] This document relates to the field of drone interception mechanisms and reinforcement learning technology, specifically to a reinforcement learning environment simulation platform and method for drone defense. Background Technology

[0002] With the development of drone technology, drones have gradually become a low-cost, highly flexible, highly destructive, and difficult-to-detect means of sabotage or attack. How to efficiently utilize existing interception methods for drone defense under environmental constraints is an important research topic. One important method is training agents using reinforcement learning for anti-drone task allocation. However, reinforcement learning requires interaction between the agent and the environment. The agent influences the environment by performing actions, and the environment responds by updating its state and outputting immediate rewards. Reinforcement learning aims to maximize the cumulative reward within a task cycle. While mature reinforcement learning environment models have been developed in fields such as robot control to support agent reinforcement training, no fully functional environment model is available in the field of drone defense. Current reinforcement learning-based anti-drone task allocation methods mainly simulate the environmental state transitions after a single drone interception by manually defining simple state transition probabilities, as seen in patents CN202011482387.0 and CN202311702266.6. These methods cannot assess the complete process from target discovery, target tracking, interception, interception encounter, to the end of the confrontation based on the continuous movement of elements such as drones, interception equipment, and detection equipment. However, they are highly random and differ greatly from the actual situation. The reliability of the trained agents is insufficient, and they are only applicable to specific scenarios with insufficient robustness. Summary of the Invention

[0003] This specification provides a reinforcement learning environment simulation platform and method for drone defense, which addresses the problems of insufficient reliability of agents trained by existing reinforcement learning methods and their applicability only to specific scenarios.

[0004] Firstly, this specification provides a reinforcement learning environment simulation platform for drone defense, including: a drone model library, an initialization module, a reset module, a drone defense stepping module, and a drone defense human-computer interaction module;

[0005] The drone model library contains abstract models of drones, detection equipment, and six types of interception equipment. Each model includes ID, type, status, location attributes, and the calculation method adapted to the model.

[0006] The initialization module is used to perform system initialization, target initialization, detection equipment initialization, and interception equipment initialization, and to generate a UAV instance dictionary, a UAV instance list, a detection equipment instance list, and an interception equipment instance list.

[0007] The reset module is used to reset system parameters, target instances, detection equipment instances, and interception equipment instances;

[0008] The drone defense stepping module includes two sub-modules: feature acquisition and state transition. The feature acquisition sub-module is used to extract feature tensors containing target information and interception equipment information by calculating the state of the target drone, interception equipment, and detection equipment instance in the environment at the current moment. The state transition sub-module is used to execute action responses, update the environmental state, and calculate the output action reward and round end flag based on the input interception action.

[0009] The human-computer interaction module includes a scenario editing submodule and an adversarial simulation submodule. The scenario editing submodule is used to set system parameters, target parameters, interception and detection equipment parameters, preview equipment deployment locations, and the power coverage range of detection and interception equipment. The adversarial simulation submodule is used to load scenarios, dynamically simulate the adversarial process, and dynamically adjust the simulation speed.

[0010] Secondly, this specification provides a reinforcement learning environment simulation method for drone defense, including:

[0011] T1. Run the human-computer interaction module and edit the combat scenario file through the scenario editing submodule;

[0012] T2, Initialization Module, performs system initialization, target initialization, detection equipment initialization, and detection equipment initialization;

[0013] T3, Feature Acquisition Submodule, calculates the status of target UAVs, interception equipment, and detection equipment instances in the environment at the current moment, and extracts feature tensors containing target information and interception equipment information;

[0014] T4, the state transition submodule, executes the action response based on the input interception action, updates the environment state, and calculates the output action reward and round end flag;

[0015] T5, Human-Computer Interaction Module, previews equipment deployment locations and the power coverage of detection and interception equipment; dynamically simulates the confrontation process and dynamically adjusts the simulation speed.

[0016] The beneficial effects of this invention are as follows:

[0017] This specification provides a reinforcement learning environment simulation platform and method for drone defense. The simulation platform includes a drone model library, an initialization module, a reset module, a drone defense stepping module, and a drone defense human-machine interaction module. Addressing the lack of corresponding reinforcement learning environment models in current reinforcement learning-based anti-drone task allocation methods, this paper constructs a general and highly realistic reinforcement learning environment model and simulation platform by modeling and simulating drones, various interception equipment, detection equipment, interception and countermeasure processes, and detection and tracking processes. This solves the problems of insufficient reliability of agents trained by existing reinforcement learning methods and their applicability only to specific scenarios; it also improves the decision-making ability of agents in complex environments. The simulation platform supports six types of interception equipment: missile-based, artillery-based, laser-based, wave-based, jamming, and air superiority drones, basically covering the main current drone countermeasures. The drone models have the ability to fly autonomously according to planned routes and respond to interference, exhibiting high reliability. The detection equipment has a follow-up function, simulating the dynamic tracking process of targets. The entire environment model supports multiple defense scenarios, has strong versatility, and the countermeasure process follows the physical laws of missile-target flight, exhibiting high realism. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 This is a schematic diagram of a reinforcement learning environment simulation platform for drone defense provided in the embodiments of this specification;

[0020] Figure 2 This is a schematic diagram of a scenario editing submodule in a human-computer interaction module for drone defense provided in the embodiments of this specification;

[0021] Figure 3 This is a schematic diagram of an adversarial simulation submodule in a human-computer interaction module for drone defense provided in the embodiments of this specification;

[0022] Figure 4 This is a schematic diagram of a reinforcement learning environment simulation method for drone defense provided in the embodiments of this specification. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this document.

[0024] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings. Specific Implementation Example 1:

[0026] This embodiment provides a reinforcement learning environment simulation platform for drone defense. (See also...) Figures 1-3 It includes: drone model library, initialization module, reset module, drone defense step module, and drone defense human-computer interaction module;

[0027] like Figure 1 As shown, the UAV model library contains abstract models of UAVs, detection equipment, and six types of interception equipment. Each model includes ID, type, status, location attributes, and the calculation method adapted to the model.

[0028] Six types of interception equipment include: missiles, anti-aircraft guns, lasers, high-power microwaves, electromagnetic interference, and dogfighting drones.

[0029] The drone model has methods for initialization, state update, position update, velocity update, azimuth calculation, and position offset calculation.

[0030] The detection and interception equipment models have initialization and pointing angle update calculation methods.

[0031] Furthermore, the initialization module is used to perform system initialization, target initialization, detection equipment initialization, and interception equipment initialization, and to generate a UAV instance dictionary, a UAV instance list, a detection equipment instance list, and an interception equipment instance list.

[0032] Specifically, the initialization module includes four sub-modules: system initialization, target initialization, detection equipment initialization, and interception equipment initialization, which are executed sequentially. Among them:

[0033] The system initialization module reads the input scenario file information to initialize the relevant parameters of the environment model and perform coordinate transformation;

[0034] The target initialization module initializes the UAV model by reading the input scenario file information and generates a UAV instance dictionary and a UAV instance list.

[0035] The detection equipment initialization module initializes the detection equipment model and generates a list of detection equipment instances by reading the input scenario file information.

[0036] The intercept equipment initialization module initializes the intercept equipment model and generates a list of intercept equipment instances by reading the input scenario file information.

[0037] Furthermore, the reset module is used to reset system parameters, target instances, detection equipment instances, and interception equipment instances;

[0038] Specifically, the reset module comprises four sub-modules: system reset, target reset, detection equipment reset, and interception equipment reset. These are executed sequentially. By reading scenario file information, the module resets relevant system parameters, target instances, detection equipment instances, and interception equipment instances. Unlike the initialization module, the reset module automatically and cyclically reads multiple input scenario files, which can increase the diversity of samples during policy network training.

[0039] Furthermore, the drone defense stepping module includes two sub-modules: feature acquisition and state transition. The feature acquisition sub-module is used to extract feature tensors containing target information and interception equipment information by calculating the state of the target drone, interception equipment, and detection equipment instance in the environment at the current moment. The state transition sub-module is used to execute action responses, update the environmental state, and calculate the output action reward and round end flag based on the input interception action.

[0040] The feature acquisition submodule is specifically used for:

[0041] S11. Traverse the target instance list and update the list of trackable detection equipment IDs for each target;

[0042] S12. Traverse the list of interceptor equipment instances and update the list of guided detection equipment IDs corresponding to each interceptor equipment.

[0043] S13. Perform state sampling on all targets to generate target features T; the sampleable states of the targets include: target type, northeast-sky coordinates, northeast-sky velocity, threat level, and interception status.

[0044] The target feature T is represented in matrix form:

[0045] Among them, element t i1 The value t represents the type of the i-th target, with 1 for micro drones, 2 for light drones, and 3 for loitering munitions; i2 ~t i4 t represents the northeast-central coordinates of the i-th target; i5 ~t i7 Represents the velocity component of the i-th target in the northeast-sky direction; element t i8 Represents the threat level of the i-th target; element t i9 This indicates the interception status of the i-th target, with a value of 1 for intercepted and 0 for uninterrupted.

[0046] S14. Calculate the firing condition matrix F of the interceptor equipment against the target, where the firing condition matrix F is a three-dimensional matrix, with element indices starting from 1, and matrix elements f 1ij =1, indicating that the i-th interceptor is ready to fire at the j-th target; otherwise, it is 0; matrix element f 2ij Represents the predicted time of encounter between projectile and target; matrix element f 3ij This indicates the predicted distance to the target;

[0047] S15. Perform state sampling on all interception equipment to generate interception equipment feature A; wherein the sampleable states of the interception equipment include the predicted encounter time between the missile and the target, the predicted encounter distance between the missile and the target, the kill probability, the interception cost, the number of remaining resources, and the number of remaining channels.

[0048] Interception equipment feature matrix A is represented as follows:

[0049] Where: a m1 This represents the interception cost of the m-th interceptor; a m2 This represents the remaining resources of the m-th interceptor equipment;

[0050] a m3 This represents the number of remaining channels for the m-th interceptor; a m,3n+1 This represents the predicted encounter time between the m-th interceptor and the n-th target; a m,3n+2 This represents the predicted encounter distance between the m-th interceptor and the n-th target.

[0051] a m,3n+3 Let f represent the kill probability of the m-th interceptor against the n-th target, where f is the probability that the interceptor is not in a firing position against the target. 1mn When a = 0, m,3n+3 =0.

[0052] S16. Normalize the target characteristics and interception equipment characteristics;

[0053] S17. Convert the format of target feature T and interception equipment feature A.

[0054] The state transition submodule is specifically used for:

[0055] S21. Initialize the reward matrix R as a 6-row, n-column two-dimensional matrix, where n represents the number of incoming targets;

[0056] S22. Traverse the target instance list, respond to the input interception action, fill the reward matrix R, and record the reward information;

[0057] S23. Calculate the average reward r: r = sum(rpn·R) / n; where sum(·) represents summation; rpn is a list of positive and negative reward weights for the system parameters;

[0058] S24. Update simulation time, increasing the simulation time by one simulation time step;

[0059] S25. Traverse the target instance list, the detection equipment instance list, and the interception equipment list to update the environment status;

[0060] S26, Round end judgment;

[0061] S27, output reward r and turn end flag done.

[0062] The methods used for updating the environmental status include:

[0063] S251. Traverse the list of target instances and call the relevant methods of the target class to update the target's state, position, and speed;

[0064] S252. Traverse the list of interceptor equipment instances and update the status, azimuth, and elevation angle of the interceptor equipment; the status attributes of the equipment are switched using a random number θ.

[0065] S253. Traverse the list of detection equipment instances and update the status, pointing angle, assigned target list, and remaining channel count of the detection equipment; the status attributes of the equipment are switched using a random number θ.

[0066] S254. Traverse the list of target instances. For the target being intercepted, determine whether the interception process has ended and update the relevant attributes.

[0067] S255. Iterate through the interception equipment list again and update the list of assigned targets, status, and number of remaining channels for each interception equipment instance.

[0068] S256. Iterate through the detection equipment list again and update the allocated target list, status, and remaining channel count of the detection equipment instances.

[0069] Furthermore, the human-computer interaction module includes a scenario editing submodule and an adversarial simulation submodule. The scenario editing submodule is used to set system parameters, target parameters, interception and detection equipment parameters, preview equipment deployment locations, and the power coverage range of detection and interception equipment. The adversarial simulation submodule is used to load scenarios, dynamically simulate the adversarial process, and dynamically adjust the simulation speed.

[0070] In summary, this embodiment addresses the lack of corresponding reinforcement learning environment models in current reinforcement learning-based anti-drone task allocation methods. By modeling and simulating drones, various interception equipment, detection equipment, the interception and countermeasure process, and the detection and tracking process, it constructs a general and highly realistic reinforcement learning environment model and simulation platform. This solves the problems of insufficient reliability of agents trained by existing reinforcement learning methods and their applicability only to specific scenarios; it also improves the decision-making ability of agents in complex environments. The simulation platform supports six types of interception equipment: missiles, artillery, light, wave, jamming, and air superiority drones, basically covering the main current drone countermeasures. The drone model has the ability to fly autonomously according to planned routes and respond to interference, ensuring high reliability. The detection equipment has a homing function, simulating the dynamic tracking process of targets. The entire environment model supports multiple defense scenarios, has strong versatility, and the countermeasure process follows the physical laws of missile-target flight, resulting in high realism. Specific Implementation Example 2:

[0072] This embodiment provides a reinforcement learning environment simulation method for drone defense. (See also...) Figure 4 Specifically, it includes the following steps:

[0073] T1. Run the human-computer interaction module and edit the combat scenario file through the scenario editing submodule;

[0074] T2, Initialization Module, performs system initialization, target initialization, detection equipment initialization, and interception equipment initialization;

[0075] T3, Feature Acquisition Submodule, calculates the status of target UAVs, interception equipment, and detection equipment instances in the environment at the current moment, and extracts feature tensors containing target information and interception equipment information;

[0076] T4, the state transition submodule, executes the action response based on the input interception action, updates the environment state, and calculates the output action reward and round end flag;

[0077] T5, Human-Computer Interaction Module, previews equipment deployment locations and the power coverage of detection and interception equipment; dynamically simulates the confrontation process and dynamically adjusts the simulation speed. Specific Implementation Example 3:

[0079] This embodiment provides a reinforcement learning environment modeling and simulation platform for drone defense, such as... Figure 1 As shown, it mainly includes a model library, an initialization module, a reset module, a stepping module, and a human-computer interaction module.

[0080] (1) Model Library

[0081] The model library contains abstract models of drones, detection equipment, and six types of interception equipment. Each model includes attributes and methods such as ID, type, state, and location.

[0082] The drone model has methods for initialization, state update, position update, velocity update, azimuth calculation, and position offset calculation;

[0083] The detection and interception equipment models have methods such as initialization and pointing angle update.

[0084] (2) Initialization module

[0085] The initialization module comprises four sub-modules: system initialization, target initialization, detection equipment initialization, and detection equipment initialization, which are performed sequentially. Among them:

[0086] The system initialization module reads the input scenario file information to initialize the relevant parameters of the environment model and perform coordinate transformation;

[0087] The target initialization module initializes the UAV model by reading the input scenario file information and generates a UAV instance dictionary and a UAV instance list.

[0088] The detection equipment initialization module initializes the detection equipment model and generates a list of detection equipment instances by reading the input scenario file information.

[0089] The intercept equipment initialization module initializes the intercept equipment model and generates a list of intercept equipment instances by reading the input scenario file information.

[0090] (3) Reset module

[0091] The reset module comprises four sub-modules: system reset, target reset, detection equipment reset, and interception equipment reset. These are executed sequentially. By reading scenario file information, the module resets relevant parameters of the environment model, target instances, detection equipment instances, and interception equipment instances. Unlike the initialization module, the reset module automatically and cyclically reads multiple input scenario files, increasing sample diversity during policy network training.

[0092] (4) Stepping module

[0093] The stepping module comprises two sub-modules: feature acquisition and state transition. The feature acquisition sub-module calculates the state of each element in the environment at the current moment and outputs a feature tensor containing target information and interception equipment information. The state transition sub-module executes the action response based on the input interception action, then updates the environmental state, calculates the output action reward, and provides a signal indicating whether the round has ended.

[0094] The specific working process of the feature acquisition submodule includes:

[0095] Step 1: Iterate through the list of target instances and update the available_detector attribute of each target (a list of detection equipment IDs that can be used to track the target);

[0096] Step 2: Iterate through the list of interceptor equipment instances and update the available_detector attribute of each interceptor equipment (which can be a list of detector equipment IDs that the interceptor equipment can target).

[0097] Step 3: Sample the state of all targets and generate target features T. The sampled target states include, but are not limited to: target type (type attribute), northeast-sky coordinates (enu_position attribute), northeast-sky velocity (enu_velocity attribute), threat level (threaten attribute), and intercepted state (attacked attribute), etc.

[0098] The target feature T is represented in matrix form:

[0099]

[0100] Among them, element t i1 The value t represents the type of the i-th target, with 1 for micro drones, 2 for light drones, and 3 for loitering munitions; i2 ~t i4 t represents the northeast-central coordinates of the i-th target; i5 ~t i7 Represents the velocity component of the i-th target in the northeast-sky direction; element t i8 Represents the threat level of the i-th target; element t i9 This indicates the interception status of the i-th target, with a value of 1 for intercepted and 0 for uninterrupted.

[0101] Step 4: Calculate the firing condition matrix F of the interceptor equipment against the target.

[0102] The shooting condition matrix F is a three-dimensional matrix, with element indices starting from 1. The matrix elements f... 1ij =1, indicating that the i-th interceptor is ready to fire at the j-th target; otherwise, it is 0; matrix element f 2ij Represents the predicted time of encounter between projectile and target; matrix element f 3ij This indicates the predicted distance to the target.

[0103] Step 5: Perform status sampling on all interception equipment to generate interception equipment feature A.

[0104] Sampleable states include, but are not limited to, the predicted encounter time with the target, the predicted encounter distance, the kill probability (kill_p attribute of the interceptor equipment), the interception cost (cost attribute of the interceptor equipment), the remaining resources (n_resource attribute of the interceptor equipment), and the remaining channels (n_channel attribute of the interceptor equipment). The interceptor equipment feature matrix A represents:

[0105]

[0106] in:

[0107] a m1 This represents the interception cost of the m-th interceptor device;

[0108] a m2 This represents the remaining resources of the m-th interceptor equipment;

[0109] a m3 This represents the number of remaining channels for the m-th interceptor.

[0110] a m，3n+1 This represents the predicted encounter time between the m-th interceptor and the n-th target.

[0111] a m，3n+2 This represents the predicted encounter distance between the m-th interceptor and the n-th target.

[0112] a m,3n+3 Let f represent the kill probability of the m-th interceptor against the n-th target, where f is the probability that the interceptor is not in a firing position against the target. 1mn When a = 0, m,3n+3 =0.

[0113] Step 6: If the system parameter normalization is True, then normalize the target features and interception equipment features.

[0114] Step 7: Convert the target feature T and the interception equipment feature A according to the design requirements of the intelligent agent.

[0115] The specific working process of the state transition submodule includes:

[0116] Step 1: Initialize the reward matrix R as a 6-row, n-column two-dimensional matrix, where n represents the number of incoming targets. The first row stores the target threat level (positive reward), the second row stores the reciprocal of the target encounter time (positive reward), the third row stores the target encounter distance (positive reward), the fourth row stores the interception cost (negative reward), the fifth row stores the target threat level (negative reward), and the sixth row stores collateral damage.

[0117] Step 2: Traverse the target instance list, respond to the input interception action, fill the reward matrix R, and record the reward information.

[0118] The interception action is represented by matrix K, where the matrix element k i1 k represents the id of the i-th target; i2 This represents the interception equipment ID for intercepting the i-th target; k i3 This represents the ID of the detection equipment tracking the i-th target.

[0119] Step 3: Calculate the average reward r. The calculation formula is as follows:

[0120] r = sum(rpn·R) / n

[0121] In the formula: sum(·) represents summation calculation; rpn is the list of positive and negative reward weights for system parameters; n is the number of incoming targets.

[0122] Step 4: Update the simulation time by increasing the simulation time by one simulation time step.

[0123] Step 5: Update the environment status. Perform the following steps in sequence.

[0124] Step 5.1: Iterate through the list of target instances and call the relevant methods of the target class to update the target's state, position, speed, and other attributes;

[0125] Step 5.2: Traverse the list of interceptor equipment instances and update the status, azimuth, elevation, and other attributes of the interceptor equipment. The status attributes of the equipment are switched using a random number θ.

[0126] Step 5.3: Traverse the list of detection equipment instances and update the status, pointing angle, assigned target list, and remaining channel count of the detection equipment. The status attributes of the detection equipment are toggled using a random number θ.

[0127] Specifically, the status attribute of the detection equipment is changed from "working" / "standby" to "fault" by random number. When the status changes to "fault", the assigned target list is cleared, the pointing angle remains unchanged, and the number of remaining channels is directly set to zero.

[0128] Step 5.4: Traverse the list of target instances. For targets that are being intercepted, determine whether the interception process has ended and update the relevant attributes.

[0129] Step 5.5: Iterate through the intercept equipment list again and update the assigned target list, status, remaining channel count, and other attributes of the intercept equipment instance.

[0130] Step 5.6: Iterate through the detection equipment list again and update the assigned target list, status, remaining channel count, and other attributes of the detection equipment instances. The update method is the same as in Step 5.5.

[0131] Step 6: Round end judgment.

[0132] Step 7: Output the reward r and the round end flag done.

[0133] (5) Human-computer interaction module

[0134] The human-computer interaction module includes two functions: scenario editing and adversarial simulation. Running the module generates a user interface with two tabs: "Scenario Editing" and "Adversarial Simulation." Figure 2 As shown, the "Scenario Editing" tab can be used to set scenario information such as system parameters, target parameters, and interception and detection equipment parameters. The edited scenario is saved as a JSON file; it allows previewing equipment deployment locations, as well as the coverage area of ​​detection and interception equipment; for example... Figure 3 As shown, the "Adversarial Simulation" tab can be used to load scenarios, dynamically simulate the adversarial process, and has the function of dynamically adjusting the simulation speed. It has the functions of scenario loading, system initialization, dynamic simulation of the adversarial process, and process control. The map area supports interactive functions such as zooming and panning. The left side of the map is a display area for equipment and target instances. Selecting an instance and right-clicking allows you to select to display the instance's detailed attribute information, as shown in the table below the map.

[0135] Furthermore, the method of using the environment modeling and simulation platform in reinforcement learning in this embodiment is as follows:

[0136] Step 1: Run the human-computer interaction module, and edit the combat scenario file in the "Scenario Editing" tab;

[0137] Step 2: Import the CustomEnv class of the environment model from the corresponding Python module;

[0138] Step 3: Initialize an instance of CustomEnv. The program will automatically call the constructor to complete the initialization.

[0139] Step 4: Call the sample_state method of CustomEnv to extract state features;

[0140] Step 5: Call the step method of CustomEnv to perform environment state transition;

[0141] Step 6: When the interception process ends, i.e., when the round end flag done = True, call the reset method of CustomEnv to reset the state;

[0142] Step 7: Repeat steps 4 and 5 to enable the interception strategy to automatically generate an agent that continuously interacts with the environment model, and train the agent using reinforcement learning algorithms.

[0143] Step 8: Automatic Generation of Interception Strategy. After the intelligent agent training is completed, it can be embedded into the human-computer interaction module. Simulation can be performed through the "Simulation Adversarial" tab to test the interception effect of the intelligent agent.

[0144] In summary, this embodiment addresses the lack of corresponding reinforcement learning environment models in current reinforcement learning-based anti-drone task allocation methods. By modeling and simulating drones, various interception equipment, detection equipment, the interception and countermeasure process, and the detection and tracking process, it constructs a general and highly realistic reinforcement learning environment model and simulation platform. This solves the problems of insufficient reliability of agents trained by existing reinforcement learning methods and their applicability only to specific scenarios; it also improves the decision-making ability of agents in complex environments. The simulation platform supports six types of interception equipment: missiles, artillery, light, wave, jamming, and air superiority drones, basically covering the main current drone countermeasures. The drone model has the ability to fly autonomously according to planned routes and respond to interference, ensuring high reliability. The detection equipment has a homing function, simulating the dynamic tracking process of targets. The entire environment model supports multiple defense scenarios, has strong versatility, and the countermeasure process follows the physical laws of missile-target flight, resulting in high realism.

[0145] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A reinforcement learning environment simulation platform for drone defense, characterized in that, include: Drone model library, initialization module, reset module, drone defense stepping module, drone defense human-computer interaction module; The drone model library contains abstract models of drones, detection equipment, and six types of interception equipment. Each model includes ID, type, status, location attributes, and the corresponding calculation method. The initialization module is used to perform system initialization, target initialization, detection equipment initialization, and interception equipment initialization, and to generate a UAV instance dictionary, a UAV instance list, a detection equipment instance list, and an interception equipment instance list. The reset module is used to reset system parameters, target instances, detection equipment instances, and interception equipment instances; The drone defense step module includes two sub-modules: feature acquisition and state transition. The feature acquisition sub-module is used to extract feature tensors containing target information and interception equipment information by calculating the state of the target drone, interception equipment, and detection equipment instance in the environment at the current moment. The state transition submodule is used to execute action responses, update the environment state, and calculate the output action reward and round end flag based on the input interception action. The human-computer interaction module includes a scenario editing submodule and an adversarial simulation submodule. The scenario editing submodule is used to set system parameters, target parameters, interception and detection equipment parameters, preview equipment deployment locations, and the power coverage range of detection and interception equipment. The adversarial simulation submodule is used to load scenarios, dynamically simulate the adversarial process, and dynamically adjust the simulation speed.

2. The simulation platform according to claim 1, characterized in that, The feature acquisition submodule is specifically used for: S11. Traverse the target instance list and update the list of trackable detection equipment IDs for each target; S12. Traverse the list of interceptor equipment instances and update the list of guided detection equipment IDs corresponding to each interceptor equipment. S13. Perform state sampling on all targets to generate target features T; The sampleable states of the target include: target type, northeast-sky coordinates, northeast-sky velocity, threat level, and interception status. S14. Calculate the firing condition matrix F of the interceptor equipment against the target, where the firing condition matrix F is a three-dimensional matrix, with element indices starting from 1, and matrix elements f 1ij =1, indicating that the i-th interceptor is ready to fire at the j-th target; otherwise, it is 0; matrix element f 2ij Represents the predicted time of encounter between projectile and target; matrix element f 3ij This indicates the predicted distance to the target; S15. Perform state sampling on all interception equipment to generate interception equipment feature A; wherein the sampleable states of the interception equipment include the predicted encounter time between the missile and the target, the predicted encounter distance between the missile and the target, the kill probability, the interception cost, the number of remaining resources, and the number of remaining channels. S16. Normalize the target characteristics and interception equipment characteristics; S17. Convert the format of target feature T and interception equipment feature A.

3. The simulation platform according to claim 2, characterized in that, The target feature T is represented in matrix form: Among them, element t i1 The value t represents the type of the i-th target, with 1 for micro drones, 2 for light drones, and 3 for loitering munitions; i2 ~t i4 t represents the northeast-central coordinates of the i-th target; i5 ~t i7 Represents the velocity component of the i-th target in the northeast-sky direction; element t i8 Represents the threat level of the i-th target; element t i9 This indicates the interception status of the i-th target, with a value of 1 for intercepted and 0 for uninterrupted.

4. The simulation platform according to claim 3, characterized in that, Interception equipment feature matrix A is represented as follows: Where: a m1 This represents the interception cost of the m-th interceptor; a m2 a represents the remaining resources of the m-th interceptor; m3 This represents the number of remaining channels for the m-th interceptor; a m，3n+1 This represents the predicted encounter time between the m-th interceptor and the n-th target; a m，3n+2 This represents the predicted encounter distance between the m-th interceptor and the n-th target; a m,3n+3 Let f represent the kill probability of the m-th interceptor against the n-th target, where f is the probability that the interceptor is not in a firing position against the target. 1mn When a = 0, m,3m+3 =0.

5. The simulation platform according to claim 4, characterized in that, The state transition submodule is specifically used for: S21. Initialize the reward matrix R as a 6-row, n-column two-dimensional matrix, where n represents the number of incoming targets; S22. Traverse the target instance list, respond to the input interception action, fill the reward matrix R, and record the reward information; S23. Calculate the average reward r: r = sum(rpn·R) / n; where sum(·) represents summation; rpn is a list of positive and negative reward weights for the system parameters; S24. Update simulation time, increasing the simulation time by one simulation time step; S25. Traverse the target instance list, the detection equipment instance list, and the interception equipment list to update the environment status; S26, Round end judgment; S27, output reward r and turn end flag done.

6. The simulation platform according to claim 5, characterized in that, The methods used to update the environment status include: S251. Traverse the list of target instances and call the relevant methods of the target class to update the target's state, position, and speed; S252. Traverse the list of interceptor equipment instances and update the status, azimuth, and elevation angle of the interceptor equipment; the status attributes of the interceptor equipment are switched using a random number θ. S253. Traverse the list of detection equipment instances and update the status, pointing angle, assigned target list, and remaining channel count of the detection equipment; the status attributes of the detection equipment are switched using a random number θ. S254. Traverse the list of target instances. For the target being intercepted, determine whether the interception process has ended and update the relevant attributes. S255. Iterate through the interception equipment list again and update the list of assigned targets, status, and number of remaining channels for each interception equipment instance. S256. Iterate through the detection equipment list again and update the allocated target list, status, and remaining channel count of each detection equipment instance.

7. The simulation platform according to claim 6, characterized in that, Six types of interception equipment include: missiles, anti-aircraft guns, lasers, high-power microwaves, electromagnetic interference, and dogfighting drones.

8. The simulation platform according to claim 7, characterized in that, The drone model has methods for initialization, state update, position update, velocity update, azimuth calculation, and position offset calculation. The detection and interception equipment models have initialization and pointing angle update calculation methods.

9. The simulation platform according to claim 8, characterized in that, The initialization module includes four sub-modules: system initialization sub-module, target initialization sub-module, detection equipment initialization sub-module, and detection equipment initialization sub-module. The system initialization submodule is used to initialize the relevant parameters of the environment model and perform coordinate transformation; The target initialization submodule is used to initialize the drone model and generate a drone instance dictionary and a drone instance list; The detection equipment initialization submodule is used to initialize the detection equipment model and generate a list of detection equipment instances. The intercept equipment initialization submodule is used to initialize the intercept equipment model and generate a list of intercept equipment instances.

10. A reinforcement learning environment simulation method for unmanned aerial vehicle (UAV) defense, applied to the simulation platform described in any one of claims 1 to 9, characterized in that, include: T1. Run the human-computer interaction module and edit the combat scenario file through the scenario editing submodule; T2, Initialization Module, performs system initialization, target initialization, detection equipment initialization, and detection equipment initialization; T3, Feature Acquisition Submodule, calculates the status of target UAVs, interception equipment, and detection equipment instances in the environment at the current moment, and extracts feature tensors containing target information and interception equipment information; T4, the state transition submodule, executes the action response based on the input interception action, updates the environment state, and calculates the output action reward and round end flag; T5, Human-Computer Interaction Module, previews equipment deployment locations and the power coverage of detection and interception equipment; dynamically simulates the confrontation process and dynamically adjusts the simulation speed.

Images