Unmanned combat equipment intelligent decision-making system based on reinforcement learning

Abstract

The invention discloses an unmanned combat equipment intelligent decision-making system based on reinforcement learning. The system comprises a task understanding module which analyzes a combat intention according to a task instruction, analyzes a battlefield situation according to battlefield information, forms a sub-first-level tactical task oriented to each agent, and transmits the sub-first-level tactical task to a scheme construction module; the scheme construction module is used for generating a plurality of feasible combat schemes according to each sub-first-level tactical task and transmitting the combat schemes to the task sequence generation module; the task sequence generation module is used for converting the combat scheme into task sequences corresponding to the intelligent agents and distributing the task sequences to the corresponding intelligent agents; and the agent control module is used for controlling the agents to execute combat tasks according to the distributed tasks, controlling the agents to sense battlefield information and feeding back the battlefield information to the task understanding module. The system collects and fuses battlefield situations obtained by equipment in the system in real time, and optimizes a decision process based on situation information so as to adapt to a complex and changeable battlefield environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned system collaboration, and in particular relates to an intelligent decision-making system for unmanned combat equipment based on reinforcement learning. Background Art

[0002] Intelligent drone perception technology enables drones to collect key information, such as maps and target locations, through sensors such as visual systems to enhance the capabilities of unmanned vehicles in autonomous navigation and obstacle avoidance. In addition, communication technology between drones and unmanned vehicles supports information exchange and collaborative control. Collaborative control technology based on deep reinforcement learning further optimizes the collaborative strategy between drones and unmanned vehicles, learning and performing complex tasks such as reconnaissance and rescue through neural networks and reinforcement learning algorithms.

[0003] Using sensors such as cameras and lidar, drones can achieve autonomous positioning, attitude estimation, environmental modeling, and target detection, providing necessary environmental information for autonomous flight and mission execution. Through communication equipment such as radio and satellite, drones establish communication protocols and links with unmanned vehicles, ensure communication security, and support information exchange and collaborative control. Drones process and provide key information such as maps, road networks, and target locations, assisting unmanned vehicles in achieving autonomous navigation, obstacle avoidance, and reconnaissance capabilities. However, these technologies have the following problems:

[0004] One-way information transmission and control: Existing technologies usually implement one-way information transmission from drones to unmanned vehicles, lacking a two-way collaborative control mechanism, resulting in information asymmetry and limiting the collaboration and autonomous response capabilities of both parties in abnormal situations. Summary of the invention

[0005] The present invention aims to propose an integrated ground-air unmanned collaborative decision-making system. The core design concept of the system is to build a two-way collaboratively controlled air-ground unmanned system collaborative decision-making system, which can quickly obtain the real-time situation of the air-ground integration and respond quickly. The system collects and integrates the battlefield situation data obtained by all equipment in the system in real time, and can optimize the decision-making process based on this situation information, and continuously evolve while learning to adapt to the complex and changing battlefield environment.

[0006] The present invention proposes an intelligent decision-making system for unmanned combat equipment based on reinforcement learning, which includes: a task understanding module, a solution building module, a task sequence generation module, and an intelligent agent control module;

[0007] A task understanding module receives task instructions, analyzes combat intent according to the task instructions, analyzes battlefield situation according to battlefield information, forms sub-level tactical tasks for each UAV agent and / or ground robot agent according to the task instructions and battlefield situation, and transmits the sub-level tactical tasks to the solution construction module;

[0008] The plan building module generates multiple feasible operational plans according to each sub-level tactical task and transmits the operational plans to the task sequence generation module;

[0009] A task sequence generation module converts the combat plan into a task sequence corresponding to each UAV agent and / or ground robot agent, and assigns the task sequence to the corresponding UAV agent and / or ground robot agent;

[0010] The intelligent agent control module controls the UAV intelligent agent and / or the ground robot intelligent agent to perform combat tasks according to their respective assigned tasks, and controls the UAV intelligent agent and / or the ground robot intelligent agent to perceive battlefield information and feed back the battlefield information to the task understanding module.

[0011] The present invention realizes an intelligent decision-making system that "learns by using". The task understanding module integrates the ground-air situation data, and then combines the military language instructions of human commanders to realize task-oriented ground-air integrated collaborative decision-making. After the optimization mode is turned on, the scoring feedback of human commanders can be recorded, and the model fine-tuning optimization based on reinforcement learning can be completed between tasks to achieve "learning by using". BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is an unmanned combat equipment intelligent decision-making system based on reinforcement learning of the present invention. DETAILED DESCRIPTION

[0013] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The present invention proposes an intelligent decision-making system for unmanned combat equipment based on reinforcement learning, which includes: a task understanding module, a solution building module, a task sequence generation module, and an intelligent agent control module.

[0014] The task understanding module receives the natural language of the human commander, parses the combat intent according to the natural language, analyzes the battlefield situation according to the battlefield information fed back by the intelligent agent control module, forms sub-first-level tactical tasks for each UAV intelligent agent and / or ground robot intelligent agent according to the battlefield situation, and transmits the sub-first-level tactical tasks to the solution construction module.

[0015] The plan construction module generates multiple feasible combat plans based on each sub-level tactical task and transmits the combat plans to the task sequence generation module.

[0016] The task sequence generation module converts the combat plan into a task sequence corresponding to each UAV agent and / or ground robot agent, and assigns the task sequence to the corresponding UAV agent and / or ground robot agent.

[0017] The intelligent agent control module controls the UAV intelligent agent and / or the ground robot intelligent agent to perform combat tasks according to their respective assigned tasks, and controls the UAV intelligent agent and / or the ground robot intelligent agent to perceive battlefield information and feed back the battlefield information to the task understanding module.

[0018] When performing their respective assigned tasks, the UAV intelligent agent and / or ground robot intelligent agent independently decides on actions based on the assigned task objectives and adjusts combat methods based on real-time battlefield information.

[0019] The task understanding module analyzes the battlefield situation based on the battlefield information fed back by the agent control module, specifically including:

[0020] The task understanding module aggregates the battlefield information fed back by the intelligent agent control module, which is the battlefield situation semantic information perceived by different UAV intelligent agents and / or ground robot intelligent agents, and further aggregates the battlefield situation semantic information with the combat intent as the battlefield situation, and further forms sub-level tactical tasks for each UAV intelligent agent and / or ground robot intelligent agent based on the battlefield situation.

[0021] The system optimizes the task understanding module, and the specific process is: the task understanding module first generates multiple sub-first-level tactical tasks, sorts the sub-first-level tactical tasks from best to worst according to the evaluation scores of the sub-first-level tactical tasks, and selects the tasks to be executed from the sub-first-level tactical task sequence in a descending order.

[0022] The ground-air integrated unmanned collaborative decision-making system is built based on a task-driven hierarchical decision-making framework. The mission objectives will be broken down into tactical mission objectives for both air-ground unmanned equipment.

[0023] The system architecture of the present invention is as follows: the task-driven hierarchical decision-making framework includes a task understanding module, a solution construction module, a task sequence generation module, and an agent control module. The task understanding module can analyze the combat intent according to the natural language of the human commander, analyze the battlefield situation according to the feedback of the agent control module, and form sub-level tactical tasks for different equipment. Each sub-level tactical task will be handed over to the solution construction module to generate multiple feasible combat plans. After the combat plan passes through the task sequence generation module, it will be converted into a specific task sequence and assigned to different agents in the agent control module for execution.

[0024] The UAV agent and the ground robot agent are responsible for the execution of specific tasks. They make autonomous decisions and actions based on the assigned task sequence objectives and dynamically adjust according to the real-time battlefield situation. The UAV agent and the ground robot agent will process the situation information they obtain through the model and then send it back to the task understanding module.

[0025] The task planning module aggregates the situation semantic information sent back by different unmanned equipment, and further aggregates it with the task-level combat intent, then iterates the sub-task splitting as the battlefield situation changes.

[0026] The present invention also proposes a solution for optimizing the task understanding module. The task understanding module is centered on the task planning model and is responsible for processing natural language task instructions and battlefield situations and generating subtask objectives. When the system enters the optimization process, the training optimization process is as follows:

[0027] Task understanding module: At least n subtasks will be generated each time, and the human commander needs to sort these n subtasks. That is, through the commander's expert experience, the n output solutions are scored and sorted to complete the reinforcement learning fine-tuning based on human feedback.

[0028] The remaining modules are the same as the non-training state. The true value of the planning model optimization comes from the human commander's score of its output plan, and the remaining modules are only necessary supplements to maintain the training process. The basic principle of training is to improve the planning effect of the planning model by continuously adjusting the task instructions and situation information. The task instructions come from the human commander, and the situation information can come from the real environment or the simulation engine.

[0029] Collect data collected by UAV agents and ground robot agents during their actions, such as position, speed, direction, target, status, etc., as well as task completion status, such as whether the expected effect is achieved, whether losses are incurred, whether conflicts occur, etc. This system uses the task effect evaluation module to learn tactical task execution effect prediction models from data, such as regression models, classification models, clustering models, etc., and selects the optimal model based on different evaluation indicators, such as accuracy, recall rate, F1 value, etc., to evaluate the current task effect and predict future task effects. This system feeds back the evaluated effect data to the task planning model. The task planning model processes a large amount of combat process data into important information that human commanders need to pay attention to, and feeds back to human commanders in the process. It is also used to adjust and optimize task sequences and combat plans, realizing a data-driven tactical task effect evaluation method.

[0030] Example 1

[0031] The following is an example of using the system of the present invention to perform an unmanned combat mission. The steps of performing the combat mission by the system include:

[0032] (1) Command intention analysis: The human commander issues a mission command, and the mission understanding module analyzes the command content, which includes UAV formation configuration, reconnaissance and jamming task allocation, target indication, and retreat instructions. The mission understanding module converts the combat intent into the initial task sequence.

[0033] (2) Situation monitoring and plan building: Based on the battlefield situation (including enemy and friendly positions, force distribution, etc.), design multiple combat plans, including task allocation, execution sequence, and action strategy. Consider various tactical combinations and resource allocation.

[0034] (3) Task sequence generation and feedback: The selected combat plan is converted into a detailed task sequence and fed back to the commander for confirmation:

[0035] Generate specific flight routes, mission roles and action instructions. Break down the mission sequence into individual drones and ground robot agents.

[0036] (4) Autonomous decision-making and collaborative execution: Unmanned intelligent agents make autonomous decisions and collaborative execution based on the task sequence:

[0037] The drone reconnaissance team and the jamming team perform reconnaissance and jamming tasks respectively according to the instructions. The ground robot breaks through the defense line based on the information provided by the drone and enters the stronghold to perform the clearing mission. The intelligent agent adjusts its action strategy in real time to cope with the dynamically changing battlefield environment.

[0038] (5) Action data collection and learning: The unmanned agent collects real-time data during execution:

[0039] Collect information such as the agent's position, behavior, target state, and environmental feedback. The agent updates its action strategy in real time through distributed deep reinforcement learning.

[0040] (6) Task effect evaluation and optimization: Analyze and collect data to evaluate the task execution effect:

[0041] Use machine learning models (such as regression and classification) to predict and evaluate combat effectiveness. Optimize task sequences and combat plans based on evaluation results.

[0042] (7) Result reporting and strategy adjustment: After completing the mission, the unmanned intelligent agent returns to the base and reports to the commander:

[0043] Provide detailed reports on mission execution results (whether the target has been cleared, whether there have been any losses, etc.). Based on the evaluation data, the mission planning model adjusts future combat strategies.

[0044] Example 2

[0045] Mission background: Use drones and ground robots to clear out an enemy stronghold, eliminate all enemy militants, protect our personnel and equipment, and avoid unnecessary casualties and losses.

[0046] The human commander issues mission instructions to the system through military language, such as: UAV formation 1 is divided into two teams, one team is responsible for high-altitude reconnaissance of enemy strongholds to obtain the distribution of enemy forces and firepower, and the other team is responsible for low-altitude interference with the enemy's air defense system to reduce the enemy's air defense capabilities. At the same time, it provides target instructions to ground robot formation 1 to assist ground robot formation 1 in breaking through the enemy's defense line and entering the enemy's stronghold. After completing the clearing mission, UAV formation 1 and ground robot formation 1 return to the base and report the mission results to the human commander.

[0047] The system parses the combat intentions of human commanders through the task understanding module, analyzes the battlefield situation, such as the enemy's position, number, equipment, actions, etc., as well as our resources, capabilities, limitations, etc., according to the situation monitoring module of the task planning large model, and generates multiple feasible combat plans through the plan construction module, such as different task sequences, allocation plans, action strategies, etc. The combat plans are converted into specific task sequences through the task sequence generation module and assigned to different intelligent agents for execution. For example, UAVs 1-1, 1-2, and 1-3 fly over area A at an altitude of 5,000 meters, conduct high-altitude reconnaissance of enemy strongholds, obtain the distribution of enemy forces and firepower, and transmit the information to UAVs 1-4, 1-5, and 1-6 and ground robots 1-1, 1-2, and 1-3 in real time; UAVs 1-4, 1-5, and 1-6 fly over area B at an altitude of 1,000 meters to conduct high-altitude reconnaissance of enemy strongholds, obtain the distribution of enemy forces and firepower, and transmit the information to UAVs 1-4, 1-5, and 1-6 and ground robots 1-1, 1-2, and 1-3 in real time; UAVs 1-4, 1-5, and 1-6 fly over area B at an altitude of 1,000 meters to conduct high-altitude reconnaissance of enemy air defense The system conducts low-altitude interference and uses electronic countermeasure equipment to interfere with enemy radars and missiles, reduce enemy air defense capabilities, and at the same time receives information transmitted by drones 1-1, 1-2, and 1-3, provides target instructions to ground robots 1-1, 1-2, and 1-3, and assists ground robots 1-1, 1-2, and 1-3 in breaking through enemy defense lines and entering enemy strongholds; ground robots 1-1, 1-2, and 1-3, according to the target instructions provided by drones 1-4, 1-5, and 1-6, select the best attack route, use their own concealment, mobility, firepower and other capabilities to break through enemy defense lines, enter enemy strongholds, and eliminate all enemy militants, eliminate all enemy threats, protect our personnel and equipment, and avoid unnecessary casualties and losses; after completing the clearing mission, drone formation 1 and ground robot formation 1 fly or drive to area C, return to the base, and report the mission results to the human commander. UAV agents and ground robot agents make autonomous decisions and collaborative executions based on the task sequence, dynamically adjust action strategies according to battlefield conditions, and use their own perception, control, communication and other capabilities to complete complex tactical tasks such as reconnaissance, strike, interference, and protection.

[0048] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent decision-making system for unmanned combat equipment based on reinforcement learning, characterized in that: It includes: Task understanding module, solution building module, task sequence generation module, agent control module; A task understanding module receives task instructions, analyzes combat intent according to the task instructions, analyzes battlefield situation according to battlefield information, forms sub-level tactical tasks for each UAV agent and / or ground robot agent according to the task instructions and battlefield situation, and transmits the sub-level tactical tasks to the solution construction module; The plan building module generates multiple feasible operational plans according to each sub-level tactical task and transmits the operational plans to the task sequence generation module; A task sequence generation module converts the combat plan into a task sequence corresponding to each UAV agent and / or ground robot agent, and assigns the task sequence to the corresponding UAV agent and / or ground robot agent; The intelligent agent control module controls the UAV intelligent agent and / or the ground robot intelligent agent to perform combat tasks according to their respective assigned tasks, and controls the UAV intelligent agent and / or the ground robot intelligent agent to perceive battlefield information and feed back the battlefield information to the task understanding module.

2. The unmanned combat equipment intelligent decision-making system based on reinforcement learning according to claim 1 is characterized in that: When performing their respective assigned tasks, the UAV intelligent agent and / or ground robot intelligent agent independently decides on actions based on the assigned task objectives and adjusts combat methods based on real-time battlefield information.

3. The unmanned combat equipment intelligent decision-making system based on reinforcement learning according to claim 1 is characterized in that: The task understanding module analyzes the battlefield situation based on the battlefield information fed back by the agent control module, specifically including: The task understanding module aggregates the battlefield information fed back by the intelligent agent control module, which is the battlefield situation semantic information perceived by different UAV intelligent agents and / or ground robot intelligent agents, and further aggregates the battlefield situation semantic information with the combat intent as the battlefield situation, and further forms sub-level tactical tasks for each UAV intelligent agent and / or ground robot intelligent agent based on the battlefield situation.

4. The unmanned combat equipment intelligent decision-making system based on reinforcement learning according to claim 1 is characterized in that: It further includes: the system optimizes the task understanding module, and the specific process is: The task understanding module first generates multiple sub-level tactical tasks, sorts the sub-level tactical tasks from best to worst according to their evaluation scores, and selects the tasks to be executed from the sub-level tactical task sequence in a descending order.

5. The unmanned combat equipment intelligent decision-making system based on reinforcement learning according to claim 1 is characterized in that: The mission instructions include natural language from a human commander.

6. The unmanned combat equipment intelligent decision-making system based on reinforcement learning according to claim 1 is characterized in that: The battlefield information is battlefield information fed back by the intelligent agent control module or battlefield information generated by a simulation engine.

7. The unmanned combat equipment intelligent decision-making system based on reinforcement learning according to claim 6 is characterized in that: It further includes: evaluating the effectiveness of combat missions based on battlefield information fed back by the intelligent body control module, sorting the corresponding sub-level tactical tasks from best to worst according to the combat mission effects, and optimizing the combat plan.

Images