An unmanned swarm evolution system and method based on meta-action sequence reinforcement learning

Through the unmanned cluster evolution system based on meta-action sequence reinforcement learning, the problems of difficulty in building an unmanned cluster decision model and poor environmental adaptability are solved, and the evolution of unmanned cluster task strategy and task capabilities are achieved.

CN114510012BActive Publication Date: 2024-11-29THE 54TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210142965.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-16
Publication Date
2024-11-29
Estimated Expiration
2042-02-16

AI Technical Summary

Technical Problem

It is difficult to build an unmanned cluster decision model, poor environmental adaptability, and lack of evolutionary capabilities, making it difficult to achieve effective collaborative decision-making in a diversified environment.

Method used

Adopt unmanned cluster evolution system based on meta-action sequence reinforcement learning, including unmanned cluster simulation environment, cluster strategy model library, performance evaluation module, situational awareness module, cluster strategy evolution module and cluster decision-making module, and form new ones through self-learning algorithms Cluster strategy to achieve the evolution of unmanned cluster task strategy.

Benefits of technology

It improves the generalization ability and robustness of unmanned clusters, can effectively adapt to environmental changes in complex dynamic scenarios, and improves the mission ability and combat effectiveness of unmanned clusters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114510012B_ABST
    Figure CN114510012B_ABST
Patent Text Reader

Abstract

The present invention discloses an unmanned cluster evolution system and method based on meta-action sequence reinforcement learning, which belongs to the field of unmanned cluster autonomous task collaboration. The present invention first decomposes the many tasks faced by the unmanned cluster into basic meta-actions, constructs the mapping relationship between environmental information and meta-action sequences through a valuation network, uses the task performance evaluation index as the reward function of reinforcement learning, and realizes the evolution of the unmanned cluster task capability through adaptive reinforcement learning in many scenarios. The system has strong generalization ability and high robustness when dealing with complex dynamic scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of unmanned swarm autonomous task collaboration, and in particular refers to an unmanned swarm evolution system and method based on meta-action sequence reinforcement learning. Background Art

[0002] At present, the demand for unmanned operations has risen sharply in all fields. The traditional single-platform unmanned autonomous system model has gradually limited its due role in more scenarios due to its low risk resistance and single task type. Seeking new technological breakthroughs is the only way to maintain the advantages of unmanned operations. Among them, the use of existing unmanned units to build an "unmanned cluster", in which different units with limited capabilities work together to complete complex tasks and achieve low-cost, low-risk, and high-efficiency task goals, can represent the development trend of the unmanned system field in the future. The unmanned cluster system is composed of a certain number of single-function or multi-function unmanned systems. Under the support of the sympathetic network, nodes interact with each other and perform sympathetic behaviors such as information exchange and feedback, incentives, and responses. The unmanned cluster system as a whole has the characteristics of capability emergence, which can realize the autonomous decision-making of a single platform behavior and the behavior coordination between platforms, and finally generate the emergence of task capabilities. The unmanned cluster system is not a simple formation of multiple unmanned systems, and its task capability is not the superposition of the single capabilities of many platforms. Instead, it is a collection, scheduling, and arrangement of multiple unmanned platforms through scientific methods, and the organic coupling of the cluster self-organization mechanism and the behavior regulation mechanism has generated new capabilities or the original capabilities have undergone qualitative changes.

[0003] In a diverse environment, the unmanned swarm mission elements are interrelated, restricted and influenced, making the collaborative decision-making of unmanned swarms very difficult. The main problems involved are mainly reflected in:

[0004] 1) The model is difficult to establish: Unmanned swarms are composed of many drones with different characteristics, types and purposes. The mission process depends not only on their own capabilities, but also on the constraints of the natural environment and target status. This leads to a sharp increase in the difficulty of unmanned swarm collaborative modeling;

[0005] 2) Difficulty in problem solving: Cluster collaborative decision-making is a multi-parameter, multi-constrained non-deterministic polynomial problem, which can easily lead to combinatorial explosion under multiple input conditions. It is necessary to study efficient dimensionality reduction methods and optimization strategies;

[0006] 3) High task coupling: Tasks in a cluster are usually interrelated and mutually constrained, that is, there are complex constraints and different task requirements, such as time, space, load matching, and task priority. Multi-task coupling greatly increases the difficulty of self-coordination;

[0007] 4) Weak evolutionary ability: Drone swarms mostly perform tasks in highly dynamic and highly confrontational environments. The time, space, and state of the objects are all unknown in advance. The unmanned swarm system needs to continuously learn decision-making capabilities in new environments. Without evolutionary capabilities, it is not enough to support the correctness of decisions in a changing environment.

[0008] With the spread and diffusion of artificial intelligence technology and unmanned system technology around the world, reinforcement learning has shown great application potential in many fields. Multiple agents learn in a "trial and error" manner, and the rewards obtained by interacting with the environment guide actions to enable agents to obtain the maximum reward. Compared with traditional decision-making methods, reinforcement learning systems need to rely mainly on their own experience to learn, and the reinforcement signals provided by the environment evaluate the quality of the actions produced. By executing the actions that can obtain the maximum reward a limited number of times, the best model is determined, which can promote the transformation of unmanned swarms into evolvable intelligent systems and improve the scenario adaptability of unmanned swarm systems. Summary of the invention

[0009] In order to overcome the problems of difficulty in constructing current unmanned swarm decision models, poor environmental adaptability, and lack of evolutionary capabilities, the present invention provides an unmanned swarm evolution system and method based on meta-action sequence reinforcement learning. The present invention has strong generalization capabilities and high robustness when dealing with complex dynamic scenes.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] An unmanned swarm evolution system based on meta-action sequence reinforcement learning, including an unmanned swarm simulation environment, a swarm strategy model library, an effectiveness evaluation module, a situation awareness module, a swarm strategy evolution module and a swarm decision module;

[0012] The unmanned swarm simulation environment is used to provide various unmanned systems, including drones, unmanned vehicles, and unmanned ship models. It can import various 3D environment models and provide unmanned system drive and environment information reading interfaces. It supports the simulation and deduction of intelligent algorithms such as unmanned swarm collaborative flight, unmanned swarm mission planning, and deep reinforcement learning.

[0013] The cluster strategy model library consists of a sequence of cluster meta-actions, which include all possible states of each unmanned system and all actions that can be executed; the cluster strategy model library is a mapping set of unmanned system states and individual actions based on a hierarchical state machine;

[0014] The effectiveness evaluation module is used to evaluate the cluster strategies in the strategy model library, determine the starting and ending time of the strategy, and the cluster combat effectiveness value at the end, and perform quantitative evaluation and calculation on the increase and decrease values ​​according to the initial cluster combat effectiveness value and the cluster combat effectiveness value at the end to obtain the strategy evaluation results;

[0015] The situation awareness module is used to perceive the current enemy and friendly situation and geographical environment information, and provide input data for the subsequent cluster strategy learning evolution module and cluster decision-making module. The perception data includes enemy and friendly position data, movement direction data, speed data, and the position of typical environmental elements. The enemy and friendly position data consists of longitude, latitude, and altitude.

[0016] The cluster strategy evolution module forms a new cluster strategy through a self-learning algorithm based on the simulation data of the unmanned cluster simulation environment and the strategy evaluation results, provides alternative strategies for the strategy library, forms a supplementary strategy for the strategy sample library, and realizes the evolution of unmanned cluster task strategies;

[0017] The cluster decision module is used to evaluate the matching degree between the real-time perceived situation data and the initial state conditions of the decision in the decision library. If the match is successful, the execution sequence of each drone in the unmanned cluster strategy library is obtained for subsequent control; if the match is unsuccessful, it switches to online self-learning calculation to obtain the strategy, and each unmanned system executes the online learned strategy as the subsequent control sequence.

[0018] An unmanned swarm evolution method based on meta-action sequence reinforcement learning comprises the following steps:

[0019] Step 1: Provide the entire system with drone, vehicle, and ship models through the unmanned cluster simulation environment, and provide an unmanned system drive and environment information reading interface. The input and output data of the entire system are generated in this environment.

[0020] Step 2: Perform situation awareness in the unmanned swarm simulation environment through the situation awareness module to obtain the current enemy and friendly situation and geographical environment information, including enemy and friendly position data, movement direction data, speed data, building location, and terrain undulation data;

[0021] Step 3: The situation data from the situation awareness module is received through the cluster decision module, and the matching degree is calculated with the cluster state in the cluster strategy model library. If the matching is successful, the unmanned system driving interface of the unmanned cluster simulation environment is input according to the meta-action sequence of the unmanned cluster individuals in the cluster strategy model library; if the matching is unsuccessful, the cluster strategy evolution module is triggered to enter the online learning process;

[0022] Step 4: Perform self-learning evolution through the group strategy evolution module, deduce alternative strategies for the current state information of the unmanned cluster through the unmanned cluster simulation environment, generate new cluster strategies through the multi-agent self-learning algorithm, and output the strategy to the performance evaluation module; the self-learning process adopts the multi-agent deep reinforcement learning algorithm, uses the task performance evaluation results as the reward incentive for reinforcement learning, and conducts centralized training and distributed execution of deep neural networks through multiple scenarios provided by the unmanned cluster simulation environment;

[0023] Step 5: Calculate the performance value of the candidate strategy by using the performance evaluation algorithm through the performance evaluation module. If the performance value reaches the set threshold, output the new cluster strategy to add to the cluster strategy model library, and update the cluster strategy model library; if the performance value is less than the set threshold, continue to deduce the candidate strategy through the unmanned cluster simulation environment, repeat the self-learning process until a cluster strategy with a performance value that meets the standard is generated, and the mapping relationship between the current cluster state information and the current cluster strategy is fitted to the deep neural network; the performance evaluation algorithm constructs an indicator system and performance impact weights based on expert knowledge, and quantitatively calculates the performance value through the comprehensive weighting of the indicator system parameters and weights;

[0024] Step 6: The cluster decision module executes the decision and updates the cluster status for the successfully matched unmanned cluster status. The unmanned system model in the unmanned cluster simulation environment completes model control according to the received cluster strategy, drives the entire unmanned cluster to the next state, and iterates situational awareness until the mission is completed.

[0025] The beneficial effects of the present invention are:

[0026] 1. The present invention decomposes the many tasks faced by unmanned swarms into basic meta-actions, constructs the mapping relationship between environmental information and meta-action sequences through the valuation network, uses the task performance evaluation index as the reward function of reinforcement learning, and realizes the evolution of unmanned swarm task capabilities through adaptive reinforcement learning in many scenarios. The system has strong generalization ability and high robustness when dealing with complex dynamic scenes.

[0027] 2. The present invention constructs a mapping relationship between cluster state information and meta-action sequences through a neural network, uses task performance evaluation results as reward incentives for reinforcement learning, and realizes the evolution of unmanned cluster task capabilities through multi-hypothesis adaptive reinforcement learning. It can promote the transformation of unmanned cluster task capabilities from optimization to evolution, and can bring about a systematic improvement in the combat effectiveness of unmanned clusters.

[0028] 3. The present invention can model the complex evolution process of unmanned swarm decision-making capabilities as a reinforced self-learning process of meta-action sequences, which can overcome the problems of the current unmanned swarm decision-making model, such as difficulty in constructing, poor environmental adaptability, and lack of evolutionary capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is an architecture diagram of the system of an embodiment of the present invention.

[0030] Figure 2 It is a diagram for constructing the unmanned cluster performance evaluation index in an embodiment of the present invention.

[0031] Figure 3 It is a schematic diagram of the evolution of unmanned cluster strategy in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0033] An unmanned swarm evolution system based on meta-action sequence reinforcement learning includes six main components: unmanned swarm simulation environment, swarm strategy model library, effectiveness evaluation module, situation awareness module, swarm strategy evolution module and swarm decision module. The functions of each component are described as follows:

[0034] Unmanned swarm simulation environment: It provides models of various unmanned systems such as drones, unmanned vehicles, and unmanned ships. It can import various 3D environment models and provide interfaces for driving unmanned systems and reading environmental information. It supports the simulation and deduction of intelligent algorithms such as unmanned swarm collaborative flight, unmanned swarm mission planning, and deep reinforcement learning.

[0035] Cluster strategy model library: The cluster strategy model consists of a sequence of cluster meta-actions. The meta-actions include all possible states of each unmanned system and all actions that can be performed. Based on the idea of ​​a hierarchical state machine, a mapping set of unmanned system states and individual actions is constructed to form a strategy model library.

[0036] Effectiveness evaluation module: Evaluate the cluster strategies in the strategy model library through the task effectiveness evaluation model, determine the starting and ending time of the strategy, and the cluster combat effectiveness value at the end; obtain the cluster's initial combat effectiveness value and the cluster combat effectiveness value at the end, perform quantitative evaluation and calculation on the increase and decrease values, and obtain the strategy evaluation results.

[0037] Situation awareness module: Based on the unmanned swarm simulation environment, it perceives the current enemy and friendly situations and geographical environment information, and provides input data for the subsequent swarm strategy learning evolution module and swarm decision-making module. The perception data mainly includes enemy and friendly position data (consisting of longitude, latitude, altitude, etc.), movement direction data, speed data, and the location of typical environmental elements.

[0038] Cluster strategy evolution module: Through the collected simulation data and strategy evaluation results, the degree of change of the cluster effectiveness value is examined from the meta-action sequence process in the sample data, and a new cluster strategy is formed through a self-learning algorithm. Alternative strategies for the strategy library are provided to form a supplementary strategy for the strategy sample library, thereby realizing the evolution of unmanned cluster task strategies.

[0039] Cluster decision module: The real-time perceived situation data is evaluated and calculated for matching with the initial state conditions of the decision in the decision library. If the match is successful, the execution sequence of each drone in the unmanned cluster strategy library is obtained for subsequent control; if the match is unsuccessful, the strategy is obtained through online self-learning calculation, and each unmanned system executes the subsequent control sequence of the online learned strategy.

[0040] An unmanned swarm evolution method based on meta-action sequence reinforcement learning comprises the following steps:

[0041] (1) The unmanned swarm simulation environment provides the entire system with models of drones, unmanned vehicles, and unmanned ships. It can import multiple types of 3D environmental models and provide interfaces for driving the unmanned system and reading environmental information. The input and output data of the entire system are generated in this environment.

[0042] (2) The situation awareness module performs situation awareness in an unmanned swarm simulation environment to obtain the current enemy and friendly situation and geographical environment information, mainly including enemy and friendly position data (consisting of longitude, latitude, and altitude), movement direction data, speed data, building location, terrain undulation data, etc.

[0043] (3) The cluster decision module receives the situation data from the situation awareness module and calculates the matching degree with the cluster state in the cluster strategy model library. If the matching is successful (the matching degree is greater than or equal to the set threshold), the meta-action sequence of the unmanned cluster individuals in the cluster strategy model library is input into the unmanned system driving interface of the unmanned cluster simulation environment; if the matching is unsuccessful (the matching degree is less than the set threshold), the cluster strategy evolution module is triggered to enter the online learning process.

[0044] (4) The swarm strategy evolution module performs self-learning evolution, deduce alternative strategies for the current state information of the unmanned swarm through the unmanned swarm simulation environment, generate new swarm strategies through the multi-agent self-learning algorithm, and output the strategies to the performance evaluation module.

[0045] (5) The performance evaluation module uses the performance evaluation algorithm to calculate the performance value of the alternative strategy. If the performance value is greater than or equal to the set threshold, the new cluster strategy is output and added to the cluster strategy model library, and the strategy model library is updated. If the performance value is less than the set threshold, the alternative strategy is continued to be deduced through the unmanned cluster simulation environment, and the self-learning process is repeated until the optimal cluster strategy is generated.

[0046] (6) The cluster decision module executes the decision and updates the cluster status for the successfully matched unmanned cluster status. The unmanned system model in the unmanned cluster simulation environment completes the model control according to the received cluster strategy, drives the entire unmanned cluster to the next state, and iterates the situation awareness until the mission is completed.

[0047] Here is a more specific example:

[0048] An unmanned swarm evolution system based on meta-action sequence reinforcement learning includes six components: unmanned swarm simulation environment, swarm strategy model library, effectiveness evaluation module, situation awareness module, swarm strategy evolution module and swarm decision module. Figure 1As shown, the implementation of each part of the system is as follows:

[0049] 1. Unmanned swarm simulation environment: It provides various unmanned system models such as drones, unmanned vehicles, and unmanned ship models, can import various 3D environment models, and provides unmanned system drive and environment information reading interfaces, supports the simulation and deduction of intelligent algorithms such as unmanned swarm collaborative flight, unmanned swarm mission planning, and deep reinforcement learning.

[0050] This embodiment is based on the unmanned cluster simulation environment of ROS+Gazebo, where ROS (Robot Operating System) is a robot operating system, which is a highly flexible middleware for writing robot software programs. It can provide some standard operating system services such as hardware abstraction, underlying device control, common function implementation, inter-process messaging, and data packet management, and realize information interaction between different nodes through the publish-subscribe mechanism. Gazebo is a three-dimensional multi-robot dynamic simulation environment that comes with ROS. It can simulate complex drones, unmanned vehicle models, and real-world environment models. The main functions include: 1) Building an unmanned platform motion simulation model; 2) Building simulation models of various scenes in the real world; 3) Building simulation models of multiple types of sensors; 4) Adding real-world physical properties to the unmanned platform model.

[0051] 2. Cluster strategy model library: The cluster strategy model consists of a sequence of cluster meta-actions. The meta-actions include all possible states of each unmanned system and all actions that can be performed. Based on the idea of ​​a hierarchical state machine, a mapping set of unmanned system states and individual actions is constructed to form a strategy model library.

[0052] 2.1 Meta-action modeling is implemented using a hierarchical state machine: a mapping between actions and individual states of unmanned swarms is constructed. The main design idea is as follows: unmanned swarm individuals can execute certain specific action requests at any given moment, but unmanned swarm individuals usually cannot jump from any state to another arbitrary state. The option for unmanned swarm individuals to achieve the correct jump state is called state transition, and the set of states, the set of state transitions, and the variables used to record the correct state are integrated to define a state machine.

[0053] 2.2 Constructing an action state machine through configuration files: By configuring the Moves attribute for each unmanned cluster individual, an action system is given to it, and in the action system, the action state machine is constructed by configuring the States attribute. The configuration of the Moves attribute is only related to the movement capability of the drone. The States attribute can construct a directed graph. Each vertex of the directed graph is an action fragment. Each edge of the directed graph has a corresponding weight. A shortest path can be calculated based on the weight and added to the current action queue. Different action queues together constitute the unmanned cluster strategy model library.

[0054] 3. Effectiveness evaluation module: Evaluate the cluster strategies in the strategy model library through the task effectiveness evaluation model, determine the starting and ending time of the strategy, and the cluster combat effectiveness value at the end; obtain the cluster's initial combat effectiveness value and the cluster combat effectiveness value at the end, perform quantitative evaluation and calculation on the increase and decrease values, and obtain the strategy evaluation results.

[0055] 3.1 Construction of mission effectiveness indicators: The evaluation of mission effectiveness can be carried out from three aspects: mission performance evaluation, survival performance evaluation, and stability performance evaluation. Mission performance refers to the ability of UAVs to perform tasks such as environmental reconnaissance and surveillance. UAVs need to make judgments immediately after detecting targets, and identify, track or make decisions on them; survival performance refers to the ability of UAVs to maintain safe flight under complex geographical and meteorological conditions. This indicator is the basic prerequisite for them to perform various tasks; stability performance refers to the high stability required by UAVs when performing various tasks in complex and diverse environments, so that the UAVs can quickly return to their original equilibrium state after being disturbed. The three types of evaluation indicators are broken down into the following categories: Figure 2 shown.

[0056] 3.2 Mission effectiveness evaluation: This embodiment adopts the multivariate statistical analysis method of the structural equation model. Starting from the three indicators of mission performance evaluation, survivability performance evaluation, and stability performance evaluation, a scientific and reasonable combat effectiveness evaluation model for different objects and different tasks is constructed. For different evaluation objects (rotary-wing UAVs, fixed-wing UAVs, bionic birds, etc.), a unified standard object-oriented design specification and different mission models (such as reconnaissance, support, strike, etc.) that comply with the open system architecture are constructed. The types, numbers, and setting ranges of evaluation parameters for different objects are clarified, and a quantitative relationship model between mission efficiency, mission effect, mission cost, mission capability, and mission contribution is constructed.

[0057] 4. Situation awareness module: Based on the unmanned cluster simulation environment, it perceives the current enemy and friendly situations and geographical environment information, and provides input data for the subsequent cluster strategy learning evolution module and cluster decision-making module. The perception data mainly includes enemy and friendly position data (consisting of longitude, latitude, altitude, etc.), movement direction data, speed data, and the location of typical environmental elements.

[0058] The situational awareness module simulates the perception payload carried by the drone based on the unmanned cluster simulation environment, and cooperates with the map model to realize the drone's active perception of the environmental situation. The model parameters include detection range, detection height, detection angle, etc. This embodiment takes the unmanned cluster carrying visual payload to perform target search tasks as an example. The mission area can be detected at all times, and each drone can independently obtain the situation information within the current detection range of its own sensor.

[0059] 5. Cluster decision module: The real-time perceived situation data is evaluated and calculated for matching with the initial state conditions of the decision in the decision library. If the match is successful, the execution sequence of each unmanned platform in the unmanned cluster strategy model library is obtained for subsequent control; if the match is unsuccessful, the online self-learning calculation is used to obtain the strategy, and each unmanned system executes the online learning strategy as the subsequent control sequence.

[0060] 6. Cluster strategy evolution module: Through the collected simulation data and strategy evaluation results, the degree of change of the cluster effectiveness value is examined from the meta-action sequence process in the sample data, and a new cluster strategy is formed through a self-learning algorithm. Alternative strategies for the strategy library are provided to form a supplementary strategy for the strategy sample library, thereby realizing the evolution of unmanned cluster task strategies.

[0061] 6.1 Obtaining cluster strategy sample library data: The simulation sample data collected through the unmanned cluster simulation environment includes enemy and our position data (composed of longitude, latitude, altitude and other information), load data, movement direction data, speed data, building location data, remaining fuel data, etc. The sample library of different unmanned cluster strategies can be obtained through different constraints (the cluster effectiveness value is 20% of the initial value as the end condition of the strike mission, 20% to 0 of the initial value as the end condition of the reconnaissance mission, and the cluster effectiveness value of our side is greater than the enemy cluster effectiveness value in a fixed area and fixed time period as the constraint condition for the success of the support mission).

[0062] 6.2 Obtain strategy evaluation results: Evaluate the strategies in the unmanned cluster strategy sample library through mission effectiveness (focusing on the three performance indicators of UAV survivability, mission capability, and stability capability), determine the start and end time of the strategy (covering support, reconnaissance, strike and other tasks), and the cluster mission effectiveness value at the end; obtain the cluster's initial mission effectiveness value and the cluster mission effectiveness value at the end, and perform quantitative evaluation and calculation on the increase and decrease values; perform quantitative evaluation and calculation on the effectiveness values ​​of multiple cluster strategies under the same conditions, obtain strategy evaluation results, and form a strategy evaluation sequence; extract different unmanned cluster strategy libraries formed by the best performance strategy, including strategy constraints and implementation steps for each UAV in the cluster.

[0063] 6.3 Multi-agent self-learning: Figure 3As shown in the figure, a multi-agent deep reinforcement learning algorithm is used to implement the self-learning process. Multi-agent deep reinforcement learning achieves the goal of cluster strategy evolution through centralized training and distributed execution. Allowing strategies to use additional information to simplify training and adding additional shared information about other agent strategies in the valuation network (Q network) can improve the efficiency of decision generation. In each round, a specific sub-strategy is randomly selected for each agent to execute, and a replay buffer is established for each sub-strategy of the agent. The combat task is divided into multiple sub-tasks and carried out simultaneously, which is conducive to each agent forming a new cluster strategy.

[0064] 6.4 Unmanned cluster task deduction: The cluster strategy newly generated by the cluster strategy evolution module is deduced again in the simulation system, and the same cluster strategy effectiveness evaluation algorithm is used to implement the effectiveness evaluation of the new cluster strategy, calculate the effectiveness value of the new cluster strategy, and implement the strategy update of the cluster strategy sample library. The cluster strategy self-learning evolution process ends.

[0065] In summary, the present invention first decomposes the many tasks faced by unmanned swarms into basic meta-actions, constructs the mapping relationship between environmental information and meta-action sequences through the valuation network, uses the task performance evaluation index as the reward function of reinforcement learning, and realizes the evolution of unmanned swarm task capabilities through adaptive reinforcement learning in many scenarios. The system has strong generalization ability and high robustness when dealing with complex dynamic scenes.

Claims

1. An unmanned swarm evolution system based on meta-action sequence reinforcement learning, characterized in that: It includes unmanned swarm simulation environment, swarm strategy model library, effectiveness evaluation module, situation awareness module, swarm strategy evolution module and swarm decision module; The unmanned swarm simulation environment is used to provide multiple types of unmanned systems, including drones, unmanned vehicles, and unmanned ship models. It can import multiple types of 3D environment models and provide unmanned system drive and environment information reading interfaces. It supports unmanned swarm collaborative flight, unmanned swarm mission planning, and simulation and deduction of deep reinforcement learning intelligent algorithms. The cluster strategy model library consists of a sequence of cluster meta-actions, which include all possible states of each unmanned system and all actions that can be executed; the cluster strategy model library is a mapping set of unmanned system states and individual actions based on a hierarchical state machine; The effectiveness evaluation module is used to evaluate the cluster strategies in the strategy model library, determine the starting and ending time of the strategy, and the cluster combat effectiveness value at the end, and perform quantitative evaluation and calculation on the increase and decrease values ​​according to the initial cluster combat effectiveness value and the cluster combat effectiveness value at the end to obtain the strategy evaluation results; The situation awareness module is used to perceive the current enemy and friendly situation and geographical environment information, and provide input data for the subsequent cluster strategy learning evolution module and cluster decision-making module. The perception data includes enemy and friendly position data, movement direction data, speed data, and the position of typical environmental elements. The enemy and friendly position data consists of longitude, latitude, and altitude. The cluster strategy evolution module forms a new cluster strategy through a self-learning algorithm based on the simulation data of the unmanned cluster simulation environment and the strategy evaluation results, provides alternative strategies for the strategy library, forms a supplementary strategy for the strategy sample library, and realizes the evolution of unmanned cluster task strategies; The cluster decision module is used to evaluate the matching degree between the real-time perceived situation data and the initial state conditions of the decision in the decision library. If the match is successful, the execution sequence of each drone in the unmanned cluster strategy library is obtained for subsequent control; if the match is unsuccessful, it switches to online self-learning calculation to obtain the strategy, and each unmanned system executes the online learned strategy as the subsequent control sequence.

2. An unmanned swarm evolution method based on meta-action sequence reinforcement learning, characterized in that: The following steps are involved: Step 1: Provide the entire system with drone, vehicle, and ship models through the unmanned cluster simulation environment, and provide an unmanned system drive and environment information reading interface. The input and output data of the entire system are generated in this environment. Step 2: Perform situation awareness in the unmanned swarm simulation environment through the situation awareness module to obtain the current enemy and friendly situation and geographical environment information, including enemy and friendly position data, movement direction data, speed data, building location, and terrain undulation data; Step 3: The situation data from the situation awareness module is received through the cluster decision module, and the matching degree is calculated with the cluster state in the cluster strategy model library. If the matching is successful, the unmanned system driving interface of the unmanned cluster simulation environment is input according to the meta-action sequence of the unmanned cluster individuals in the cluster strategy model library; if the matching is unsuccessful, the cluster strategy evolution module is triggered to enter the online learning process; Step 4: Perform self-learning evolution through the group strategy evolution module, deduce alternative strategies for the current state information of the unmanned cluster through the unmanned cluster simulation environment, generate new cluster strategies through the multi-agent self-learning algorithm, and output the strategy to the performance evaluation module; the self-learning process adopts the multi-agent deep reinforcement learning algorithm, uses the task performance evaluation results as the reward incentive for reinforcement learning, and conducts centralized training and distributed execution of deep neural networks through multiple scenarios provided by the unmanned cluster simulation environment; Step 5: Calculate the performance value of the candidate strategy through the performance evaluation module using the performance evaluation algorithm. If the performance value reaches the set threshold, output the new cluster strategy to add to the cluster strategy model library, and update the cluster strategy model library. If the performance value is less than the set threshold, continue to deduce the candidate strategy through the unmanned cluster simulation environment, repeat the self-learning process, and generate a cluster strategy with a qualified performance value. The mapping relationship between the current cluster state information and the current cluster strategy is fitted to the deep neural network. The performance evaluation algorithm constructs an index system and performance impact weights based on expert knowledge, and quantitatively calculates the performance value through comprehensive weighting of index system parameters and weights; Step 6: The cluster decision module executes the decision and updates the cluster status for the successfully matched unmanned cluster status. The unmanned system model in the unmanned cluster simulation environment completes model control according to the received cluster strategy, drives the entire unmanned cluster to the next state, and iterates situational awareness until the mission is completed.

Citation Information

Patent Citations

  • Unmanned cluster cooperative game confrontation control method and system

    CN112182977A

  • VR virtual confrontation training system based on hierarchical behavior model

    CN112820164A