Unmanned cluster cooperative control method based on similar organism and data driving fusion and related equipment

Abstract

The invention provides an unmanned cluster cooperative control method based on similar organism and data driving fusion and related equipment. The method comprises the following steps: selecting a biological-like control model, and changing numerical values of decision parameters in the biological-like control model from traditional fixed values into self-adaptive time-varying decision parameters which change along with environmental perception through subsequent steps; aiming at the task requirement of the unmanned cluster, constructing a heterogeneous cross drilling environment; establishing a deep reinforcement learning framework based on a heterogeneous cross drilling environment; training a deep reinforcement learning framework by using historical state information and action data of an unmanned cluster, so that the deep reinforcement learning framework can generate adaptive time-varying decision parameters adaptive to a heterogeneous cross drilling environment; and inputting real-time state information sensed by the unmanned cluster at the current moment into the trained deep reinforcement learning framework to obtain an adaptive time-varying decision parameter, and taking the adaptive time-varying decision parameter as a decision parameter in the biological-like control model to perform cooperative control on the unmanned cluster.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent unmanned system collaborative control, and in particular to an unmanned cluster collaborative control method based on biological-like and data-driven fusion and related equipment. Background Art

[0002] Unmanned swarms are composed of many relatively autonomous, artificially developed unmanned autonomous moving bodies that have emerged through mutual cooperation and division of labor, thus showing the characteristics of complex intelligent behavior. These unmanned swarms meet the functional requirements of complex tasks in a low-cost, decentralized form, and autonomously coordinate planning and multi-domain collaboration in complex environments, showing an intelligence level that is difficult for a single unmanned autonomous moving body to achieve.

[0003] Among the many key technologies that affect the efficiency of unmanned swarm applications, the research on intelligent decision-making of swarms occupies a core position. Intelligent decision-making of unmanned swarms needs to stimulate individual wisdom, aggregate group intelligence, and make full use of individual distributed decision-making and group collaborative decision-making to provide the best decision results and solutions to achieve mission goals. Good intelligent swarm decision-making design can effectively improve the cluster's task execution capabilities, reduce cluster task execution resource consumption, and avoid task execution risks.

[0004] However, in complex environments, including adversarial environments, unmanned swarms still face problems such as strong limitations in communication perception, the complexity of large-scale systems, and the uncertainty of unknown environments. The decision parameters of existing biologically inspired unmanned swarm control models are fixed, which makes it impossible for swarms to adaptively adjust and optimize action decisions according to the environment in dynamic uncertain scenarios where the above problems exist. Summary of the invention

[0005] In order to solve the problem that the decision parameters of the existing unmanned swarm control model based on biological inspiration are fixed, resulting in the inability of the swarm to adaptively adjust and optimize action decisions according to the environment in dynamic uncertain scenarios, the present invention provides an unmanned swarm collaborative control method and related equipment based on the fusion of biological inspiration and data drive.

[0006] In a first aspect, the present invention provides an unmanned swarm collaborative control method based on biological-like and data-driven fusion, comprising:

[0007] Step 1: Select a quasi-biological control model, and through subsequent steps, change the value of the decision parameter in the quasi-biological control model from a traditional fixed value to an adaptive time-varying decision parameter that changes with environmental perception;

[0008] Step 2: Build a heterogeneous cross-training environment based on the mission requirements of the unmanned swarm;

[0009] Step 3: Establish a deep reinforcement learning framework based on a heterogeneous cross-training environment;

[0010] Step 4: Using the historical state information and action data of the unmanned swarm to train the deep reinforcement learning framework, so that it can generate adaptive time-varying decision parameters that are suitable for the heterogeneous cross-exercise environment;

[0011] Step 5: Input the real-time status information perceived by the unmanned cluster at the current moment into the trained deep reinforcement learning framework to obtain adaptive time-varying decision parameters, and use the adaptive time-varying decision parameters as decision parameters in the biological-like control model to perform collaborative control of the unmanned cluster.

[0012] Furthermore, step 2 specifically includes:

[0013] Define basic environment types; each basic environment corresponds to a set of environment configuration parameters, and different basic environment types correspond to different environment configuration parameters;

[0014] According to the task requirements of the unmanned cluster, at least two basic environments are selected from all types of basic environments and weighted to obtain the heterogeneous cross-rehearsal environment.

[0015] Furthermore, the environmental configuration parameters include cluster state parameters and environmental state parameters; wherein the cluster state parameters include at least one of the reachable communication distance, communication frequency, group density, travel area volume and group center moving speed; the environmental state parameters include at least one of the number of static obstacles, the number of dynamic obstacles and the number of randomly damaged drones.

[0016] Furthermore, the state information of the unmanned cluster includes the node's own state, neighbor node state and obstacle state.

[0017] Furthermore, the node's own state includes at least one of the node's own speed, acceleration and position information; the neighbor node state includes the neighbor node's speed, position and distance from the current node; the obstacle state includes the obstacle's shape, speed, position information and distance from the current node.

[0018] Furthermore, step 3 specifically includes:

[0019] The decision parameters of the biological control model are used as the action space of the unmanned swarm individuals;

[0020] The mission capability of the unmanned cluster in a heterogeneous cross-exercise environment is quantified to obtain the mission capability value of the unmanned cluster, and the mission capability value of the unmanned cluster is used as a reward value.

[0021] Furthermore, the biological control model adopts a bird flock algorithm;

[0022] Correspondingly, the decision parameter p of the bird flock algorithm is the following 8-tuple:

[0023] p={w align ,w coh ,w sep ,w target ,α,K,d0,d1}

[0024] Among them, w align 、w coh and w sep are the alignment coefficient, clustering coefficient and dispersion coefficient respectively, and w target is the target coefficient, and (α, K, d0, d1) is the obstacle avoidance coefficient in the artificial potential field obstacle avoidance method.

[0025] Furthermore, the deep reinforcement learning framework adopts the SAC deep reinforcement learning algorithm.

[0026] In a second aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0027] In a third aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect when the computer program is executed by a processor.

[0028] Beneficial effects of the present invention:

[0029] The existing rule-based biological cluster decision parameters are all fixed values ​​obtained by optimizing parameters according to known environment and task information, and are unable to adaptively adjust parameters according to perceived environmental changes. This is inconsistent with the environmental adaptability that unmanned clusters need to have. Therefore, the present invention takes into account that data-driven multi-agent decision-making represented by reinforcement learning can give a single agent stronger environmental adaptability and autonomous decision-making capabilities, and is therefore more suitable for autonomous intelligent decision-making of large-scale clusters. Therefore, the biological rules are integrated with data-driven, so that individuals in the cluster have the ability to learn different decision parameters according to different environments on the basis of having biological behavior rules, thereby realizing true biological self-organization and self-learning, and realizing autonomous decision-making of unmanned clusters in unknown environments, thereby enhancing the robustness and adaptability of clusters in uncertain scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A principle framework diagram of an unmanned cluster collaborative control method based on biological-like and data-driven fusion provided in an embodiment of the present invention;

[0031] Figure 2A schematic diagram of building a heterogeneous cross-training environment provided by an embodiment of the present invention;

[0032] Figure 3 A deep reinforcement learning framework based on a heterogeneous cross-training environment provided by an embodiment of the present invention;

[0033] Figure 4 A SAC-based deep intensity learning framework provided for an embodiment of the present invention;

[0034] Figure 5 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Example 1

[0037] like Figure 1 As shown, an embodiment of the present invention provides an unmanned cluster collaborative control method based on biological-like and data-driven fusion, including:

[0038] S101: Selecting a biological control model, and changing the value of a decision parameter in the biological control model from a traditional fixed value to an adaptive time-varying decision parameter that changes with environmental perception through subsequent steps;

[0039] Specifically, the biological control model refers to the existing biologically inspired cluster distributed control algorithm. It can be understood that the biologically inspired swarm intelligence research algorithm has provided algorithmic or conceptual support for the physical application of unmanned clusters. Among them, the biologically inspired cluster distributed control can complete intelligent decision-making only through the neighborhood information of individuals, so it is more suitable for highly dynamic scenarios, such as short communication distances and frequent changes in communication topology.

[0040] In this embodiment, the biological control model f is expressed as follows:

[0041]

[0042] Among them, x is the state variable vector of the biological population, p is the decision parameter vector of the biological control model, and t is time. The form of f is known, but the value of the parameter vector p needs to be calculated.

[0043] It should be noted that the embodiment of the present invention selects the existing biological-like control model as the basic framework for unmanned cluster collaborative control. The decision parameters in the traditional biological-like control model are fixed (which means that it can only achieve optimal performance in a specific environment and cannot achieve environmental adaptation). Therefore, the idea of ​​the embodiment of the present invention is to dynamically adjust the decision parameter p of the basic framework by using a data-driven algorithm in subsequent steps, and change it to an adaptive time-varying decision parameter p(t) that can change with environmental perception, thereby improving the robustness and adaptability of the cluster in dynamic uncertain scenarios.

[0044] S102: Build a heterogeneous cross-training environment based on the mission requirements of unmanned swarms;

[0045] It is understandable that under different task requirements, unmanned clusters generally need to work in different complex environments. The inventor believes that most unknown complex environments are the intersection of various heterogeneous basic environments. Therefore, the heterogeneous cross-exercise environment in this embodiment refers to an unmanned cluster working environment composed of at least two heterogeneous basic environments. The heterogeneity here refers to the main characteristics of the two basic environments being different.

[0046] For example, when an unmanned swarm needs to perform an enemy penetration mission, the working environment may have many obstacles and weak communication strength; when an unmanned swarm needs to perform a swarm confrontation mission, the working environment may have a high drone damage rate and weak communication strength.

[0047] In this embodiment, the following methods are mainly used to build a heterogeneous cross-exercise environment according to task requirements:

[0048] S1021: define basic environment types, such as weak communication environment, multi-obstacle environment, high damage environment, narrow channel environment, and high wind speed environment, etc.; each basic environment corresponds to a set of environment configuration parameters, and different basic environment types correspond to different environment configuration parameters;

[0049] Specifically, according to the characteristics of different basic environments, the basic environment i is defined as:

[0050]

[0051] Among them, e 1～N For configuration parameters.

[0052] The environmental configuration parameters may include cluster state parameters and environmental state parameters; wherein the cluster state parameters include at least one of the reachable communication distance, communication frequency, group density, travel area volume and group center moving speed; the environmental state parameters include at least one of the number of static obstacles, the number of dynamic obstacles and the number of randomly damaged drones.

[0053] In actual applications, parameters are configured according to the specific conditions of each basic environment. For example, in a weak communication environment, the achievable communication distance between nodes is low, and the distance between individuals in the swarm is compressed; in a multi-obstacle environment, there are a large number of static or dynamic obstacles, and in order to achieve collision avoidance, the swarm movement speed will inevitably be suppressed and the swarm topology will change rapidly; for a high-damage environment, it is mainly reflected in the large number of randomly damaged drones, resulting in a decrease in the ability to complete the task.

[0054] S1022: According to the task requirements of the unmanned cluster, at least two basic environments are selected from all types of basic environments for weighting to obtain the heterogeneous cross-exercise environment E. Figure 2 shown.

[0055] Specifically,

[0056]

[0057] in, It can be defined as configuration parameters E for multiple heterogeneous basic environments i The weighting of is the weight.

[0058] For example, in an enemy penetration scenario, the complex heterogeneous cross-exercise environment E can be set as an intersection of a multi-obstacle environment and a weak communication environment; in a cluster confrontation environment, E can be set as an intersection of a high-damage environment and a weak communication environment.

[0059] S103: Establish a deep reinforcement learning framework based on a heterogeneous cross-training environment; Figure 3 shown.

[0060] Specifically, the deep reinforcement learning framework mainly involves action space and reward function. In the embodiment of the present invention, the action space depends on the selected biological control model, and the decision parameter p of the control model f is used as the action space of the unmanned cluster individual; at the same time, the evaluation function of the task capability of the unmanned cluster in a heterogeneous cross-exercise environment is used as the reward function, that is, the task capability value is used as the reward value.

[0061] In addition, in practical applications, we can comprehensively consider the short-time window rewards and long-time window rewards under cluster tasks according to the cluster task requirements, and determine the corresponding reward mechanism according to different decision time steps.

[0062] Furthermore, according to the differences in functions and performance of each node in the cluster, the cluster task capability can generally include task completion capability, collision avoidance capability, alignment and coordination capability, etc. Thus, when quantitatively evaluating the cluster task capability, the cluster task capability quantitative evaluation value g can be defined as the weighted value of the above-mentioned various capabilities according to the task requirements.

[0063] It is understandable that in practical applications, users can choose appropriate deep reinforcement learning algorithms to implement the deep strength learning framework according to specific clustering tasks. It should be noted that the selected algorithm emphasizes the trade-off between automatic adaptation of exploration and utilization in different tasks and environments. And according to the observation space and action space of cluster individuals, design a suitable neural network structure, including the dimension of the input layer, the dimension of the output layer, the number of hidden layers, and the number of units contained in each hidden layer.

[0064] S104: Using the historical state information and action data of the unmanned cluster to train the deep reinforcement learning framework, so that it can generate an adaptive time-varying decision parameter p(t) adapted to the heterogeneous cross-exercise environment;

[0065] Specifically, a complete algorithm is established based on the task logic and reward function for learning and training. Figure 3 As shown, p(t) can be expressed as follows:

[0066] p(t)=W(S env ,t)

[0067] Among them, W(.) is the environment adaptive adjustment function, and the input of W(.) is the environmental state information S perceived by the cluster individuals. env , the output is the model decision parameter p; the W(.) function is expressed as a neural network model in the deep reinforcement learning framework.

[0068] Environmental status information S env is the quantitative value of the observation space of the cluster individuals. The observation space of the cluster individuals depends on the ability of the individuals to perceive information about the environment. In one possible implementation, define S env ={s self ,s neig ,s obst}. Among them, s self Represents the node's own state, s neig Represents the state of neighbor nodes, s obst Represents the obstacle state. It is understandable that there may be multiple neighbor nodes and obstacles.

[0069] Specifically, the status information may include the current node's own speed, acceleration and position information, neighbor node speed, position and distance from the current node, obstacle shape, speed, position information and distance from the current node, etc.

[0070] S105: Input the real-time status information perceived by the unmanned cluster at the current moment into the trained deep reinforcement learning framework to obtain adaptive time-varying decision parameters, and use the adaptive time-varying decision parameters as decision parameters in the biological control model to collaboratively control the unmanned cluster.

[0071] Specifically, based on the above-mentioned representation of the adaptive time-varying decision parameter p(t), it can be understood that by using the state information under the historical time step to train the given neural network model, after the training is completed, the trained deep reinforcement learning framework can be used to generate time-varying decision parameters according to the real-time state information. The process of using the biological control model to coordinate the control of unmanned clusters can refer to existing research and will not be repeated here.

[0072] The embodiments of the present invention take into account that data-driven multi-agent decision-making represented by reinforcement learning can give a single agent stronger environmental adaptability and autonomous decision-making capabilities, and is therefore more suitable for autonomous intelligent decision-making of large-scale clusters. Therefore, biological-like rules are integrated with data-driven, so that individuals in the cluster have the ability to learn different decision parameters according to different environments on the basis of biological-like behavioral rules, thereby realizing true biological-like self-organization and self-learning, and realizing autonomous decision-making of unmanned clusters in unknown environments.

[0073] Example 2

[0074] On the basis of the above embodiments, the embodiment of the present invention takes the classic algorithm Reynolds three principles (also known as the bird flock algorithm) as the biological control model and the SAC (Soft Actor Critic) algorithm as the deep reinforcement learning algorithm as an example to further illustrate the unmanned cluster collaborative control method of the present invention, which specifically includes the following steps:

[0075] S201: Select the Reynolds three principles based on the classic algorithm in biological cluster research as the biological control model, and through subsequent steps change the numerical value of the decision parameter in the biological control model from the traditional fixed value to an adaptive time-varying decision parameter that changes with environmental perception.

[0076] Specifically, each drone in the unmanned swarm is a node (also called an individual) in the swarm, and the decision-making rules for collaboration between individuals are quantified into a second-order control algorithm:

[0077] Speed ​​Alignment Force——

[0078] Spatial concentration

[0079] Spatial dispersion force

[0080] Among them, x i is the position vector of node i, |N i | is the neighbor node set N of node i i The size of f iis the force acting on node i, that is, the traction of the acceleration of node i; w align 、w coh and w sep are the alignment coefficient, clustering coefficient and dispersion coefficient of distributed decision parameters.

[0081] The task traction of the cluster includes the traction force in the target area and the obstacle avoidance force for known / unknown obstacles:

[0082] Target area traction——

[0083] Obstacle avoidance capability

[0084] Among them, x 目标 is the position coordinate of the target area, w target is the target coefficient, V i is the potential field function of the artificial potential field obstacle avoidance method, and (α, K, d0, d1) is the obstacle avoidance coefficient in the artificial potential field obstacle avoidance method.

[0085] The final acceleration control value of node i is acc i =f i 对齐 +f i 聚集 +f i 分散 +f i 目标 +f i 避障 , then the individual decision parameter p is an 8-tuple:

[0086] p={w align ,w coh ,w sep ,w target ,α,K,d0,d1}

[0087] S202: Construct a heterogeneous cross-exercise environment E based on task requirements.

[0088] S2021: The basic environment is defined as including weak communication environment, multi-obstacle environment and high-damage environment.

[0089] Specifically, define the weak communication environment parameter vector Multi-obstacle environment parameter vector High damage environment parameter vector The environment configuration parameters of the basic environment are shown in Table 1.

[0090] Table 1 Environmental configuration parameters

[0091] parameter describe <![CDATA[e1]]> Reachable communication distance <![CDATA[e2]]> Communication frequency <![CDATA[e3]]> Group Density <![CDATA[e4]]> Travel area volume <![CDATA[e5]]> Group center moving speed <![CDATA[e6]]> Number of static obstacles <![CDATA[e7]]> Number of dynamic obstacles <![CDATA[e8]]> Randomly destroyed drones

[0092] S2022: Targeting cluster confrontation tasks, while facing high damage and weak communication environments, build a complex heterogeneous cross-exercise environment E.

[0093] Specifically, in

[0094] S203: Establish a deep reinforcement learning framework based on a heterogeneous cross-exercise environment E;

[0095] Specifically, first, the parameter p in step S201 is align ,w coh ,w sep ,w target ,α,K,d0,d1} corresponds to the action space. Then, the cluster task capability quantitative evaluation value g is defined to include two indicators: task completion and cluster coordination:

[0096] Mission Completion

[0097] Cluster coordination

[0098] in, represents the number of collisions of node i within the T task time window, Indicates the situation of reaching the mission target area, v swarm Indicates the expected movement speed of the group.

[0099] Therefore, the cluster task capability quantitative evaluation value g: g = c1g 任务 +c2g 协同 .

[0100] Finally, a reinforcement learning framework is established with the cluster task capability value g as the reward and the adaptive decision parameter p(t) as the action.

[0101] In this embodiment, the SAC (Soft Actor Critic) deep reinforcement learning algorithm is selected as the cluster environment adaptive algorithm. The SAC algorithm network structure is designed, such as Figure 4 As shown in Figure 1, it includes five neural networks, namely, actor network π(s|θ), critic network Q(s,a|ω1), critic network Q(s,a|ω2), target critic network Q′(s,a|ω′1), and target critic network Q′(s,a|ω′2). Among them, s represents the state, corresponding to the environmental state information S observed by the cluster individuals env (will be described in detail in the subsequent steps); a represents the action, corresponding to the adaptive decision parameter p(t), that is,

[0102] The actor network outputs the mean and variance of the probability distribution of the action space based on the input state. The critic network evaluates the actions taken by the actor network in the current state. The target critic network has the same structure as the critic network, but with different weights, and is used to calculate the target Q value.

[0103] S204: Use the historical state information and action data of the unmanned cluster to train the deep reinforcement learning framework so that it outputs adaptive time-varying decision parameters that are adapted to the heterogeneous cross-exercise environment.

[0104] Specifically, a complete algorithm is established for learning and training based on the task logic and reward function, etc. The decision parameter p of the biological control model is defined to be output according to the environment adaptive adjustment function W(.).

[0105] p(t)=W(S env ,t)

[0106] Among them, W(.) is the neural network model, S env is the state information of the cluster individual observation space, and defines

[0107] Among them, m is the number of observed neighbor nodes, and n is the number of observed obstacles. The parameters are shown in Table 2.

[0108] Table 2 Individual observation space parameters

[0109]

[0110] The training process specifically includes: in step t, each agent first inputs the observed state s(t) into the actor network to obtain the mean and variance of the action probability distribution. Next, the action a(t) is sampled according to the action probability distribution and executed. Then, the environment is updated and each agent observes the next state s(t+1). Each agent receives a reward r(t). Finally, (s(t), a(t), r(t), s(t+1)) is stored in the experience replay buffer. When the number of experiences in the experience replay buffer exceeds N, N (s(i), a(i), r(i), s(i+1)) are randomly selected from the experience replay buffer to update the weights of the neural network.

[0111] In step t, a(i) and s(i) are first input into the two critic networks together to obtain the expected value of the two cumulative rewards of step i Q(s(i),a(i)|ω j)(j=1,2). Then, s(i+1) is input into the actor network and an action a(i+1) is selected. Next, a(i+1) and s(i+1) are input into the two target critic networks to obtain the estimated values ​​of the two cumulative rewards at step i+1, Q′(s(i+1), a(i+1)|ω′ j ). Then, the estimated value of the cumulative reward at step i is calculated as:

[0112]

[0113] Where γ∈[0,1] is the discount factor, which is used to determine the current value of future rewards, and α∈[0,∞] is the temperature parameter, which determines the relative importance of the entropy term and the reward, thereby controlling the randomness of the optimal strategy. In order to improve Q(s(i),a(i)|ω j ), each critic network uses a temporal difference method to update the weights. The loss function of each critic network is:

[0114]

[0115] The goal of the Actor network is to increase Q(s(i),a(i)|ω j ), so use Q(s(i),a(i)|ω j ) to update the weights of the actor network. The loss function of the actor network is:

[0116]

[0117] In order to adaptively adjust α, the minimum policy entropy threshold H0 is pre-set. α is updated according to the actual value of policy entropy, and the loss function is:

[0118]

[0119] The target critic network uses soft update to update weights:

[0120] w′ j ←τw j +(1-τ)w′ j

[0121] Where τ∈(0,1) is the update coefficient. Soft update makes the output of the target critic network change slowly and improves the stability of self-learning.

[0122] S205: Input the real-time status information perceived by the individuals in the cluster at the current moment into the generated W(.) neural network model, calculate the adaptive decision parameter p(t), and make decisions as the parameter of the biological individual decision model.

[0123] The present invention uses data-driven multi-agent decision-making represented by SAC reinforcement learning to dynamically adjust the decision parameters of the biological control model according to the cluster environment, giving the cluster stronger environmental adaptability and autonomous decision-making ability, so that the cluster has the ability to learn different decision parameters according to different environments based on biological-like behavior rules, thereby realizing true biological-like self-organization and self-learning.

[0124] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. The processor 501 can call the logic instructions in the memory 503 to execute the unmanned cluster collaborative control method, which includes: Step 1: Select a biological control model, and change the numerical value of the decision parameter in the biological control model from a traditional fixed value to an adaptive time-varying decision parameter that changes with environmental perception through subsequent steps; Step 2: Construct a heterogeneous cross-exercise environment based on the task requirements of the unmanned cluster; Step 3: Establish a deep reinforcement learning framework based on the heterogeneous cross-exercise environment; Step 4: Use the historical state information and action data of the unmanned cluster to train the deep reinforcement learning framework so that it can generate adaptive time-varying decision parameters that are adapted to the heterogeneous cross-exercise environment; Step 5: Input the real-time state information perceived by the unmanned cluster at the current moment into the trained deep reinforcement learning framework to obtain adaptive time-varying decision parameters, and use the adaptive time-varying decision parameters as decision parameters in the biological control model to collaboratively control the unmanned cluster.

[0125] In addition, when the logic instructions in the above-mentioned memory 503 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0126] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: Step 1: Select a biological control model, and change the numerical value of the decision parameter in the biological control model from a traditional fixed value to an adaptive time-varying decision parameter that changes with environmental perception through subsequent steps; Step 2: Construct a heterogeneous cross-exercise environment according to the task requirements of the unmanned cluster; Step 3: Establish a deep reinforcement learning framework based on the heterogeneous cross-exercise environment; Step 4: Use the historical state information and action data of the unmanned cluster to train the deep reinforcement learning framework so that it can generate adaptive time-varying decision parameters adapted to the heterogeneous cross-exercise environment; Step 5: Input the real-time state information perceived by the unmanned cluster at the current moment into the trained deep reinforcement learning framework to obtain adaptive time-varying decision parameters, and use the adaptive time-varying decision parameters as decision parameters in the biological control model to collaboratively control the unmanned cluster.

[0127] An embodiment of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the methods provided by the above-mentioned method embodiments are implemented, for example, including: Step 1: Select a biological control model, and change the numerical value of the decision parameter in the biological control model from a traditional fixed value to an adaptive time-varying decision parameter that changes with environmental perception through subsequent steps; Step 2: Construct a heterogeneous cross-exercise environment based on the task requirements of the unmanned cluster; Step 3: Establish a deep reinforcement learning framework based on the heterogeneous cross-exercise environment; Step 4: Use the historical state information and action data of the unmanned cluster to train the deep reinforcement learning framework so that it can generate adaptive time-varying decision parameters that are adapted to the heterogeneous cross-exercise environment; Step 5: Input the real-time state information perceived by the unmanned cluster at the current moment into the trained deep reinforcement learning framework to obtain adaptive time-varying decision parameters, and use the adaptive time-varying decision parameters as decision parameters in the biological control model to collaboratively control the unmanned cluster.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unmanned swarm collaborative control method based on biological-like and data-driven fusion, characterized in that: include: Step 1: Select a quasi-biological control model, and through subsequent steps, change the value of the decision parameter in the quasi-biological control model from a traditional fixed value to an adaptive time-varying decision parameter that changes with environmental perception; Step 2: Build a heterogeneous cross-training environment based on the mission requirements of the unmanned swarm; Step 3: Establish a deep reinforcement learning framework based on a heterogeneous cross-training environment; Step 4: Using the historical state information and action data of the unmanned swarm to train the deep reinforcement learning framework, so that it can generate adaptive time-varying decision parameters that are suitable for the heterogeneous cross-exercise environment; Step 5: Input the real-time status information perceived by the unmanned cluster at the current moment into the trained deep reinforcement learning framework to obtain adaptive time-varying decision parameters, and use the adaptive time-varying decision parameters as decision parameters in the biological-like control model to perform collaborative control of the unmanned cluster.

2. According to claim 1, the unmanned swarm collaborative control method based on biological-like and data-driven fusion is characterized in that: Step 2 specifically includes: Define basic environment types; each basic environment corresponds to a set of environment configuration parameters, and different basic environment types correspond to different environment configuration parameters; According to the task requirements of the unmanned cluster, at least two basic environments are selected from all types of basic environments and weighted to obtain the heterogeneous cross-rehearsal environment.

3. The unmanned swarm collaborative control method based on biological-like and data-driven fusion according to claim 2 is characterized in that: The environmental configuration parameters include cluster state parameters and environmental state parameters; wherein the cluster state parameters include at least one of the reachable communication distance, communication frequency, group density, travel area volume and group center moving speed; the environmental state parameters include at least one of the number of static obstacles, the number of dynamic obstacles and the number of randomly damaged drones.

4. The unmanned swarm collaborative control method based on biological-like and data-driven fusion according to claim 1 is characterized in that: The state information of the unmanned cluster includes the node's own state, neighbor node state and obstacle state.

5. The unmanned swarm collaborative control method based on biological-like and data-driven fusion according to claim 4 is characterized in that: The node's own state includes at least one of the node's own speed, acceleration and position information; the neighbor node state includes the neighbor node's speed, position and distance from the current node; the obstacle state includes the obstacle's shape, speed, position information and distance from the current node.

6. The unmanned swarm collaborative control method based on biological-like and data-driven fusion according to claim 1 is characterized in that: Step 3 specifically includes: The decision parameters of the biological control model are used as the action space of the unmanned swarm individuals; The mission capability of the unmanned cluster in a heterogeneous cross-exercise environment is quantified to obtain the mission capability value of the unmanned cluster, and the mission capability value of the unmanned cluster is used as a reward value.

7. The unmanned swarm collaborative control method based on biological-like and data-driven fusion according to claim 6 is characterized in that: The biological control model adopts a bird flock algorithm; Correspondingly, the decision parameter p of the bird flock algorithm is the following 8-tuple: p={w align ,w coh ,w sep ,w target ,α,K,d0,d1} Among them, w align 、w coh and w sep are the alignment coefficient, clustering coefficient and dispersion coefficient respectively, and w target is the target coefficient, and (α, K, d0, d1) is the obstacle avoidance coefficient in the artificial potential field obstacle avoidance method.

8. The unmanned swarm collaborative control method based on biological-like and data-driven fusion according to any one of claims 1 to 7, characterized in that: The deep reinforcement learning framework adopts the SAC deep reinforcement learning algorithm.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images