Unmanned bee colony intelligent task scheduling method based on game confrontation self-learning

Abstract

The invention relates to an unmanned bee colony intelligent task scheduling method based on game confrontation self-learning, and the method comprises the steps: setting a state space, an action space, a state transition rule and a reward function which are higher in adaptability, and can be better used for game confrontation self-learning; an execution termination strategy is not adopted for the forbidden action, but the forbidden action is disturbed and converted into an executable action, so that effective samples of the algorithm are more, and the operation efficiency of the algorithm is improved; an actor-commentator algorithm in reinforcement learning is improved to be used for unmanned bee colony task scheduling, a game confrontation self-learning unmanned bee colony task scheduling strategy is generated, and the method is different from supervised learning and does not need a large number of data samples.

Description

Technical Field

[0001] The present application relates to the field of multi-agent system task planning, and specifically, to an unmanned swarm intelligent task scheduling method based on game-themed self-learning. Background Art

[0002] Unmanned swarms are composed of a large number of patrolling aircraft. Unmanned swarm task scheduling is an important research direction in the field of aircraft control technology. Its technical background covers multiple fields such as aircraft dynamics, control theory, optimization algorithms and artificial intelligence. With the rapid development of aircraft technology, unmanned swarms are increasingly used in agricultural monitoring, environmental monitoring, disaster relief and other fields. In order to ensure that unmanned swarms can complete complex tasks efficiently and safely, task scheduling technology is particularly important. The core of unmanned swarm task scheduling lies in how to formulate reasonable flight plans and task allocation plans based on factors such as task requirements, aircraft performance and environmental conditions. This involves multiple aspects such as aircraft route planning, flight altitude and speed setting, energy distribution and coordination with other aircraft. In recent years, with the rapid development of artificial intelligence technology, especially the application of algorithms such as deep learning and reinforcement learning in aircraft control, aircraft can make real-time decisions and adjustments based on real-time environmental information, so as to complete complex tasks more intelligently. However, the intelligence of unmanned swarm task scheduling still needs to be further improved. The development and improvement of task scheduling technology will provide strong support for the widespread application of unmanned swarms. Summary of the invention

[0003] In order to overcome at least one shortcoming in the prior art, the present application provides an unmanned swarm intelligent task scheduling method based on game-themed self-learning.

[0004] In a first aspect, a method for scheduling unmanned bee colony tasks based on game confrontation self-learning is provided, comprising:

[0005] Construct the state space, action space, and state transition rules for unmanned swarm task scheduling; the state space includes task required load information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm load information; the action space includes the actions taken by the unmanned swarm at the current moment;

[0006] Set the reward function of game confrontation self-learning in the context of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm load information and the unmanned swarm endurance time information;

[0007] Based on the state space, action space, state transition rules and reward function, a game-based self-learning unmanned bee colony task scheduling model is constructed; the game-based self-learning unmanned bee colony task scheduling model includes an Actor network and a Critic network;

[0008] Train the game-adversarial self-learning unmanned bee colony task scheduling model to obtain a trained model;

[0009] The state space corresponding to the task to be scheduled is input into the trained model to obtain the action space, that is, the task scheduling strategy.

[0010] In one embodiment, the state space is represented by the state feature vector S=(D g ,D r ,D p ,D l ,D u ) indicates that D g is the load information required for the task, D r is the task time information, D p is the location information of the unmanned bee colony, D l D is the flight time information of the unmanned bee colony. u It is the payload information of the unmanned swarm;

[0011] D g =(d g1 ,d g2 ,…d gj …,d gm ), d gj Indicates the required load information of task j, that is, how much load is required;

[0012] D r =(d r1 ,d r2 ,…d rj …,d rm ), d rj Indicates the time information of task j, that is, how much time is needed to complete the current task;

[0013] D p =(d p1 ,d p2 ,…d pi …,d pn ), d pi Represents the location information of unmanned bee colony i, that is, the task number that the current unmanned bee colony is executing;

[0014] D l =(d l1 ,d l2 ,…d li …,d ln ), dli Represents the endurance information of unmanned bee colony i, that is, the remaining endurance information of the current unmanned bee colony;

[0015] D u =(d u1 ,d u2 ,…d ui …,d un ), d ui represents the load information of unmanned bee swarm i, that is, the remaining load information of the current unmanned bee swarm;

[0016] The state eigenvector has the following constraints:

[0017] d gj , d rj , d pi , d li , d ui ≥0

[0018]

[0019]

[0020] Among them, m is the number of tasks and n is the number of unmanned bee colonies.

[0021] In one embodiment, the action space is A=(a1, a2, ... a i …a n ), where a i is the action taken by unmanned bee colony i at the current moment; a i Including information about whether to stay on the current task i (α), the number of the task currently being executed is a i (β) and the number of the task to be executed next a i (γ).

[0022] In one embodiment, the state transition rule is:

[0023] Case 1: When action a i When it happens, if action a i If the prohibited action does not belong to the unmanned bee swarm i, a new state is generated according to the following formula:

[0024]

[0025]

[0026]

[0027] (d li ) t+1 =(d li) t -ΔT

[0028]

[0029] Among them, a i (α) is action a i Whether to stay on the current task, α i (β) is action α i The number of the task currently being executed in a i (γ) is action a i The number of the task to be executed next, d gai(β) For task number a i (β) The required load information of the task, d rai(β) For task number a i (β) The time information of the task; d gai(γ) For task number a i (γ) The required load information of the task, d rai(γ) For task number a i (γ) The time information of the task, d pi is the location information of unmanned bee colony i, d li represents the flight time information of unmanned bee colony i, d ui Represents the load information of unmanned bee colony i; () t Represents the data at time t, () t Represents the data at time t+1; ΔT is the sampling interval.

[0030] Case 2, when action a i When it happens, if action a i The prohibited actions of unmanned bee colony i are for action a i Perform perturbations to obtain executable actions, and then generate new states according to the formula of case 1.

[0031] In one embodiment, the reward function is:

[0032]

[0033]

[0034] Among them, (r) t is the reward function value at time t, i is the number of the unmanned bee colony, n is the number of unmanned bee colonies, (r i ) t is the reward function of unmanned bee colony i at time t, K is the reward discount factor, d li represents the flight time information of unmanned bee colony i, d ui Represents the load information of unmanned bee swarm i.

[0035] In one embodiment, training the game confrontation self-learning unmanned swarm task scheduling model to obtain the trained model includes:

[0036] Step S41, initialize the state space S and the experience pool Ψ, and the initial time t = 0;

[0037] Step S42, let t = t + 1, and input the state space (S) at the current time t into the Actor network to obtain λ action samples A = (a1, a2,... a i …a n ), where a i is the action taken by the unmanned swarm i at the current time;

[0038] Step S43, input the λ action samples into the Critic network to obtain the reward function value corresponding to each action sample, and determine the maximum value r * of the reward function value and the corresponding action sample A * ;

[0039] Step S44, if there is an action a * in A i that belongs to the prohibited action, perturb the action a i to obtain an executable action A σ ; based on the state space (S) at the current time t execute the action A σ , and generate the state space (S) at the next time t+1 ;

[0040] If there is no action belonging to the prohibited action in A * , based on the state space (S) at the current time t execute the action A * , and generate the state space (S) at the next time t+1 ;

[0041] Step S45, construct a five-tuple {(S) t , A * , r * , (S) t+1 , t} and put it into the experience pool Ψ;

[0042] Step S46, if t < T, where T is the length of the Episode, return to Step S42; otherwise, execute Step S47;

[0043] Step S47, use the data in the experience pool Ψ to perform gradient backpropagation to update the parameters ω of the Actor network aand Critic network parameter ω c ; If the Actor network parameter ω a and Critic network parameter ω c The changes are less than 10 -4 , the training is terminated and the trained model is obtained; otherwise, return to step S41.

[0044] In a second aspect, an unmanned bee colony task scheduling device based on game confrontation self-learning is provided, comprising:

[0045] The first construction module is used to construct the state space, action space, and state transition rules for unmanned swarm task scheduling; the state space includes task required load information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm load information; the action space includes the actions taken by the unmanned swarm at the current moment;

[0046] The reward function setting module is used to set the reward function of the game confrontation self-learning in the context of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm load information and the unmanned swarm endurance time information;

[0047] The second building block is used to build a game-based self-learning unmanned bee colony task scheduling model based on the state space, action space, state transition rules and reward function; the game-based self-learning unmanned bee colony task scheduling model includes an Actor network and a Critic network;

[0048] A model training module is used to train the game-adversarial self-learning unmanned bee colony task scheduling model to obtain a trained model;

[0049] The task scheduling strategy determination module is used to input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, that is, the task scheduling strategy.

[0050] In a third aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the unmanned swarm task scheduling method based on game confrontation self-learning is implemented.

[0051] Compared with the prior art, this application has the following beneficial effects:

[0052] 1. Based on the unmanned swarm task scheduling background, a more adaptable state space, action space, state transition rules and reward function are set up, which can be better used for game adversarial self-learning.

[0053] 2. For prohibited actions, the strategy of terminating execution is not adopted. Instead, it is disturbed and converted into executable actions, which makes the algorithm have more effective samples and improves the efficiency of algorithm operation.

[0054] 3. The actor-critic algorithm in reinforcement learning is improved for unmanned swarm task scheduling, generating a game-adversarial self-learning unmanned swarm task scheduling strategy. This method is different from supervised learning and does not require a large amount of data samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present application may be better understood by referring to the following description given in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification. In the drawings:

[0056] Figure 1 A flowchart of an unmanned bee colony task scheduling method based on game confrontation self-learning according to an embodiment of the present application is shown;

[0057] Figure 2 A structural block diagram of an unmanned bee colony task scheduling device based on game confrontation self-learning according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] The exemplary embodiments of the present application will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.

[0059] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, while other details that are not closely related to the present application are omitted.

[0060] It should be understood that the present application is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.

[0061] The present application provides an unmanned bee swarm task scheduling method based on game-theoretic adversarial self-learning, which aims to establish a mapping relationship between the task state space and the unmanned bee swarm action space, and to generate an unmanned bee swarm task scheduling strategy with low time and cost consumption based on the actor-critic reinforcement learning algorithm, thereby realizing intelligent scheduling of unmanned bee swarms in multi-task scenarios.

[0062] Figure 1 A flowchart of an unmanned bee colony task scheduling method based on game confrontation self-learning according to an embodiment of the present application is shown, see Figure 1 , methods include:

[0063] Step S1, construct the state space, action space, and state transition rules for unmanned swarm task scheduling; the state space includes task required load information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm load information; the action space includes the actions taken by the unmanned swarm at the current moment.

[0064] Unmanned bee swarm task scheduling involves m tasks and n unmanned bee swarms. Markov Decision Process (MDP), as a key model for sequential decision making, is the mathematical foundation and theoretical core of reinforcement learning. Under the MDP framework, task scheduling problems can be quantitatively analyzed.

[0065] Specifically, the state space is represented by the state eigenvector S = (D g ,D r ,D p ,D l ,D u ) indicates that D g is the load information required for the task, D r is the task time information, D p is the location information of the unmanned bee colony, D l D is the flight time information of the unmanned bee colony. u It is the payload information of the unmanned swarm;

[0066] D g =(d g1 ,d g2 ,…d gj …,d gm ), d gj Indicates the required load information of task j, that is, how much load is required;

[0067] D r =(d r1 ,d r2 ,…d rj …,d rm ), d rj Indicates the time information of task j, that is, how much time is needed to complete the current task;

[0068] D p =(d p1 ,d p2 ,…d pi …,d pn ), d piRepresents the location information of unmanned bee colony i, that is, the task number that the current unmanned bee colony is executing;

[0069] D l =(d l1 ,d l2 ,…d li …,d ln ), d li Represents the endurance information of unmanned bee colony i, that is, the remaining endurance information of the current unmanned bee colony;

[0070] D u =(d u1 ,d u2 ,…d ui …,d un ), d ui represents the load information of unmanned bee swarm i, that is, the remaining load information of the current unmanned bee swarm;

[0071] The state eigenvector has the following constraints:

[0072] d gj , d rj , d pi , d li , d ui ≥0

[0073]

[0074]

[0075] Among them, m is the number of tasks and n is the number of unmanned bee colonies.

[0076] The first constraint describes that the state space is in non-negative real space; the second constraint indicates that the endurance of all current unmanned swarms must ensure that all tasks can be completed; the third constraint indicates that the remaining load of all current unmanned swarms must ensure that the required load of all tasks can be met.

[0077] Specifically, the task scheduling strategy is ultimately presented by an action sequence. According to the task scheduling, the action space is A = (a1, a2, ... a i …a n ), where a i is the action taken by unmanned bee colony i at the current moment; a i Including information about whether to stay on the current task i (α), the number of the task currently being executed is a i (β) and the number of the task to be executed next a i (γ).a i (α)=0 means continue to stay on the current task, ai (α) = 1 means leaving the task; in practice, some tasks can only be completed by a designated unmanned bee colony, in which case the forbidden action a f to complete the setting constraints.

[0078] Specifically, each time an action is executed at each sampling moment, a state transition occurs. The state transition of MDP refers to the current state (S) t Next, action (A) occurs t , generating a new state (S) t+1 When action a i When this happens, there are two possible situations depending on the legality of the action:

[0079] The state transition rules are:

[0080] Case 1: When action a i When it happens, if action a i If the prohibited action does not belong to the unmanned bee swarm i, a new state is generated according to the following formula:

[0081]

[0082]

[0083]

[0084] (d li ) t+1 =(d li ) t -ΔT

[0085]

[0086] Among them, a i (α) is action a i Whether to stay on the current task, a i (β) is action a i The number of the task currently being executed in a i (γ) is action a i The number of the task to be performed next is expected. For task number a i (β) The required load information of the task, For task number a i (β) Task time information; For task number a i (γ) The required load information of the task, For task number a i (γ) The time information of the task, d piis the location information of unmanned bee colony i, d li represents the flight time information of unmanned bee colony i, d ui Represents the load information of unmanned bee colony i; () t Represents the data at time t, () t Represents the data at time t+1; ΔT is the sampling interval.

[0087] Case 2, when action a i When it happens, if action a i The prohibited actions of unmanned bee colony i are for action a i Perform perturbations to obtain executable actions, and then generate new states according to the formula of case 1.

[0088] Step S2, setting the reward function of game confrontation self-learning in the context of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm load information and the unmanned swarm endurance time information.

[0089] The reward function is the key design of the game-adversarial self-learning method. A reasonable reward function will guide the direction of game-adversarial self-learning and is directly related to the success of strategy search. The essence of the reward function is to set up a feedback mechanism for the algorithm to evaluate the strategy of the algorithm search. Every time an action is performed, the environment will feedback a reward. The main goal of game-adversarial self-learning is to maximize long-term benefits and obtain the optimal task scheduling strategy by maximizing the cumulative discounted rewards. For the unmanned swarm task scheduling scenario, the following real-time reward function is designed:

[0090]

[0091]

[0092] Among them, (r) t is the reward function value at time t, i is the number of the unmanned bee colony, n is the number of unmanned bee colonies, (r i ) t is the reward function of unmanned bee colony i at time t, K is the reward discount factor, 0 <K<1,d li represents the flight time information of unmanned bee colony i, d ui Represents the load information of unmanned bee swarm i.

[0093] According to the calculation formula of the reward function, as time goes by, the future cumulative reward weight becomes lower and lower; the remaining load d of the unmanned bee colony i ui The smaller the reward value, the greater the task execution efficiency; the endurance information d of the unmanned bee swarm i liThe larger the value, the greater the reward value, which means that the shorter the time consumption of task execution, the better. Once the execution action belongs to the prohibited action, a reward value of -500 is directly assigned, which is equivalent to a soft constraint on the prohibited action.

[0094] Step S3, based on the state space, action space, state transition rules and reward function, construct a game adversarial self-learning unmanned bee swarm task scheduling model; the game adversarial self-learning unmanned bee swarm task scheduling model includes an Actor network and a Critic network.

[0095] Step S4, training the game adversarial self-learning unmanned bee colony task scheduling model to obtain a trained model.

[0096] Step S5, input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, that is, the task scheduling strategy. Here, the state space is constructed according to the task to be scheduled and the unmanned bee colony.

[0097] This embodiment is based on the background of unmanned bee colony task scheduling, and sets a more adaptable state space, action space, state transition and reward function, which can be better used for game confrontation self-learning. For prohibited actions, the strategy of terminating execution is not adopted, but it is disturbed and converted into executable actions, so that the algorithm has more effective samples and improves the algorithm operation efficiency. The actor-critic algorithm in reinforcement learning is improved for unmanned bee colony task scheduling, and the generation of game confrontation self-learning unmanned bee colony task scheduling strategy is realized. Unlike supervised learning, it does not require a large number of data samples.

[0098] In one embodiment, in step S4, the game-adversarial self-learning unmanned bee colony task scheduling model is trained to obtain a trained model, including:

[0099] Step S41, initializing the state space S and the experience pool Ψ, with the initial time t=0;

[0100] Step S42, let t = t + 1, and convert the current state space (S) t Input into the Actor network to get λ action samples A=(a1,a2,…a i …a n ), where a i is the action taken by unmanned bee swarm i at the current moment;

[0101] Step S43: Input λ action samples into the Critic network, obtain the reward function value corresponding to each action sample, and determine the maximum value r of the reward function value. * Corresponding action sample A * ;

[0102] Step S44, if A* There is an action a i belongs to the prohibited action, then for the action a i perturb it to obtain an executable action A σ ; Based on the state space (S) at the current moment t Execute the action A σ , generate the state space (S) at the next moment t+1 ; Here, the state space at the next moment is generated according to the formula in Case 1 of the state transition rule.

[0103] If there is no action belonging to the prohibited action in A * , based on the state space (S) at the current moment t Execute the action A * , generate the state space (S) at the next moment t+1 ;

[0104] Step S45, construct a five-tuple {(S) t , A * , r * , (S) t+1 , t} and put it into the experience pool Ψ;

[0105] Step S46, if t < T, where T is the length of the Episode, then return to Step S42; otherwise, execute Step S47;

[0106] Step S47, use the data in the experience pool Ψ to perform gradient backpropagation to update the network parameters ω of the Actor network a and the network parameters ω of the Critic network c ; If the changes in the network parameters ω of the Actor network a and the network parameters ω of the Critic network c are both less than 10 -4 , then the training terminates and the trained model is obtained; otherwise, return to Step S41.

[0107] In this embodiment, first set the length T of the Episode, the network parameters ω of the Actor network a , the network parameters ω of the Critic network c , the state space S = (D g , D r , D p , D l , D u ) composed of n unmanned swarms and m task requirements, the action space A = (a1, a2,... a i ... a n ), the reward discount factor K, the sampling interval ΔT, and the batch sample quantity λ.

[0108] Using the same inventive concept as the unmanned bee colony task scheduling method based on game confrontation self-learning, this embodiment also provides a corresponding unmanned bee colony task scheduling device based on game confrontation self-learning. Figure 2 The structural block diagram of an unmanned bee colony task scheduling device based on game confrontation self-learning according to an embodiment of the present application is shown, and the device includes:

[0109] The first construction module 21 is used to construct the state space, action space, and state transition rules of the unmanned swarm task scheduling; the state space includes the task load information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm load information; the action space includes the actions taken by the unmanned swarm at the current moment;

[0110] The reward function setting module 22 is used to set the reward function of the game confrontation self-learning in the context of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm load information and the unmanned swarm endurance time information;

[0111] The second construction module 23 is used to construct a game-based self-learning unmanned bee colony task scheduling model based on the state space, action space, state transition rules and reward function; the game-based self-learning unmanned bee colony task scheduling model includes an Actor network and a Critic network;

[0112] A model training module 24 is used to train the game confrontation self-learning unmanned bee colony task scheduling model to obtain a trained model;

[0113] The task scheduling strategy determination module 25 is used to input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, that is, the task scheduling strategy.

[0114] The unmanned bee swarm task scheduling device based on game adversarial self-learning of this embodiment has the same inventive concept as the unmanned bee swarm task scheduling method based on game adversarial self-learning mentioned above. Therefore, the specific implementation method of the device can be seen in the implementation example part of the unmanned bee swarm task scheduling method based on game adversarial self-learning in the previous text, and its technical effect corresponds to the technical effect of the above method, which will not be repeated here.

[0115] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned unmanned bee colony task scheduling method based on game confrontation self-learning.

[0116] In summary, this application has the following technical effects:

[0117] Through the unmanned bee swarm task scheduling method based on game confrontation self-learning, multiple unmanned bee swarms can generate scheduling strategies according to existing task scenarios and complete tasks at a lower cost. Unmanned bee swarm task scheduling is a widely used and significant technology. It plays an important role in many fields such as power inspection, traffic monitoring, environmental monitoring, agricultural applications, and fire reconnaissance. Through automated and intelligent task scheduling, unmanned bee swarms can greatly improve inspection efficiency, reduce manpower and material input and operating costs, and enter high-risk or difficult-to-reach areas for safe and accurate inspections. The research and development and application of this technology not only promotes the progress and innovation of related technologies, but also provides strong support for improving inspection efficiency, safety and accuracy, which is of great significance to the sustainable development of modern society.

[0118] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for unmanned bee colony task scheduling based on game confrontation self-learning, characterized in that: Including: Construct the state space, action space, and state transition rules for the task scheduling of unmanned swarm; the state space includes task required payload information, task time information, unmanned swarm position information, unmanned swarm endurance time information, and unmanned swarm payload information; the action space includes the actions taken by the unmanned swarm at the current moment; Set the reward function for game adversarial self-learning in the context of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm payload information, and the unmanned swarm endurance time information; Based on the state space, the action space, the state transition rules, and the reward function, construct a game adversarial self-learning unmanned swarm task scheduling model; the game adversarial self-learning unmanned swarm task scheduling model includes an Actor network and a Critic network; Train the game adversarial self-learning unmanned swarm task scheduling model to obtain a trained model; Input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, that is, the task scheduling strategy.

2. The method according to claim 1, characterized in that The state space is represented by the state eigenvector S = (D g ,D r ,D p ,D l ,D u ) indicates that D g is the load information required for the task, D r is the task time information, D p is the location information of the unmanned bee colony, D l D is the flight time information of the unmanned bee colony. u It is the payload information of the unmanned swarm; D g =(d g1 ,d g2 ,…d gj …,d gm ), d gj Indicates the required load information of task j, that is, how much load is required; D r =(d r1 ,d r2 ,…d rj …,d rm ), d rj Indicates the time information of task j, that is, how much time is needed to complete the current task; D p =(d p1 ,d p2 ,…d pi …,d pn ), d pi Represents the location information of unmanned bee colony i, that is, the task number that the current unmanned bee colony is executing; D l =(d l1 ,d l2 ,…d li …,d ln ), d li Represents the endurance information of unmanned bee colony i, that is, the remaining endurance information of the current unmanned bee colony; D u =(d u1 ,d u2 ,…d ui …,d un ), d ui represents the load information of unmanned bee swarm i, that is, the remaining load information of the current unmanned bee swarm; The state feature vector has the following constraints: d gj ,d rj ,d pi ,d li ,d ui ≥0 Where m is the number of tasks and n is the number of unmanned swarms.

3. The method according to claim 1, characterized in that The action space is A = (a1, a2, ... a i …a n ), where a i is the action taken by unmanned bee colony i at the current moment; a i Including information about whether to stay on the current task i (α), the number of the task currently being executed is a i (β) and the task number a that is expected to be executed next i (γ).

4. The method according to claim 3, characterized in that The state transition rule is: Case 1: When action a i When it happens, if action a i If the prohibited action does not belong to the unmanned bee swarm i, a new state is generated according to the following formula: (d li ) t+1 =(d li ) t -ΔT Among them, a i (α) is action a i Whether to stay on the current task, a i (β) is action a i The number of the task currently being executed in a i (γ) is action a i The number of the task to be performed next is expected. For task number a i (β) The required load information of the task, For task number a i (β) Task time information; For task number a i (γ) The required load information of the task, For task number a i (γ) The time information of the task, d pi is the location information of unmanned bee colony i, d li represents the flight time information of unmanned bee colony i, d ui Represents the load information of unmanned bee colony i; () t Represents the data at time t, () t Represents the data at time t+1; ΔT is the sampling interval. Case 2, when action a i When it happens, if action a i The prohibited actions of unmanned bee colony i are for action a i Perform perturbations to obtain executable actions, and then generate new states according to the formula of case 1.

5. The method according to claim 1, characterized in that The reward function is: Among them, (r) t is the reward function value at time t, i is the number of the unmanned bee colony, n is the number of unmanned bee colonies, (r i ) t is the reward function of unmanned bee colony i at time t, K is the reward discount factor, d li represents the flight time information of unmanned bee colony i, d ui Represents the load information of unmanned bee swarm i.

6. The method according to claim 1, characterized in that Where Training the game adversarial self-learning unmanned swarm task scheduling model to obtain a trained model includes: Step S41, initialize the state space S and the experience pool Ψ, and the initial time t = 0; Step S42, let t = t + 1, and convert the current state space (S) t Input into the Actor network to get λ action samples A=(a1,a2,…a i …a n ), where a i is the action taken by unmanned bee swarm i at the current moment; Step S43: Input λ action samples into the Critic network, obtain the reward function value corresponding to each action sample, and determine the maximum value r of the reward function value. * Corresponding action sample A * ; Step S44, if A * There is action a in i It is a prohibited action, then for action a i Perform perturbation to obtain executable action A σ ; Based on the current state space (S) t Execute action A σ , generate the state space (S) at the next moment t+1 ; If A * There is no action that is prohibited, based on the current state space (S) t Execute action A * , generate the state space (S) at the next moment t+1 ; Step S45, construct a five-tuple {(S) t , A * , r * , (S) t+1 , t} into the experience pool Ψ; Step S46, if t < T, where T is the length of the Episode, then return to step S42; otherwise, execute step S47; Step S47, use the data in the experience pool Ψ to perform gradient back propagation on the Actor network and the Critic network to update the Actor network parameters ω a and Critic network parameter ω c ; If the Actor network parameter ω a and Critic network parameter ω c The changes were less than 10 -4 , the training is terminated and the trained model is obtained; otherwise, return to step S41.

7. An unmanned bee colony task scheduling device based on game confrontation self-learning, characterized in that: Including: The first construction module is used to construct the state space, action space, and state transition rules for the task scheduling of unmanned swarm; the state space includes task required payload information, task time information, unmanned swarm position information, unmanned swarm endurance time information, and unmanned swarm payload information; the action space includes the actions taken by the unmanned swarm at the current moment; The reward function setting module is used to set the reward function for game adversarial self-learning in the context of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm payload information, and the unmanned swarm endurance time information; The second construction module is used to construct a game adversarial self-learning unmanned swarm task scheduling model based on the state space, the action space, the state transition rules, and the reward function; the game adversarial self-learning unmanned swarm task scheduling model includes an Actor network and a Critic network; The model training module is used to train the game adversarial self-learning unmanned swarm task scheduling model to obtain a trained model; The task scheduling strategy determination module is used to input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, that is, the task scheduling strategy.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method for unmanned swarm task scheduling based on game adversarial self-learning according to any one of claims 1-6.

Images