A Game Theory-Based Adversarial Self-Learning-Based Intelligent Task Scheduling Method for Unmanned Bee Swarms

Abstract

This application relates to an intelligent task scheduling method for unmanned swarms based on game-theoretic adversarial self-learning. It establishes a more adaptive state space, action space, state transition rules, and reward function, which are better suited for game-theoretic adversarial self-learning. Instead of terminating prohibited actions, it perturbs them, transforming them into executable actions, thus increasing the number of effective samples and improving algorithm efficiency. Furthermore, it improves the actor-critic algorithm from reinforcement learning for unmanned swarm task scheduling, generating a game-theoretic adversarial self-learning unmanned swarm task scheduling strategy. Unlike supervised learning, this method does not require a large amount of data samples.

Description

Technical Field

[0001] This application relates to the field of task planning for multi-agent systems, and more specifically, to a method for intelligent task scheduling of unmanned bee swarms based on game-theoretic adversarial self-learning. Background Technology

[0002] Unmanned aerial vehicle (UAV) swarms consist of numerous patrol aircraft. Task scheduling for UAV swarms is a crucial research area in aircraft control technology, encompassing multiple fields such as aircraft dynamics, control theory, optimization algorithms, and artificial intelligence. With the rapid development of aircraft technology, UAV swarms are increasingly widely used in agricultural monitoring, environmental monitoring, and disaster relief. To ensure that UAV swarms can efficiently and safely complete complex tasks, task scheduling technology is particularly important. The core of UAV swarm task scheduling lies in how to formulate reasonable flight plans and task allocation schemes based on factors such as task requirements, aircraft performance, and environmental conditions. This involves multiple aspects, including aircraft route planning, altitude and speed settings, energy allocation, and coordination with other aircraft. In recent years, with the rapid development of artificial intelligence technology, especially the application of algorithms such as deep learning and reinforcement learning in aircraft control, aircraft can make real-time decisions and adjustments based on real-time environmental information, thus completing complex tasks more intelligently. However, the intelligence of current UAV swarm task scheduling still needs further improvement. The development and refinement of task scheduling technology will provide strong support for the widespread application of UAV swarms. Summary of the Invention

[0003] To overcome at least one deficiency in the prior art, this application provides an unmanned bee swarm intelligent task scheduling method based on game-theoretic adversarial self-learning.

[0004] Firstly, a game-theoretic adversarial self-learning-based unmanned bee swarm task scheduling method is provided, including:

[0005] Construct the state space, action space, and state transition rules for unmanned swarm task scheduling; the state space includes task payload information, task time information, unmanned swarm location information, unmanned swarm endurance information, and unmanned swarm payload information; the action space includes the actions taken by the unmanned swarm at the current moment;

[0006] Set a reward function for game-based adversarial self-learning under the background of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm's payload information, and the unmanned swarm's endurance information.

[0007] Based on state space, action space, state transition rules, and reward function, a game-theoretic adversarial self-learning unmanned swarm task scheduling model is constructed. The game-theoretic adversarial self-learning unmanned swarm task scheduling model includes an Actor network and a Critic network.

[0008] The game-adversarial self-learning unmanned bee swarm task scheduling model was trained to obtain the trained model.

[0009] The state space corresponding to the task to be scheduled is input into the trained model to obtain the action space, i.e., the task scheduling strategy.

[0010] In one embodiment, the state space is represented by the state feature vector S = (D g D r D p D l D u ) indicates that D g For the payload information required for the mission, D r For task time information, D p For the location information of the unmanned bee swarm, D l For information on the endurance of unmanned swarms, D u For unmanned bee swarm payload information;

[0011] D g =(d g1 ,d g2 ,…d gj …,d gm ), d gj This indicates the required payload information for task j, i.e., how much payload needs to be consumed;

[0012] D r =(d r1 ,d r2 ,…d rj …,d rm ), d rj This indicates the time information for task j, specifically how much more time is needed for the current task to complete.

[0013] D p =(d p1 ,d p2 ,…d pi …,d pn ), d pi This indicates the location information of the unmanned bee colony i, that is, the task number that the unmanned bee colony is currently executing;

[0014] D l =(d l1 ,d l2 ,…d li …,d ln ), dli This indicates the remaining flight time of the unmanned bee colony i, i.e., the remaining flight time of the unmanned bee colony.

[0015] D u =(d u1 ,d u2 ,…d ui …,d un ), d ui This represents the payload information of unmanned bee colony i, that is, the remaining payload information of the current unmanned bee colony;

[0016] The state feature vector is subject to the following constraints:

[0017] d gj d rj d pi d li d ui ≥0

[0018]

[0019]

[0020] Where m is the number of tasks and n is the number of unmanned bee swarms.

[0021] In one embodiment, the action space is A = (a1, a2, ... a2). i …a n ), where a i The action taken by the unmanned bee swarm i at the current moment; a i Including information on whether the task remains in progress. i (α), the currently executing task number a i (β) and the task number a that is expected to be executed next. i (γ).

[0022] In one embodiment, the state transition rule is as follows:

[0023] Scenario 1, when action a i When it occurs, if action a i If an action is not prohibited in unmanned bee colony i, a new state is generated according to the following formula:

[0024]

[0025]

[0026]

[0027] (d li ) t+1 =(d li) t -ΔT

[0028]

[0029] Among them, a i (α) represents action a i Whether the information is still on the current task, α i (β) represents action α i The currently executing task number, a i (γ) represents action a i The next task to be executed is expected to be numbered d. gai(β) For task number a i (β) The required payload information for the task, d rai(β) For task number a i (β) Task time information; d gai(γ) For task number a i (γ) The required payload information for the task, d rai(γ) For task number a i (γ) time information of the task, d pi For the location information of the unmanned bee colony i, d li This represents the battery life information of drone swarm i, d ui This represents the payload information of the unmanned bee colony i; () t Represents the data at time t, () t This represents the data at time t+1; ΔT is the sampling interval.

[0030] Case 2, when action a i When it occurs, if action a i Prohibited actions belonging to unmanned bee colony i, for action a i The perturbation is performed to obtain an executable action, and then a new state is generated according to the formula in case 1.

[0031] In one embodiment, the reward function is:

[0032]

[0033]

[0034] Among them, (r) t Let be the reward function value at time t, i be the label of the unmanned swarm, n be the number of unmanned swarms, and (r i ) t Let d be the reward function of the unmanned bee colony i at time t, K be the reward discount factor, and d be the reward function of the unmanned bee colony i at time t. li This represents the battery life information of drone swarm i, d ui This represents the payload information of unmanned bee colony i.

[0035] In one embodiment, the game confrontation self-learning unmanned swarm mission scheduling model is trained to obtain a trained model, including:

[0036] Step S41, initialize the state space S and the experience pool Ψ, and the initial time t = 0;

[0037] Step S42, let t = t + 1, and input the state space (S) at the current time t into the Actor network to obtain λ action samples A = (a1, a2,... a i …a n ), where a i is the action taken by the unmanned swarm i at the current time;

[0038] Step S43, input the λ action samples into the Critic network to obtain the reward function value corresponding to each action sample, and determine the maximum value r * of the reward function value and the corresponding action sample A * ;

[0039] Step S44, if there is an action a * in A i that belongs to the prohibited action, perturb the action a i to obtain an executable action A σ ; based on the state space (S) at the current time t execute the action A σ , and generate the state space (S) at the next time t+1 ;

[0040] If there is no action belonging to the prohibited action in A * , based on the state space (S) at the current time t execute the action A * , and generate the state space (S) at the next time t+1 ;

[0041] Step S45, construct a five-tuple {(S) t , A * , r * , (S) t+1 , t} and put it into the experience pool Ψ;

[0042] Step S46, if t < T, where T is the length of the Episode, then return to Step S42; otherwise, execute Step S47;

[0043] Step S47, use the data in the experience pool Ψ to perform gradient backpropagation to update the parameters ω of the Actor network aand Critic network parameters ω c If the Actor network parameter ω a and Critic network parameters ω c The changes were all less than 10 -4 If the training is successful, the training will terminate and the trained model will be obtained; otherwise, return to step S41.

[0044] Secondly, a game-theoretic adversarial self-learning-based unmanned swarm task scheduling device is provided, comprising:

[0045] The first construction module is used to construct the state space, action space, and state transition rules for unmanned swarm task scheduling. The state space includes task payload information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm payload information. The action space includes the actions taken by the unmanned swarm at the current moment.

[0046] The reward function setting module is used to set the reward function for game-based adversarial self-learning under the background of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm payload information, and the unmanned swarm endurance information.

[0047] The second building module is used to construct a game-theoretic adversarial self-learning unmanned swarm task scheduling model based on the state space, action space, state transition rules, and reward function; the game-theoretic adversarial self-learning unmanned swarm task scheduling model includes an Actor network and a Critic network.

[0048] The model training module is used to train the game adversarial self-learning unmanned swarm task scheduling model to obtain the trained model.

[0049] The task scheduling strategy determination module is used to input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, i.e., the task scheduling strategy.

[0050] Thirdly, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned unmanned swarm task scheduling method based on game-theoretic adversarial self-learning.

[0051] Compared with the prior art, this application has the following beneficial effects:

[0052] 1. Based on the background of unmanned bee swarm task scheduling, a more adaptive state space, action space, state transition rules and reward function are set up, which can be better used for game adversarial self-learning.

[0053] 2. Instead of terminating the execution of prohibited actions, the algorithm perturbs them to transform them into executable actions, resulting in more effective samples and improving the algorithm's running efficiency.

[0054] 3. The actor-critic algorithm in reinforcement learning is improved for unmanned swarm task scheduling to generate game-theoretic adversarial self-learning unmanned swarm task scheduling strategies. This method is different from supervised learning and does not require a large number of data samples. Attached Figure Description

[0055] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings:

[0056] Figure 1 A flowchart of an unmanned swarm task scheduling method based on game-theoretic adversarial self-learning according to an embodiment of this application is shown;

[0057] Figure 2 A structural block diagram of an unmanned swarm task scheduling device based on game-theoretic adversarial self-learning according to an embodiment of this application is shown. Detailed Implementation

[0058] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0059] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution according to this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0060] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0061] This application provides a game-theoretic adversarial self-learning-based unmanned swarm task scheduling method, which aims to establish a mapping relationship between the task state space and the unmanned swarm action space. Based on the actor-critic reinforcement learning algorithm, it generates unmanned swarm task scheduling strategies with low time and cost consumption, thereby realizing intelligent scheduling of unmanned swarms in multi-task scenarios.

[0062] Figure 1 A flowchart of an unmanned bee swarm task scheduling method based on game-theoretic adversarial self-learning according to an embodiment of this application is shown. See also: Figure 1 The methods include:

[0063] Step S1: Construct the state space, action space, and state transition rules for unmanned swarm task scheduling; the state space includes task payload information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm payload information; the action space includes the actions taken by the unmanned swarm at the current moment.

[0064] Unmanned swarm task scheduling involves m tasks and n unmanned swarms. Markov Decision Processes (MDPs), as a key model for sequential decision-making, are the mathematical foundation and theoretical core of reinforcement learning. Within the MDP framework, the task scheduling problem can be quantitatively analyzed.

[0065] Specifically, the state space is represented by the state eigenvector S = (D g D r D p D l D u ) indicates that D g For the payload information required for the mission, D r For task time information, D p For the location information of the unmanned bee swarm, D l For information on the endurance of unmanned swarms, D u For unmanned bee swarm payload information;

[0066] D g =(d g1 ,d g2 ,…d gj …,d gm ), d gj This indicates the required payload information for task j, i.e., how much payload needs to be consumed;

[0067] D r =(d r1 ,d r2 ,…d rj …,d rm ), d rj This indicates the time information for task j, specifically how much more time is needed for the current task to complete.

[0068] D p =(d p1 ,d p2 ,…d pi …,d pn ), d piThis indicates the location information of the unmanned bee colony i, that is, the task number that the unmanned bee colony is currently executing;

[0069] D l =(d l1 ,d l2 ,…d li …,d ln ), d li This indicates the remaining flight time of the unmanned bee colony i, i.e., the remaining flight time of the unmanned bee colony.

[0070] D u =(d u1 ,d u2 ,…d ui …,d un ), d ui This represents the payload information of unmanned bee colony i, that is, the remaining payload information of the current unmanned bee colony;

[0071] The state feature vector is subject to the following constraints:

[0072] d gj d rj d pi d li d ui ≥0

[0073]

[0074]

[0075] Where m is the number of tasks and n is the number of unmanned bee swarms.

[0076] The first constraint describes that the state space is in a non-negative real space; the second constraint states that the current endurance of all unmanned swarms must ensure that all tasks can be completed; the third constraint states that the current remaining payload of all unmanned swarms must ensure that the payload requirements of all tasks can be met.

[0077] Specifically, the task scheduling strategy is ultimately represented by a sequence of actions. According to the task scheduling, the action space is A = (a1, a2, ... a...). i …a n ), where a i The action taken by the unmanned bee swarm i at the current moment; a i Including information on whether the task remains in progress. i (α), the currently executing task number a i (β) and the task number a that is expected to be executed next. i (γ). a i (α) = 0 indicates that the current task will continue to be performed.i (α) = 1 indicates leaving the task; in practice, some tasks can only be completed by a designated unmanned swarm, in which case the prohibition action 'a' can be used. f To complete the setting of this constraint.

[0078] Specifically, a state transition occurs every time an action is performed at each sampling time. The state transition in MDP refers to the current state (S). t Below, action (A) occurs. t This generates a new state (S). t+1 The process. When action a i When this occurs, depending on the legality of the action, there may be two scenarios:

[0079] The state transition rules are as follows:

[0080] Scenario 1, when action a i When it occurs, if action a i If an action is not prohibited in unmanned bee colony i, a new state is generated according to the following formula:

[0081]

[0082]

[0083]

[0084] (d li ) t+1 =(d li ) t -ΔT

[0085]

[0086] Among them, a i (α) represents action a i Whether the information is still on the current task, a i (β) represents action a i The currently executing task number, a i (γ) represents action a i The next task number that China expects to execute is... For task number a i (β) The required payload information for the task. For task number a i (β) The time information of the task; For task number a i The payload information required for the task of (γ), For task number a i (γ) time information of the task, d piFor the location information of the unmanned bee colony i, d li This represents the battery life information of drone swarm i, d ui This represents the payload information of the unmanned bee colony i; () t Represents the data at time t, () t This represents the data at time t+1; ΔT is the sampling interval.

[0087] Case 2, when action a i When it occurs, if action a i Prohibited actions belonging to unmanned bee colony i, for action a i The perturbation is performed to obtain an executable action, and then a new state is generated according to the formula in case 1.

[0088] Step S2: Set the reward function for game-based adversarial self-learning under the background of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm payload information, and the unmanned swarm endurance information.

[0089] The reward function is a key design element of this game-theoretic adversarial self-learning method. A reasonable reward function guides the direction of the game-theoretic self-learning and directly affects the success of policy search. Essentially, the reward function sets a feedback mechanism for the algorithm to evaluate the merits of the searched strategies. Each time an action is executed, the environment provides a reward. The main goal of game-theoretic adversarial self-learning is to maximize long-term gains. By maximizing the cumulative discounted reward, the optimal task scheduling strategy is obtained. For the unmanned swarm task scheduling scenario, the following real-time reward function is designed:

[0090]

[0091]

[0092] Among them, (r) t Let be the reward function value at time t, i be the label of the unmanned swarm, n be the number of unmanned swarms, and (r i ) t Let K be the reward function of the unmanned bee colony i at time t, and K be the reward discount factor, where 0 < K < K. <K<1,d li This represents the battery life information of drone swarm i, d ui This represents the payload information of unmanned bee colony i.

[0093] According to the calculation formula of the reward function, the weight of the cumulative reward decreases with time; the remaining payload d of the unmanned bee colony i ui A smaller value indicates a larger reward, which means higher task execution efficiency; the endurance information d of the unmanned swarm i. liThe higher the reward value, the greater the reward, which means the less time the task takes to execute. If the action is prohibited, a reward value of -500 is directly assigned, which is equivalent to a soft constraint on prohibited actions.

[0094] Step S3: Based on the state space, action space, state transition rules, and reward function, construct a game-theoretic adversarial self-learning unmanned swarm task scheduling model; the game-theoretic adversarial self-learning unmanned swarm task scheduling model includes an Actor network and a Critic network.

[0095] Step S4: Train the game-adversarial self-learning unmanned swarm task scheduling model to obtain the trained model.

[0096] Step S5: Input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, i.e., the task scheduling strategy. Here, the state space is constructed based on the task to be scheduled and the unmanned bee swarm.

[0097] This embodiment, based on the background of unmanned swarm task scheduling, sets up a more adaptive state space, action space, state transition, and reward function, which can be better used for game-adversarial self-learning. Instead of terminating prohibited actions, it perturbs them, transforming them into executable actions, resulting in more effective samples and improving algorithm efficiency. The actor-critic algorithm from reinforcement learning is improved and applied to unmanned swarm task scheduling, realizing the generation of game-adversarial self-learning unmanned swarm task scheduling strategies. Unlike supervised learning, it does not require a large number of data samples.

[0098] In one embodiment, step S4 involves training a game-theoretic adversarial self-learning unmanned swarm task scheduling model to obtain a trained model, including:

[0099] Step S41: Initialize the state space S and the experience pool Ψ, with an initial time t = 0;

[0100] Step S42, let t = t + 1, and convert the current state space (S) to... t The input is fed into the Actor network to obtain λ action samples A = (a1, a2, ..., a...). i …a n ), where a i The action taken by the unmanned bee colony i at the current moment;

[0101] Step S43: Input λ action samples into the Critic network to obtain the reward function value corresponding to each action sample, and determine the maximum value r of the reward function value. * Corresponding action sample A * ;

[0102] Step S44, if A* There is an action a i If it belongs to a prohibited action, then for action a i Perform perturbation to obtain an executable action A σ ; Based on the state space (S) at the current moment t Execute action A σ , Generate the state space (S) at the next moment t+1 ; Here, the state space at the next moment is generated according to the formula in Case 1 of the state transition rule

[0103] If there is no action belonging to the prohibited action in A * , Based on the state space (S) at the current moment t Execute action A * , Generate the state space (S) at the next moment t+1 ;

[0104] Step S45, Construct a five - tuple {(S) t , A * , r * , (S) t+1 , t} and put it into the experience pool Ψ;

[0105] Step S46, If t < T, where T is the length of the Episode, then return to Step S42; Otherwise, execute Step S47;

[0106] Step S47, Use the data in the experience pool Ψ to perform gradient backpropagation to update the Actor network network parameter ω a and the Critic network parameter ω c ; If the changes in both the Actor network network parameter ω a and the Critic network parameter ω c are less than 10 -4 , Then the training terminates, and the trained model is obtained; Otherwise, return to Step S41.

[0107] In this embodiment, first set the length T of the Episode, the Actor network network parameter ω a , the Critic network parameter ω c , The state space S composed of n unmanned bee swarms and m task requirements = (D g , D r , D p , D l , D u ), The action space A = (a1, a2,... a i ... a n ), The reward discount factor K, the sampling interval ΔT, and the batch sample quantity λ.

[0108] Employing the same inventive concept as the game-theoretic adversarial self-learning-based unmanned swarm task scheduling method, this embodiment also provides a corresponding game-theoretic adversarial self-learning-based unmanned swarm task scheduling device. Figure 2 A structural block diagram of an unmanned swarm task scheduling device based on game-theoretic adversarial self-learning according to an embodiment of this application is shown. The device includes:

[0109] The first construction module 21 is used to construct the state space, action space, and state transition rules for unmanned swarm task scheduling; the state space includes task payload information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm payload information; the action space includes the actions taken by the unmanned swarm at the current moment;

[0110] The reward function setting module 22 is used to set the reward function for game-based adversarial self-learning under the background of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the unmanned swarm payload information, and the unmanned swarm endurance information.

[0111] The second building module 23 is used to construct a game-theoretic self-learning unmanned swarm task scheduling model based on the state space, action space, state transition rules, and reward function; the game-theoretic self-learning unmanned swarm task scheduling model includes an Actor network and a Critic network.

[0112] Model training module 24 is used to train the game adversarial self-learning unmanned swarm task scheduling model to obtain the trained model.

[0113] The task scheduling strategy determination module 25 is used to input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, i.e., the task scheduling strategy.

[0114] The unmanned swarm task scheduling device based on game-theoretic adversarial self-learning in this embodiment has the same inventive concept as the unmanned swarm task scheduling method based on game-theoretic adversarial self-learning described above. Therefore, the specific implementation of this device can be found in the embodiment section of the unmanned swarm task scheduling method based on game-theoretic adversarial self-learning described above, and its technical effects correspond to the technical effects of the above method, so they will not be repeated here.

[0115] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described game-theoretic adversarial self-learning-based unmanned swarm task scheduling method.

[0116] In summary, this application has the following technical effects:

[0117] By employing a game-theoretic adversarial self-learning-based unmanned swarm task scheduling method, multiple unmanned swarms can generate scheduling strategies based on existing task scenarios, completing tasks at a lower cost. Unmanned swarm task scheduling is a widely applied and significant technology, playing a crucial role in various fields such as power line inspection, traffic monitoring, environmental monitoring, agricultural applications, and fire reconnaissance. Through automated and intelligent task scheduling, unmanned swarms can significantly improve inspection efficiency, reduce manpower and material resources and operating costs, while simultaneously enabling safe and accurate inspections in high-risk or hard-to-reach areas. The research and application of this technology not only promotes the progress and innovation of related technologies but also provides strong support for improving inspection efficiency, safety, and accuracy, which is of great significance to the sustainable development of modern society.

[0118] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for scheduling unmanned bee swarm tasks based on game-theoretic adversarial self-learning, characterized in that, include: Construct a state space, action space, and state transition rules for unmanned swarm task scheduling; the state space includes task payload information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm payload information; the action space includes the actions taken by the unmanned swarm at the current moment; Set a reward function for game-based adversarial self-learning under the background of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the payload information of the unmanned swarm, and the endurance information of the unmanned swarm; Based on the state space, the action space, the state transition rules, and the reward function, a game-theoretic adversarial self-learning unmanned swarm task scheduling model is constructed; the game-theoretic self-learning unmanned swarm task scheduling model includes an Actor network and a Critic network; The game-theoretic self-learning unmanned bee swarm task scheduling model is trained to obtain the trained model. The state space corresponding to the task to be scheduled is input into the trained model to obtain the action space, i.e., the task scheduling strategy. The state transition rule is as follows: Scenario 1, when the action When it occurs, if the action Not belonging to unmanned bee colonies If a prohibited action is selected, a new state is generated according to the following formula: in, For action Whether the information is still on the current task For action The currently executing task number in the middle. For action The next task number that China expects to execute is... The task number is The payload information required for the task. The task number is Task time information; The task number is The payload information required for the task. The task number is Task time information, Unmanned bee colony Location information, Indicates unmanned bee swarm Battery life information, Indicates unmanned bee swarm Load information; express Data at any given time express Data at any given time; The sampling interval; Case 2, when the action When it occurs, if the action Belongs to an unmanned bee colony Prohibited actions, actions The perturbation is performed to obtain an executable action, and then a new state is generated according to the formula in case 1.

2. The method as described in claim 1, characterized in that, The state space uses state feature vectors express, The mission requires payload information. For task time information, For the location information of the unmanned bee swarm, Information on the battery life of unmanned bee swarms. For unmanned bee swarm payload information; , Indicates task The required load information, i.e., how much load needs to be consumed; , Indicates task The time information, i.e., how much time is still needed to complete the current task; , Indicates unmanned bee swarm The location information, i.e., the task number that the unmanned swarm is currently executing; , Indicates unmanned bee swarm The remaining battery life information, that is, the remaining battery life information of the current unmanned bee swarm; , Indicates unmanned bee swarm The load information, that is, the remaining load information of the current unmanned bee colony; The state feature vector is subject to the following constraints: in, For the number of tasks, This represents the number of unmanned bee colonies.

3. The method as described in claim 1, characterized in that, The action space is ,in, Unmanned bee colony The actions taken at the present moment; Including information on whether to remain on the current task. The currently executing task number And the expected next task number .

4. The method as described in claim 1, characterized in that, The reward function is: in, For a moment The reward function value, The label for the unmanned bee colony. The number of unmanned bee colonies. Unmanned bee colony At any moment The reward function, As a reward discount factor, Indicates unmanned bee swarm Battery life information, Indicates unmanned bee swarm Load information.

5. The method as described in claim 1, characterized in that, in, The game-theoretic adversarial self-learning unmanned swarm task scheduling model is trained to obtain the trained model, which includes: Step S41, Initialize the state space and experience pool Initial time ; Step S42, let The state space at the current moment Input into the Actor network to obtain Action Samples ,in, Unmanned bee colony The actions taken at the present moment; Step S43, will Each action sample is input into the Critic network to obtain the reward function value corresponding to each action sample, and the maximum value of the reward function value is determined. Corresponding action samples ; Step S44, if There are actions in it If it is a prohibited action, then the action is... By perturbing, an executable action can be obtained. Based on the state space at the current moment Execute action Generate the state space for the next time step. ; like There are no prohibited actions in the current state space. Execute action Generate the state space for the next time step. ; Step S45, construct a quintuple Add to experience pool ; Step S46, if , If the length is equal to the episode length, return to step S42; otherwise, proceed to step S47. Step S47, utilize the experience pool The data in the dataset is used to perform gradient backpropagation on the Actor and Critic networks to update the Actor network parameters. and Critic network parameters If the Actor network parameters and Critic network parameters The changes were all smaller than If the training is successful, the training will terminate and the trained model will be obtained; otherwise, return to step S41.

6. A game-theoretic adversarial self-learning-based unmanned bee swarm task scheduling device, characterized in that, include: The first construction module is used to construct the state space, action space, and state transition rules for unmanned swarm task scheduling; the state space includes task payload information, task time information, unmanned swarm location information, unmanned swarm endurance time information, and unmanned swarm payload information; the action space includes the actions taken by the unmanned swarm at the current moment; The reward function setting module is used to set the reward function for game-based adversarial self-learning under the background of unmanned swarm task scheduling; the reward function is related to whether the action taken by the unmanned swarm at the current moment is a prohibited action, the payload information of the unmanned swarm, and the endurance information of the unmanned swarm. The second construction module is used to construct a game-theoretic self-learning unmanned swarm task scheduling model based on the state space, the action space, the state transition rules, and the reward function; the game-theoretic self-learning unmanned swarm task scheduling model includes an Actor network and a Critic network. The model training module is used to train the game-adversarial self-learning unmanned swarm task scheduling model to obtain the trained model. The task scheduling strategy determination module is used to input the state space corresponding to the task to be scheduled into the trained model to obtain the action space, i.e., the task scheduling strategy. The state transition rule is as follows: Scenario 1, when the action When it occurs, if the action Not belonging to unmanned bee colonies If a prohibited action is selected, a new state is generated according to the following formula: in, For action Whether the information is still on the current task For action The currently executing task number in the middle. For action The next task number that China expects to execute is... The task number is The payload information required for the task. The task number is Task time information; The task number is The payload information required for the task. The task number is Task time information, Unmanned bee colony Location information, Indicates unmanned bee swarm Battery life information, Indicates unmanned bee swarm Load information; express Data at any given time express Data at any given time; The sampling interval; Case 2, when the action When it occurs, if the action Belongs to an unmanned bee colony Prohibited actions, actions The perturbation is performed to obtain an executable action, and then a new state is generated according to the formula in case 1.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the unmanned swarm task scheduling method based on game-theoretic adversarial self-learning as described in any one of claims 1-5.

Images