Unmanned aerial vehicle group frequency selection and task autonomous allocation method based on alliance formation game and reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle group frequency selection and task autonomous allocation method based on alliance formation game and reinforcement learning, and the method comprises the steps: setting a simulation scene, and carrying out the game modeling and solving for frequency selection; a reinforcement learning network is constructed, the input of the reinforcement learning network is the coordinates of the unmanned aerial vehicle, the alliance where the unmanned aerial vehicle is located, the task value of each ground node and the distance between the unmanned aerial vehicle and each ground node, the output of the reinforcement learning network is the alliance selected by the unmanned aerial vehicle, and a Q function is used to replace task rewards as an evaluation function to train the reinforcement learning network; and forming game modeling by using an alliance, evaluating alliance income by using a reinforcement learning network, and solving to obtain an unmanned aerial vehicle group task allocation mode. According to the method, the problems that continuous multi-time task allocation cannot be directly solved, task reward information in different time periods cannot be fully utilized and calculation cannot be performed when unknown reward information exists in an alliance formation game can be solved, and the long-term benefit of task execution of an unmanned aerial vehicle group in a dynamic environment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of drone swarms, and in particular to a method for selecting frequencies and autonomously allocating tasks for a drone swarm based on alliance formation game and reinforcement learning. Background Art

[0002] Drones have the characteristics of high maneuverability and easy deployment, and have been widely used in military and civilian fields. With the development of communication technology, the use of miniaturized drones to form clusters to perform tasks has become increasingly popular and has a wide range of application scenarios. Compared with a single drone, drone clusters can achieve functional complementarity, perform more complex tasks, and improve work efficiency. At present, drone clusters have been widely used in military coordinated reconnaissance and strike, information collection, link relay and other fields.

[0003] When drone swarms are working, due to the uneven distribution of task requirements, in order to better complete the task, drone swarms are required to achieve reasonable collaborative task allocation, so as to make full use of drones, spectrum and other resources and improve the work efficiency of drones. As a current research hotspot, collaborative task allocation for drone swarms has received extensive attention from scholars at home and abroad.

[0004] At present, for different working environments, the research methods of cooperative task allocation of drone swarms mainly focus on mixed integer linear programming, neural network algorithms and various heuristic algorithms. At the same time, in some complex mission scenarios, drone swarms need to solve multiple tasks at the same time, and the tasks require multiple drones to execute. At this time, using multiple drones to form an alliance to complete the task together is an effective way. However, many current studies only focus on static environments with fixed task requirements, and cannot solve the situation where task requirements change dynamically, and do not fully utilize the advantages of flexible and dynamic deployment of drone swarms. In addition, as the number of drones participating in cooperative task allocation increases, the variables that need to be considered continue to increase, which will lead to increased computational complexity. Summary of the invention

[0005] The purpose of the present invention is to propose a method for selecting the frequency and autonomously allocating tasks of a drone swarm based on alliance formation game and reinforcement learning, which can solve the problems that the alliance formation game cannot directly solve the allocation of multiple consecutive tasks, cannot make full use of the task reward information of different time periods, and cannot calculate when there is unknown reward information; it uses environmental experience data to improve the long-term benefits of drone swarms performing tasks in a dynamic environment, and can avoid the increase in computational complexity caused by the increase in the number of drones.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A method for selecting a frequency for a drone swarm and autonomously allocating tasks based on alliance formation game and reinforcement learning, the method comprising the following steps:

[0008] S1, setting up simulation scenarios, modeling and solving game problems for frequency selection;

[0009] S2, build a reinforcement learning network. The input of the reinforcement learning network is the coordinates of the drone, the alliance the drone is currently in, the mission value of each ground node, and the distance between the drone and each ground node. The output is the alliance selected by the drone. The Q function is used instead of the task reward as the evaluation function to train the reinforcement learning network.

[0010] S3, using the alliance formation game modeling, using the reinforcement learning network to evaluate the alliance benefits, and solving the drone swarm task allocation method.

[0011] Step S1 further comprises:

[0012] S11, set up a simulation scenario. The scenario system includes N ground receiving nodes and M drones. The ground receiving node set is represented by n = {1, 2, ..., N}, the drone group is represented by m = {1, 2, ..., M}, and the available channels are represented by c = {1, 2, ..., C}. Each time the drone selects a ground receiving node and channel, it unloads the data to the ground receiving node. The task requirements of the ground node are refreshed every T0 time. During the whole process, a total of N T After the task is refreshed, if the ground node where the drone data is unloaded changes, the drone moves to the ground node close to the data unloading, so that the distance between the drone and the ground node is less than the transmission threshold distance d th , the moving speed of all drones is speed v; for the ground node, there is mutual interference between the multiple drone signals it receives;

[0013] S12, calculate the communication signal-to-noise ratio (SNR) between UAV j and the i-th ground receiving node at time T j,i (T) and the maximum communication transmission rate R j,i (T);

[0014] S13, calculate the task benefit U of ground node i in a refresh cycle T according to the amount of data received by ground node i in cycle T Si (T), and then get the whole network task income U N ;

[0015] S14, maximizing the mission benefits of the entire network is taken as an optimization problem, and the optimization problem is constructed as a coalition formation game problem with transferable utility; in the constructed coalition game, the drone acts as a game participant, and its action is the selection of the ground receiving node for data unloading.

[0016] The drone communicates with the ground receiving node using the 802.11b system's Wi-Fi signal.

[0017] Step S12 further comprises:

[0018] The communication signal-to-noise ratio (SNR) between UAV j and the i-th ground receiving node at time T is j,i (T) is expressed as:

[0019]

[0020] Where P j represents the transmission power of the jth UAV, σ 2 represents the noise power received by the ground receiving node, d j,i (T) represents the distance between UAV j and the i-th ground receiving node at time T, α l represents the path loss exponent; represents the co-frequency and adjacent-frequency interference of other UAVs in the environment, g in (m,i) represents the path loss of the interference signal, Other drones select channel C m With the drone current channel C j The spectrum interference situation IF(m,j) is:

[0021]

[0022] When SNR j,i When (T)>SNR0, UAV j successfully connects to the i-th ground receiving node at time T; the maximum communication transmission rate between UAV j and the i-th ground receiving node at time T is defined as R j,i (T) = Blog2(1 + SNR j,i (T)), where B is the channel bandwidth.

[0023] Step S13 further comprises:

[0024] According to the signal receiving situation of each WiFi channel of the ground node, the set RSS={rss1,rss2,…,rss n}, solve to get the drone reward:

[0025]

[0026] Select drone j, other drone channels c in turn -j Fixed, drone j changes its channel c j Change the group reward; maximize the changed group reward as the optimization goal, and allocate the drone spectrum in the alliance in order through iteration;

[0027] Define A = [a j ] 1×n is the task allocation matrix, where a j represents the mission selection of UAV j; define S i is the set of drones that select ground node i, S i ={j∈N:a j =i}, satisfying Each drone can only select one ground node to unload data.

[0028] Define the task benefit of ground node i within a refresh cycle T for:

[0029]

[0030] Where val(i,T) represents the task value at ground node i in period T; D(S i ,T) represents the amount of data received by ground node i within period T, and the data transmission rate R j,i (T) and the length of the data unloading time t of the drone in a period T w (j) Related:

[0031]

[0032] The revenue of the whole network task is:

[0033]

[0034] Step S14 further comprises:

[0035] Alliance S i The return in period T is defined as:

[0036]

[0037] The drone utility within each alliance is defined as the value of the data contributed by the drone:

[0038] u j (S i ,T)=R j,i (T)*t w (j)*val(i,T)

[0039] The expected alliance mutual benefit criterion is adopted. Based on the expected alliance mutual benefit criterion, the alliance formation is realized at different times T. The expected alliance mutual benefit criterion is: when drone j joins the new alliance i1, the total utility of drone j and the drones leaving i0 and joining alliance i1 is improved:

[0040]

[0041] Among them, the single alliance formation process under the principle of expected alliance mutual benefit includes the following steps:

[0042] Initialize the number of iterations and randomly assign the drones to alliances;

[0043] Randomly select drone j and calculate the current estimated utility

[0044] Randomly select the ground node i1 with different data unloading and calculate the current estimated utility U(S i1 );

[0045] According to the expected alliance mutual benefit principle, compare U(S i0 ) and U(S i1 ), if U(S i0 )<U(S i1 ), change the data offloading object, and use the spectrum allocation method to change the spectrum allocation until all drones and ground nodes are traversed.

[0046] Step S2 further comprises:

[0047] Using the evaluation function Q(S i ,T) Alternative task reward val(S i ,T), for the function Q(S i ,T), train the reinforcement learning network in a single drone environment and save the Q function Q(S i ,T);

[0048] During the training process, the input state s of the reinforcement learning network T Including drone coordinates, drone current alliance, ground node and drone rewards, ground node current or past mission value, the output is the selected alliance a T ; Define the reinforcement learning network loss function as follows:

[0049]

[0050] where r T represents the reward, γ is the discount rate, θ and θ - are the weights of the training network and the target network, respectively.

[0051] Step S3 further comprises:

[0052] S31, in the current environment setting and single drone case, the evaluation function Q(S i ,T); UAVs are randomly assigned to alliances; set the current time T=0;

[0053] S32, the task requirements of the ground node are refreshed. According to the allocation of the drone alliance, the drone moves and unloads data. After the work is completed, the environmental status information is refreshed. T ;

[0054] S33, set the total number of iterations to 500, and set the current number of iterations to 0;

[0055] S34, randomly select drone j and calculate the alliance S under the reinforcement learning network evaluation i0 The utility:

[0056]

[0057] Randomly select ground node i1 for unloading different data;

[0058] According to the principle of mutual benefit of expected alliance, if G(S i0 ,T)<G(S i1 ,T), change the data offloading object and use the spectrum allocation method to change the spectrum allocation;

[0059] S35, the current number of iterations is increased by 1, and the process returns to step S34 until the current number of iterations reaches the total number of iterations;

[0060] S36, T=T+1, determine whether T is less than N T , if T <N T , return to step S32, otherwise end.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] First, the method of drone swarm frequency selection and autonomous task allocation based on alliance formation game and reinforcement learning of the present invention combines the frequency selection within the alliance with the autonomous task allocation between alliances, performs game modeling and solves the frequency selection problem, and can flexibly and quickly reduce the same-frequency interference within the alliance.

[0063] Second, the method for drone swarm frequency selection and autonomous task allocation based on alliance formation game and reinforcement learning of the present invention uses alliance formation game modeling and solving drone swarm task allocation. The number of iterations is independent of the number of drones, which can effectively reduce the algorithm calculation complexity.

[0064] Third, the method for selecting the frequency and autonomously allocating tasks of drone swarms based on alliance formation game and reinforcement learning of the present invention uses a reinforcement learning algorithm to evaluate the benefits of the alliance, which can adapt to a variety of different tasks and improve the long-term benefits of drone swarms performing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a model diagram of the UAV task allocation system of the present invention.

[0066] Figure 2 This is a schematic diagram of the transmission rate after the frequency selection method within the alliance of the present invention is used.

[0067] Figure 3 This is a comparison chart between the task autonomous allocation algorithm of the present invention and the baseline method when the ground node task reward is completely known.

[0068] Figure 4 This is a comparison chart between the task autonomous allocation algorithm of the present invention and the baseline method under the condition of random ground node task rewards. DETAILED DESCRIPTION

[0069] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0070] The present invention discloses a method for selecting a frequency for a drone swarm and autonomously allocating tasks based on alliance formation game and reinforcement learning. The method comprises the following steps:

[0071] S1, setting up simulation scenarios, modeling and solving game problems for frequency selection;

[0072] S2, build a reinforcement learning network. The input of the reinforcement learning network is the coordinates of the drone, the alliance the drone is currently in, the mission value of each ground node, and the distance between the drone and each ground node. The output is the alliance selected by the drone. The Q function is used instead of the task reward as the evaluation function to train the reinforcement learning network.

[0073] S3, using the alliance formation game modeling, using the reinforcement learning network to evaluate the alliance benefits, and solving the drone swarm task allocation method.

[0074] In summary, the method for selecting and autonomously allocating tasks for a drone swarm of the present invention includes three aspects: game modeling and solving for frequency selection, alliance benefit evaluation based on reinforcement learning, and using alliance formation game modeling and solving the task allocation of drone swarms. The specific algorithm flow includes:

[0075] Step 1. Define the simulation scenario. The system contains N ground receiving nodes and M drones. The ground receiving node set is represented by N = {1, 2, ..., N}, and the drone group is represented by M = {1, 2, ..., M}. Consider the use of unlicensed frequency bands, such as wifi channels. The available channels can be represented by C = {1, 2, ..., C}. Each time the drone selects a ground receiving node and channel, it unloads the data to the ground receiving node. The task requirements of the ground node are refreshed every T0 time. During the whole process, a total of N T To describe the demand of ground nodes for data transmission, the data value val(i) is used for evaluation, where i represents the ground receiving node i. When the UAV unloads data to the ground node, the distance between the UAV and the ground node needs to be less than the transmission threshold distance d due to the influence of wireless transmission attenuation. th Therefore, after the task is refreshed, if the ground node where the drone data is unloaded changes, the drone needs to move until it is close to the ground node where the data is unloaded. Without loss of generality, the moving speed of all drones is set to speed v. Figure 1 This is a model diagram of the UAV task allocation system of the present invention.

[0076] Step 2: The drone and ground node communicate using the 802.11b system's wifi signal, which has 11 partially overlapping channels. The bandwidth of each channel is 22MHz, and the channel spacing is 5MHz. For the ground node, there is mutual interference between the multiple drone signals it receives.

[0077] Step 3: The communication signal-to-noise ratio (SNR) between UAV j and the i-th ground receiving node at time T j,i (T) can be expressed as formula (1.1), where P j represents the transmission power of the jth UAV, σ 2 represents the noise power received by the ground receiving node, d j,i (T) represents the distance between UAV j and the i-th ground receiving node at time T, and the path loss index is expressed as α l .

[0078]

[0079] in Indicates the co-frequency and adjacent frequency interference of other drones in the environment. IF indicates the spectrum interference. Due to the use of partially overlapping WiFi signals, when other drones select channel C m With the drone current channel C j The relationship is as follows:

[0080]

[0081] And g in(m,i) represents the path loss of the interference signal:

[0082]

[0083] When SNR j,i When (T)>SNR0, it can be considered that UAV m is successfully connected to the nth ground receiving node at time t, so that the mission proceeds normally.

[0084] Based on the above analysis, the maximum transmission rate of communication between UAV j and the i-th ground receiving node at time T can be defined as R j,i (T) = Blog2(1 + SNR j,i (T)), where B is the channel bandwidth.

[0085] Step 4: When the ground node receives signals from each WiFi channel, the set RSS = {rss1, rss2, ..., rss n}When known, the drone reward can be solved directly

[0086]

[0087] When other drone channels c -j When fixed, drone j can change its channel c j Change the group reward, and the group reward after the change is U(c -j ,c j ′) is known, and the spectrum of drones in the alliance is allocated in sequence through iteration. The specific algorithm is shown in Algorithm 1.1.

[0088]

[0089] Figure 2 This is a schematic diagram of the transmission rate after the frequency selection method within the alliance of the present invention is used.

[0090] Step 5: Define A = [a j ] 1×n is the task allocation matrix, where a j Denotes the mission selection of UAV j. Define S i is the set of drones that select ground node i, namely S i ={j∈N:a j =i}, satisfying In addition, each UAV can only select one ground node to offload data, so

[0091] Considering that the mission benefit is proportional to the mission value and the amount of transmitted data, the mission benefit of ground node i in a refresh cycle T is defined as for

[0092]

[0093] Where D(S i ,T) represents the amount of data received by ground node i within period T, which is consistent with the data transmission rate R introduced above. j,i (T) and the length of the data unloading time t of the drone in a period T w (j) Related:

[0094]

[0095] In summary, the revenue of the whole network task is:

[0096]

[0097] The optimization goal is to maximize the task benefits of the entire network. Figure 3 This is a comparison chart between the task autonomous allocation algorithm of the present invention and the baseline method when the ground node task reward is completely known.

[0098] Step 6: The problem is constructed as a transferable utility (TU) coalition formation game problem. In the constructed coalition game, drones act as game participants, and their actions are to select the ground receiving node for data offloading. i The return in period T is defined as:

[0099]

[0100] The utility of drones in each alliance is defined as the data value contributed by the drone, namely:

[0101] u j (S i ,T)=R j,i (T)*t w (j)*val(i,T)

[0102] In order to comprehensively consider the impact of alliance changes on the benefits of all alliances, the expected alliance mutual benefit criterion is adopted in the current problem. In this criterion, when drone j joins the new alliance i1, the total utility of drone j and the drones that leave alliance i0 and join alliance i1 is improved.

[0103]

[0104] Under the principle of expected alliance mutual benefit, alliance formation can be achieved at different times T. This method can be regarded as a baseline algorithm (benchmark). The specific process is shown in Algorithm 1.2.

[0105]

[0106] Step 7: In Algorithm 1.2, only the alliance benefits in period T are considered, without considering the impact of alliance switching on subsequent changes. Other functions Q(S i ,T) Alternative task reward val(S i ,T). For the function Q(S i ,T), train the DQN network in a single drone environment and save the Q function Q(S i ,T).

[0107] The DQN network model is set as follows: the number of input channels is 11, and the number of output channels is 4. The coordinates of the drone, the alliance the drone is currently in, the mission value of each ground node, and the distance between the drone and each ground node are injected into the reinforcement learning network as input data. During the iteration process, the DQN algorithm is used to update the neural network parameters, which lasts for a total of 500 iterations; during this period, the latest trained network will be used for performance testing. The learning rate is initialized to 1e -2 .

[0108] During the training process, the DQN network input state s T Including drone coordinates, drone current alliance, ground node and drone rewards, ground node current or past mission value, input is the selected alliance a T The reinforcement learning network loss function is defined as follows:

[0109]

[0110] where r T represents the reward, γ is the discount rate, θ and θ - are the weights of the training network and the target network, respectively.

[0111] Step 8: When evaluating the game, use the Q function to replace val(S i ,T), that is, the estimated value of the comparative alliance G, whose mathematical expression is The specific process is shown in Algorithm 1.3:

[0112]

[0113]

[0114] After the alliance is formed and the game model is established, the present invention does not directly use the profit definition formula to solve the current benefits of the drone alliance, but uses a reinforcement learning network based on DQN (Deep Q Network) to evaluate the alliance benefits. Figure 4 This is a comparison chart between the task autonomous allocation algorithm of the present invention and the baseline method under the condition of random ground node task rewards.

[0115] In summary, the present invention realizes the frequency selection and autonomous task allocation algorithm of drone swarm based on alliance formation game and reinforcement learning, successfully realizes alliance formation convergence using alliance formation game, and uses task value experience data to train reinforcement learning DQN network, evaluates alliance value while considering future benefits, improves total task benefits after multiple task allocations, and solves the problems that alliance formation game cannot directly solve continuous multiple task allocations, cannot fully utilize task reward information in different time periods, and cannot calculate when there is unknown reward information. The good characteristics of the present invention are proved by simulation and comparison with multiple algorithms.

[0116] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0117] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0120] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0121] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for selecting frequency and autonomously allocating tasks for drone swarms based on alliance formation game and reinforcement learning, characterized in that: The method for selecting the frequency of use of the drone group and autonomously allocating tasks comprises the following steps: S1, setting up simulation scenarios, modeling and solving game problems for frequency selection; S2, build a reinforcement learning network. The input of the reinforcement learning network is the coordinates of the drone, the alliance the drone is currently in, the mission value of each ground node, and the distance between the drone and each ground node. The output is the alliance selected by the drone. The Q function is used instead of the task reward as the evaluation function to train the reinforcement learning network. S3, using the alliance formation game modeling, using the reinforcement learning network to evaluate the alliance benefits, and solving the drone swarm task allocation method.

2. The method for selecting frequency and autonomously allocating tasks for drone swarms based on alliance formation game and reinforcement learning according to claim 1 is characterized in that: Step S1 further comprises: S11, set up a simulation scenario. The scenario system includes N ground receiving nodes and M drones. The ground receiving node set is represented by n = {1, 2, ..., N}, the drone group is represented by m = {1, 2, ..., M}, and the available channels are represented by c = {1, 2, ..., C}. Each time the drone selects a ground receiving node and channel, it unloads the data to the ground receiving node. The task requirements of the ground node are refreshed every T0 time. During the whole process, a total of N T After the task is refreshed, if the ground node where the drone data is unloaded changes, the drone moves to the ground node close to the data unloading, so that the distance between the drone and the ground node is less than the transmission threshold distance d th , the moving speed of all drones is speed v; for the ground node, there is mutual interference between the multiple drone signals it receives; S12, calculate the communication signal-to-noise ratio (SNR) between UAV j and the i-th ground receiving node at time T j,i (T) and the maximum communication transmission rate R j,i (T); S13, calculate the task benefit of ground node i in a refresh cycle T according to the amount of data received by ground node i in cycle T Then we get the whole network task income U N ; S14, maximizing the mission benefits of the entire network is taken as an optimization problem, and the optimization problem is constructed as a coalition formation game problem with transferable utility; in the constructed coalition game, the drone acts as a game participant, and its action is the selection of the ground receiving node for data unloading.

3. The method for selecting frequency and autonomously allocating tasks of drone swarms based on alliance formation game and reinforcement learning according to claim 1 is characterized in that: The drone communicates with the ground receiving node using the 802.11b system's Wi-Fi signal.

4. The method for selecting frequency and autonomously allocating tasks of drone swarms based on alliance formation game and reinforcement learning according to claim 1 is characterized in that: Step S12 further comprises: The communication signal-to-noise ratio (SNR) between UAV j and the i-th ground receiving node at time T is j,i (T) is expressed as: Where P j represents the transmission power of the jth UAV, σ 2 represents the noise power received by the ground receiving node, d j,i (T) represents the distance between UAV j and the i-th ground receiving node at time T, α l represents the path loss exponent; represents the co-frequency and adjacent-frequency interference of other UAVs in the environment, g in represents the path loss of the interference signal, Other drones select channel C m With the drone current channel C j The spectrum interference situation IF(m,j) is: When SNR j,i When (T)>SNR0, UAV j successfully connects to the i-th ground receiving node at time T; the maximum communication transmission rate between UAV j and the i-th ground receiving node at time T is defined as R j,i (T) = Blog2(1 + SNR j,i (T)), where B is the channel bandwidth.

5. The method for selecting frequency and autonomously allocating tasks of drone swarms based on alliance formation game and reinforcement learning according to claim 1 is characterized in that: Step S13 further comprises: According to the signal receiving situation of each WiFi channel of the ground node, the set RSS={rss1,rss2,…,rss n }, solve to get the drone reward: Select drone j in turn, and other drones select channel c -j Fixed, drone j selects channel c by changing itself j Change the group reward; maximize the changed group reward as the optimization goal, and allocate the drone spectrum within the alliance in order through iteration; Define A = [a j ] 1×n is the task allocation matrix, where a j represents the mission selection of UAV j; define S i is the set of drones that select ground node i, S i ={j∈N:a j =i}, satisfying Each drone can only select one ground node to unload data. Define the task benefit of ground node i within a refresh cycle T for: Where val(i,T) represents the task value at ground node i in period T; D(S i ,T) represents the amount of data received by ground node i within period T, and the data transmission rate R j,i (T) and the length of the data unloading time t of the drone in a period T w (j) Related: The revenue of the whole network task is:

6. The method for selecting frequency and autonomously allocating tasks of drone swarms based on alliance formation game and reinforcement learning according to claim 1 is characterized in that: Step S14 further comprises: Alliance S i The return in period T is defined as: The drone utility within each alliance is defined as the value of the data contributed by the drone: u j (S i ,T)=R j,i (T)*t w (j)*val(i,T) The expected alliance mutual benefit criterion is adopted. Based on the expected alliance mutual benefit criterion, the alliance formation is realized at different times T. The expected alliance mutual benefit criterion is: when a drone joins a new alliance, the total utility of the drone and the drones leaving and joining the alliance is improved: Among them, the single alliance formation process under the principle of expected alliance mutual benefit includes the following steps: Initialize the number of iterations and randomly assign the drones to alliances; Randomly select drone j, whose data is unloaded to the ground node i0, and calculate the current estimated utility Randomly select the ground node i1 with different data unloading and calculate the current estimated utility U(S i1 ) and the estimated utility U(S) after UAV j changes the data offloading object i0 \j)、U(S i1 ∪j); According to the principle of mutual benefit of expected alliance, if Change the data offloading object and use the spectrum allocation method to change the spectrum allocation until all drones and ground nodes are traversed.

7. The method for selecting frequency and autonomously allocating tasks of drone swarms based on alliance formation game and reinforcement learning according to claim 1 is characterized in that: Step S2 further comprises: Using the evaluation function Q(S i ,T) Alternative task reward val(S i ,T), for the function Q(S i ,T), train the reinforcement learning network in a single drone environment and save the Q function Q(S i ,T); During the training process, the input state s of the reinforcement learning network T Including drone coordinates, drone current alliance, ground node and drone rewards, ground node current or past mission value, the output is the selected alliance a T ; Define the reinforcement learning network loss function as follows: where r T represents the reward, γ is the discount rate, θ and θ - are the weights of the training network and the target network, respectively.

8. The method for selecting frequency and autonomously allocating tasks for drone swarms based on alliance formation game and reinforcement learning according to claim 1 is characterized in that: Step S3 further comprises: S31, in the current environment setting and single drone case, the evaluation function Q(S i0 ,T); UAVs are randomly assigned to alliances; set the current time T=0; S32, the task requirements of the ground node are refreshed. According to the allocation of the drone alliance, the drone moves and unloads data. After the work is completed, the environmental status information is refreshed. T ; S33, set the total number of iterations to 500, and set the current number of iterations to 0; S34, randomly select drone j and calculate the alliance S under the reinforcement learning network evaluation i0 The utility: Randomly select ground node i1 for unloading different data; According to the principle of mutual benefit of expected alliance, if Change the data offloading object and use the spectrum allocation method to change the spectrum allocation situation; S35, the current number of iterations is increased by 1, and the process returns to step S34 until the current number of iterations reaches the total number of iterations; S36, T=T+1, determine whether T is less than N T , if T <N T , return to step S32, otherwise end.

Images