Unmanned aerial vehicle cluster topology reconstruction model training method and system

Abstract

The invention relates to an unmanned aerial vehicle cluster topology reconstruction model training method and system, belongs to the technical field of unmanned aerial vehicle communication, and solves the problem of low topology reconstruction rationality caused by large action space of an existing unmanned aerial vehicle topology reconstruction model. Comprising the steps that a topology reconstruction model is constructed, each proxy network comprises a mobile sub-proxy network and a transmission sub-proxy network, and the mobile sub-proxy network and the transmission sub-proxy network receive local state information of an unmanned aerial vehicle at all moments and actions corresponding to the previous moment respectively and output action function values of the actions at all moments; the local hybrid network is used for nonlinearly adding two action function values of the unmanned aerial vehicle at each moment into a local action value, and the global hybrid network is used for nonlinearly adding the local action values of all the unmanned aerial vehicles into a combined action value; initializing an unmanned aerial vehicle cluster network, and obtaining a plurality of training samples according to the mobile sub-agent network and the transmission sub-agent network of each unmanned aerial vehicle; and training the topology reconstruction model based on the training sample to obtain a trained topology reconstruction model. And reasonable topology reconstruction is realized.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle communication technology, and in particular to a method and system for training an unmanned aerial vehicle cluster topology reconstruction model. Background Art

[0002] With the rapid development of drone technology, drones have been widely used in agricultural plant protection, express delivery, disaster relief and other fields due to their strong adaptability, low cost and flexible configuration. However, the complex and harsh environment often causes drone nodes to fail, which not only affects the communication of the drone network, but also divides the network into multiple disconnected partitions, affecting the data exchange between nodes in the network, and even causing the drone network to paralyze, seriously affecting the coordination ability of the entire drone cluster.

[0003] The existing topology reconstruction problem is mainly concentrated in the field of wireless sensors. Due to energy exhaustion and the influence of harsh environment, sensor nodes are prone to failure, resulting in network disconnection. According to the scale of network damage, they are divided into large-scale fault repair algorithms and small-scale fault repair algorithms. In small-scale fault repair algorithms, they can be divided into two categories: algorithms that distinguish node importance and algorithms that do not distinguish node importance. Both of them repair the network by changing the position of existing drones. The former focuses on restoring network coverage as soon as possible but increases information transmission overhead, while the latter focuses on repairing network connectivity but reduces efficiency. In large-scale fault repair algorithms, network repair is mainly completed by adding relay nodes. A lot of research focuses on how to reduce the number of relay nodes and how to improve the robustness of the network after reconnection.

[0004] At present, there are differences in application requirements and scenarios between drone cluster communication networks and sensor networks. The dynamic topology of drone networks and the coordination between machines make it difficult to directly apply the topology repair method of wireless sensor networks to drone cluster communication networks. Considering the large scale of the solution space caused by the dynamic topology of drones, the complexity and numerous solutions, and the difficulty of solving them with traditional heuristic algorithms, it is particularly important to build and train a drone topology reconstruction model so that undamaged drones can make reasonable topology reconstructions in a short time while ensuring topology repair and selecting appropriate next-hop nodes. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method and system for training a drone cluster topology reconstruction model, so as to solve the problem that the existing drone topology reconstruction model has a large action space and results in low rationality of topology reconstruction.

[0006] On the one hand, an embodiment of the present invention provides a method for training a drone cluster topology reconstruction model, comprising the following steps:

[0007] A topology reconstruction model is constructed, and the topology reconstruction model includes: multiple proxy networks and local hybrid networks, and a global hybrid network in sequence; wherein each proxy network includes a mobile sub-proxy network corresponding to a mobile action and a transmission sub-proxy network corresponding to a data packet transmission selection action, which are respectively used to receive the local state information of the drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment, the local hybrid network is used to nonlinearly add the two action function values ​​of the drone at each moment into a local action value, and the global hybrid network is used to nonlinearly add the local action values ​​of all drones into a joint action value;

[0008] Initialize the drone cluster network and obtain multiple training samples based on the mobile sub-agent network and transmission sub-agent network of each drone;

[0009] The topology reconstruction model is trained based on the training samples to obtain a trained topology reconstruction model.

[0010] Based on the further improvement of the above method, the drone cluster network is initialized, including:

[0011] Set each UAV to fly at the same altitude and uniform speed in a square area, and the ground base station is the transmission destination of the data packet; initialize the local state information of each UAV, including the UAV position, queue capacity, energy, data packets to be transmitted and neighbor nodes; the experience pool is initialized to empty.

[0012] Based on the further improvement of the above method, multiple training samples are obtained according to the mobile sub-agent network and transmission sub-agent network of each drone, including:

[0013] In chronological order, the local state information of each drone at each moment is collected, and the mobile actions and data packet transmission selection actions selected by the mobile sub-agent network and the transmission sub-agent network at the previous moment are respectively transmitted to the corresponding mobile sub-agent network and the transmission sub-agent network, and the mobile actions and data packet transmission selection actions at each moment are obtained according to the ε-greedy strategy; each drone performs the action at each moment, calculates the reward value according to the reward function, and collects the corresponding local state information at the next moment to obtain the action at the next moment until the preset time period is reached;

[0014] The local state information, moving action, data packet transmission selection action, reward value and the corresponding local state information at the next moment are stored in the experience pool as a record; the records of all drones at the same moment constitute a data set, and multiple data sets are randomly sampled from the experience pool to obtain training samples.

[0015] Based on the further improvement of the above method, the loss function of the topology reconstruction model is the sum of squares of the joint action value of each training sample at each moment and the corresponding target joint action value at the next moment; the target joint action value is obtained by the following method:

[0016] Construct a target model with the same structure as the topology reconstruction model, obtain the local state information of each UAV at the next moment and the corresponding action selected at each moment in each training sample, and pass them into the mobile sub-agent network and transmission sub-agent network in the target model, respectively take out the maximum action function value and pass it into the local hybrid network corresponding to each UAV in the target model, and then pass the local action value of each UAV into the global hybrid network in the target model to obtain the maximum joint action value; according to the maximum joint action value, reward value and attenuation factor of each training sample, the target joint action value of each training sample is obtained.

[0017] Based on the further improvement of the above method, the parameters of the local hybrid network and the global hybrid network are generated by a super network, wherein the input of the super network of the local hybrid network is the local state information of the corresponding UAV, and the input of the super network of the global hybrid network is the global state information of the UAV cluster.

[0018] Based on the further improvement of the above method, the moving actions include: eastward, southward, westward, northward and hovering, and the data packet transmission selection action is to select the transmission node of the next hop of the data packet from the neighboring nodes within the communication range of the drone.

[0019] Based on the further improvement of the above method, the reward value is calculated according to the reward function, including:

[0020] According to the data packet transmission selection action at each moment and the corresponding previous moment, the data packet transmission time of each hop is calculated, and the inverse of the data packet transmission time is taken as the data packet transmission reward value; if the constraint condition is violated in each hop of data transmission, the preset negative reward value corresponding to the violated constraint condition is taken as the constraint condition reward value; if the transmission node of the next hop of the data packet is a ground base station, the preset positive reward value of the data packet transmission to the ground base station is taken as the transmission end reward value;

[0021] According to the movement action at each moment, take the preset negative reward value corresponding to the selected movement action as the reconstruction cost reward value;

[0022] The reward value is obtained by summing up the packet transmission reward value, the constraint condition reward value, the transmission end reward value and the reconstruction cost reward value.

[0023] Based on the further improvement of the above method, the constraints include: the position spacing of the UAV between adjacent moments is the product of the flight speed and the time interval; the transmission node of the next hop of the data packet is within the communication range of the UAV; the remaining queue capacity of the transmission node of the next hop of the data packet is greater than or equal to the size of the data packet; the sum of the energy consumed by the UAV in performing actions at each moment within the time period is less than or equal to the total energy of the UAV; the position of the UAV at each moment is within the preset flight area.

[0024] Based on the further improvement of the above method, according to the data packet transmission selection action at each moment and the corresponding previous moment, the data packet transmission time of each hop is calculated, including:

[0025] According to the data packet transmission selection action at each moment and the corresponding previous moment, the transmission distance and transmission node type of each hop data packet are obtained;

[0026] Calculate the path loss of the direct link based on the transmission node type and transmission distance;

[0027] The signal-to-noise ratio of the direct link is calculated based on the path loss value, the UAV transmission power and the noise power;

[0028] The transmission rate of the direct link is calculated based on the signal-to-noise ratio and the bandwidth of the drone;

[0029] The packet transmission time is calculated based on the size and transmission rate of the packet at each hop.

[0030] On the other hand, an embodiment of the present invention provides a drone cluster topology reconstruction model training system, including:

[0031] A model building module is used to build a topology reconstruction model, which includes: multiple proxy networks and local hybrid networks, and a global hybrid network in sequence; wherein each proxy network includes a mobile sub-proxy network corresponding to a mobile action and a transmission sub-proxy network corresponding to a data packet transmission selection action, which are respectively used to receive the local state information of the drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment, the local hybrid network is used to nonlinearly add the two action function values ​​of the drone at each moment into a local action value, and the global hybrid network is used to nonlinearly add the local action values ​​of all drones into a joint action value;

[0032] The sample extraction module is used to initialize the UAV cluster network and obtain multiple training samples according to the mobile sub-agent network and transmission sub-agent network of each UAV;

[0033] The model training module is used to train the topology reconstruction model based on the training samples to obtain the trained topology reconstruction model.

[0034] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: for the scenario of emergency rescue of drone cluster communication network, the drone movement characteristics and communication task requirements are comprehensively considered, and based on the multi-agent reinforcement learning algorithm, the drone's agent network only needs local information, reducing the problem search space; the divide-and-conquer idea is used to decompose the agent network of each drone into two sub-agents to reduce the action space, one is responsible for the drone movement decision, and the other is responsible for the selection decision of the data packet transmission node, so as to better realize the collaboration between drones; with minimizing the transmission time of the data packet as the optimization goal, the drone's action reward is calculated from four aspects: data packet transmission time, constraints, reconstruction cost and transmission end, and the drone task requirements are considered while repairing the topology, so as to further optimize the communication network performance, reduce the impact of topology damage on drone collaborative communication, and improve the rationality of topology reconstruction.

[0035] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0037] Figure 1 This is a flow chart of a method for training a drone cluster topology reconstruction model in Example 1 of the present invention;

[0038] Figure 2 This is a schematic diagram of the structure of the topology reconstruction model in Example 1 of the present invention;

[0039] Figure 3 This is a schematic diagram of a drone cluster network in Example 1 of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of a drone cluster topology reconstruction model training system in Example 2 of the present invention. DETAILED DESCRIPTION

[0041] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0042] Example 1

[0043] A specific embodiment of the present invention discloses a method for training a drone cluster topology reconstruction model based on a multi-hop drone network for long-distance information transmission needs, such as Figure 1 As shown, the following steps are included:

[0044] S11. Construct a topology reconstruction model, which includes: multiple agent networks and local hybrid networks, and a global hybrid network in sequence; wherein each agent network includes a mobile sub-agent network corresponding to the mobile action and a transmission sub-agent network corresponding to the data packet transmission selection action, which are respectively used to receive the local state information of the drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment, the local hybrid network is used to nonlinearly add the two action function values ​​of the drone at each moment into a local action value, and the global hybrid network is used to nonlinearly add the local action values ​​of all drones into a joint action value.

[0045] It should be noted that, considering the need for drone clusters to complete tasks through collaborative communication, it is necessary not only to complete the repair of the drone cluster communication network in a short time, but also to ensure the collaborative ability of the drone communication network in the event of node failure. That is, drones must make decisions in a short time that can both ensure topology repair and select appropriate next-hop nodes to improve communication efficiency, which leads to an increase in the action space. This embodiment uses the idea of ​​divide and conquer to decompose the actions of drones into mobile actions and data packet transmission selection actions. Correspondingly, the proxy network of each drone includes a mobile sub-agent network corresponding to the mobile action and a transmission sub-agent network corresponding to the data packet transmission selection action, which are used to select the movement direction of the drone and the transmission node of the next hop of the data packet, respectively, solving the problem of large drone action space and inter-machine collaboration, and obtaining a reasonable drone cluster topology reconstruction model.

[0046] Specifically, the movement actions of the drone include: east, south, west, north and hovering, and the data packet transmission selection action is to select the next hop transmission node of the data packet from the neighboring nodes within the communication range of the drone.

[0047] The structural diagram of the topology reconstruction model is as follows: Figure 2 As shown in the figure, the two sub-agent networks of each drone are independent, and the parameters of the same type of sub-agent networks between different drones can be shared. The sub-agent network structure includes three layers in sequence: the input layer is a multi-layer perceptron neural network MLP, the middle layer is a gated recurrent neural network GRU, and the output layer is a multi-layer perceptron neural network MLP. It selects actions under the ε-greedy strategy and has the same reward function, which is used to calculate the reward value obtained by each drone after executing the action.

[0048] The local hybrid network receives the two action function values ​​output by the mobile sub-agent network and the transmission sub-agent network corresponding to each drone, and then monotonically mixes and nonlinearly adds them to form the local action value; the local action values ​​of all drones are added to form the joint action value through the global hybrid network. The parameters of the local hybrid network and the global hybrid network are generated by the hypernetwork, but the input of the hypernetwork of the local hybrid network is the local state information of the corresponding drone, and the input of the hypernetwork of the global hybrid network is the global state information of the drone cluster. The hypernetwork for generating weights consists of a single linear layer and an absolute value activation function in turn, ensuring that the weights of the local and global hybrid networks are non-negative. The bias is also generated in the same way, but the hypernetwork for generating biases does not have an absolute value activation function and is generated by a 2-layer hypernetwork with ReLU nonlinearity.

[0049] It should be noted that the local status information includes: the location of the current UAV and the location of its neighboring nodes within its communication range, the remaining queue capacity of the current UAV and the remaining queue capacity of its neighboring nodes within its communication range, the remaining energy of the current UAV and its neighboring nodes, and the size of the data packets to be transmitted by the current UAV and its neighboring nodes; the global status information includes: the location of all UAVs, the remaining queue capacity, the remaining energy and the size of the data packets to be transmitted.

[0050] S12, initializing the drone cluster network, and obtaining multiple training samples according to the mobile sub-agent network and the transmission sub-agent network of each drone.

[0051] It should be noted that initializing the drone cluster network includes:

[0052] Set each drone to fly at the same altitude and uniform speed in a square area, and the ground base station is the transmission destination of the data packet. Figure 3 As shown, the drone cluster network includes K drones, denoted as The drone is on a side with a length of L s km square area, all drones fly at an altitude of H and a uniform flight speed of V. The drones hover near the ground base station and establish communication with the ground base station, which is the transmission destination of the data packet.

[0053] Initialize the local state information of each drone, including the drone position, queue capacity, energy, data packets to be transmitted, and neighbor nodes; the experience pool is initialized to empty. It should be noted that the experience pool is a global cache area.

[0054] The status information of each UAV is collected at time intervals within the preset time period, and the time interval is the length of the time slot δ t , the number of time slots is N, then the flight trajectory of the UAV, that is, the set of flight positions of each UAV is expressed as where ωk [n] represents the position of drone k in time slot n.

[0055] Furthermore, according to the mobile sub-agent network and the transmission sub-agent network of each UAV, multiple training samples are obtained, including:

[0056] In chronological order, the local state information of each drone at each moment is collected, and the mobile actions and data packet transmission selection actions selected by the mobile sub-agent network and the transmission sub-agent network at the previous moment are respectively transmitted to the corresponding mobile sub-agent network and the transmission sub-agent network, and the mobile actions and data packet transmission selection actions at each moment are obtained according to the ε-greedy strategy; each drone performs the action at each moment, calculates the reward value according to the reward function, and collects the corresponding local state information at the next moment to obtain the action at the next moment until the preset time period is reached;

[0057] The local state information, moving action, data packet transmission selection action, reward value and the corresponding local state information at the next moment are stored in the experience pool as a record; the records of all drones at the same moment constitute a data set, and multiple data sets are randomly sampled from the experience pool to obtain training samples.

[0058] Specifically, in the UAV cluster network, the record of UAV k at time t includes: local state information Move Action Packet transmission action Reward value r k and the local state information at the next moment Each data set includes K records. If the drone has no data packets to be transmitted, the next hop transmission node of the data packet selected by the data packet transmission action is the node where the drone itself is located.

[0059] It should be noted that the reward value is calculated according to the reward function, including:

[0060] According to the data packet transmission selection action at each moment and the corresponding previous moment, the data packet transmission time of each hop is calculated, and the inverse of the data packet transmission time is taken as the data packet transmission reward value; if the constraint condition is violated in each hop of data transmission, the preset negative reward value corresponding to the violated constraint condition is taken as the constraint condition reward value; if the transmission node of the next hop of the data packet is a ground base station, the preset positive reward value of the data packet transmission to the ground base station is taken as the transmission end reward value;

[0061] According to the movement action at each moment, take the preset negative reward value corresponding to the selected movement action as the reconstruction cost reward value;

[0062] The reward value is obtained by summing up the packet transmission reward value, the constraint condition reward value, the transmission end reward value and the reconstruction cost reward value.

[0063] Furthermore, the packet transmission time is optimized to minimize the packet transmission time, which is expressed as follows:

[0064]

[0065] in, Indicates that the data packet is from node k m To node k m+1 The transmission time of this hop, M represents the total number of hops of the data packet transmission path ξ.

[0066] The established optimization objectives also satisfy the following constraints:

[0067] 1) The distance between the positions of the drones at adjacent moments is the product of the flight speed and the time interval:

[0068] ||ω k [n+1]-ω k [n]||=Vδt Formula (2)

[0069] 2) The next hop transmission node of the data packet is within the communication range of the drone:

[0070]

[0071] Where d represents the preset UAV communication range, and Respectively represent at node k m and node k m+1 The position of the drone in time slot n;

[0072] 3) The remaining queue capacity of the next-hop transmission node of the data packet is greater than or equal to the size of the data packet to avoid network congestion:

[0073]

[0074] in, Indicates that at the next hop node k m+1 The remaining queue capacity of the drone at position n in time slot n, μ represents the packet size;

[0075] 4) The sum of the energy consumed by the drone to perform actions at each moment in the time period is less than or equal to the total energy of the drone:

[0076]

[0077] in, Indicates that at node k mThe energy consumed by the drone at position n to perform actions in time slot n, E max represents the total energy of the drone;

[0078] 5) The position of the drone at each moment is within the preset flight area:

[0079] φ l ≤ω k [n]≤φ u Formula (6)

[0080] Among them, φ l and φ u They represent the upper and lower boundary coordinates of the preset flight area, that is, the side length is L s The boundary coordinates of the square area of ​​km in the actual scene.

[0081] Each of the above constraints corresponds to a preset negative reward value. If any constraint is violated, a penalty will be given according to the preset negative reward value. According to the data packet transmission at each moment, the action is selected. If the next hop transmission node of the data packet is a ground base station, it means that the data packet is successfully transmitted to the ground base station, and a reward is given according to the preset positive reward value. When the mobile action selected by the mobile subagent network is non-hovering, a penalty is given according to the preset negative reward value.

[0082] For example, in one-hop transmission, the packet transmission time is 10, and the packet transmission reward value is -10. However, the movement of the UAV exceeds the flight area, violating the fifth constraint condition, the preset negative reward value of which is -5. The selected moving action is south, and the preset negative reward value is -2. Because the data packet is not transmitted to the ground base station, no positive reward value is obtained. The reward value obtained by the UAV after performing the action is -17.

[0083] Furthermore, in formula (1), the data packet is sent from node k m To node k m+1 The transmission time of this hop Calculate by following the steps below:

[0084] According to the data packet transmission selection action at each moment and the corresponding previous moment, the transmission distance and transmission node type of each hop data packet are obtained;

[0085] Calculate the path loss of the direct link based on the transmission node type and transmission distance;

[0086] The signal-to-noise ratio of the direct link is calculated based on the path loss value, the UAV transmission power and the noise power;

[0087] The transmission rate of the direct link is calculated based on the signal-to-noise ratio and the bandwidth of the drone;

[0088] The packet transmission time is calculated based on the size and transmission rate of the packet at each hop.

[0089] Specifically, the path loss of the direct link is calculated based on the transmission node type and transmission distance, including:

[0090] The transmission distance of each hop data packet is calculated by formula (7); if the transmission node type of the next hop of the data packet is a ground base station, the line-of-sight link probability is calculated by formula (8) according to the environment, drone altitude and transmission distance. The probability of non-line-of-sight link is calculated by formula (9): Based on the transmission distance, carrier frequency, speed of light and the additional loss of non-line-of-sight links, the path loss of the line-of-sight link is calculated using formula (10): The path loss of the non-line-of-sight link is calculated by formula (11): According to the probability and path loss of line-of-sight link and non-line-of-sight link, the path loss of direct link is calculated by formula (12):

[0091]

[0092]

[0093]

[0094] Among them, H represents the flight altitude of the drone, Represents node k m UAV and node k m+1 The distance between the base station / drone at time slot n, i.e., the data packet from node k m To node k m+1 The transmission distance of this hop, η1 and η2 represent probability parameters, which depend on the flight environment. For example, in a suburban environment, η1 is set to 4.88 and η2 is set to 0.43;

[0095]

[0096]

[0097]

[0098] Among them, f c represents the carrier frequency of the UAV communication system, c represents the speed of light, ψ NLOS Indicates the additional loss of the non-line-of-sight link. Its value depends on the environment. Exemplarily, it is set to 20 dB.

[0099] If the next-hop transmission node type of the data packet is a drone, the path loss of the line-of-sight link is calculated by formula (10) based on the transmission distance, carrier frequency and speed of light, and is used as the path loss of the direct transmission link.

[0100] Furthermore, according to the path loss value, the UAV transmission power and the noise power, the signal-to-noise ratio of the direct transmission link is calculated by formula (13):

[0101]

[0102] in, Indicates that at node k m The transmission power of the UAV at the location, σ 2 Represents the noise power.

[0103] Furthermore, according to the signal-to-noise ratio and the bandwidth of the UAV, the transmission rate of the direct link is calculated by formula (14):

[0104]

[0105] in, Indicates that at node k m The bandwidth of the drone at the location.

[0106] Finally, according to the size and transmission rate of each hop data packet, the data packet transmission time is calculated by formula (15):

[0107]

[0108] Where μ represents the size of the data packet.

[0109] Specifically, in formula (5) Calculated by the following formula:

[0110]

[0111] c1=2v0 2 Formula (17)

[0112]

[0113] Among them, P0 represents the blade profile power when the drone is in the hovering state, P1 represents the induced power when the drone is in the hovering state, and U tip represents the tip speed of the UAV rotor blade, v0 represents the average rotation induced speed of the UAV, d0 represents the fuselage drag ratio, ρ is the air density, s0 represents the rotation stability, and A represents the rotor rotation area.

[0114] Compared with the prior art, this embodiment takes the packet transmission time as the optimization target, and calculates the drone's action reward from four aspects: packet transmission time, constraints, reconstruction cost, and transmission completion. This achieves the consideration of the drone's mission requirements while repairing the topology, further improving the rationality of the topology reconstruction model.

[0115] S13. Train a topology reconstruction model based on the training samples to obtain a trained topology reconstruction model.

[0116] It should be noted that the loss function of the topology reconstruction model is the sum of squares of the errors between the joint action value of each training sample at each moment and the target joint action value at the next moment.

[0117] Specifically, according to the K records of each randomly selected training sample and the records in the experience pool, the local state information o corresponding to each record in the training sample at the current moment is obtained. t and the previous action (a t-1,h and a t-1,d ), the corresponding mobile subagent network and transmission subagent network in the input topology reconstruction model (o t and a t-1,h Incoming transport sub-agent network, o t and a t-1,d The two actions of each record are selected through the ε-greedy strategy to obtain the two action function values ​​of each record, and the two action function values ​​of each record are passed into the corresponding local hybrid network in the topology reconstruction model to obtain the local action value of each record, which is then passed into the global hybrid network in the topology reconstruction model to obtain the joint action value Q of each training sample. tot (o,a,s;θ); where o represents the set of local state information of the drone in the current state, a represents the set of drone actions in the current state, s represents the global state information in the current state, and θ represents the parameters of the topology reconstruction model.

[0118] It should be noted that the target joint action value is obtained in the following way:

[0119] Construct a target model with the same structure as the topology reconstruction model to keep the target value stable and prevent overfitting, thereby improving the stability and convergence speed of the training process;

[0120] The local state information of each UAV at the next moment and the corresponding action selected at each moment in each training sample are obtained, and the information is passed into the mobile sub-agent network and the transmission sub-agent network in the target model. The maximum action function value is taken out and passed into the local hybrid network corresponding to each UAV in the target model. Then, the local action value of each UAV is passed into the global hybrid network in the target model to obtain the maximum joint action value. According to the maximum joint action value, reward value and attenuation factor of each training sample, the target joint action value of each training sample is obtained.

[0121] Specifically, obtain the local state information o of each record in the training sample at the next moment t+1 and the current action (a t,h and a t,d ), and the corresponding mobile sub-agent network and transmission sub-agent network in the target model (o t+1 and a t,h Incoming transport sub-agent network, o t+1 and a t,d The two maximum action function values ​​of each record are obtained, and the two maximum action function values ​​of each record are passed into the corresponding local hybrid network in the target model to obtain the maximum local action value of each record, and then passed into the global hybrid network in the target model to obtain the maximum joint action value of each training sample. Among them, o' represents the set of local state information of the drone in the next state, a' represents the set of drone actions in the next state, s' represents the global state information in the next state, θ - Represents the parameters of the target model.

[0122] According to the maximum joint action value of each training sample, the reward value of the corresponding training sample and the attenuation factor, the target joint action value of each training sample is obtained by the following formula:

[0123]

[0124] in, represents the target joint action value of the i-th training sample, r i represents the average value of the reward value of each record in the i-th training sample, γ represents the decay factor, Represents the maximum joint action value of the i-th training sample.

[0125] According to the joint action value of the training sample and the target joint action value, the error function is calculated by the following formula:

[0126]

[0127] Where b represents the total number of training samples. Represents the joint action value of the i-th training sample.

[0128] During iterative training, based on the calculated error function, the parameters of the topology reconstruction model are updated based on the gradient descent method in back propagation, and the parameters of the topology reconstruction model are periodically copied to the target model; when the maximum number of iterations is reached or the preset accuracy is reached, the trained topology reconstruction model is obtained.

[0129] Compared with the prior art, the present embodiment provides a method and system for training a drone cluster topology reconstruction model. Aiming at the scenario of emergency rescue of drone cluster communication network, the present embodiment comprehensively considers the movement characteristics of drones and the communication task requirements, and based on the multi-agent reinforcement learning algorithm, the drone's agent network only needs local information, reducing the problem search space; the divide-and-conquer idea is used to decompose the agent network of each drone into two sub-agents to reduce the action space, one is responsible for the drone movement decision, and the other is responsible for the selection decision of the data packet transmission node, so as to better realize the collaboration between drones; with minimizing the transmission time of the data packet as the optimization goal, the drone's action reward is calculated from four aspects: data packet transmission time, constraints, reconstruction cost and transmission end, and the drone mission requirements are considered while repairing the topology, so as to further optimize the communication network performance, reduce the impact of topology damage on drone collaborative communication, and improve the rationality of topology reconstruction.

[0130] Example 2

[0131] Another embodiment of the present invention discloses a drone cluster topology reconstruction model training system, thereby implementing the drone cluster topology reconstruction model training method in Example 1. The specific implementation of each module refers to the corresponding description in Example 1. Figure 4 As shown, the system includes:

[0132] The model building module 101 is used to build a topology reconstruction model, which includes: multiple proxy networks and local hybrid networks, and a global hybrid network in sequence; wherein each proxy network includes a mobile sub-proxy network corresponding to a mobile action and a transmission sub-proxy network corresponding to a data packet transmission selection action, which are respectively used to receive the local state information of the drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment, the local hybrid network is used to nonlinearly add the two action function values ​​of the drone at each moment into a local action value, and the global hybrid network is used to nonlinearly add the local action values ​​of all drones into a joint action value;

[0133] The sample extraction module 102 is used to initialize the drone cluster network and obtain multiple training samples according to the mobile sub-agent network and the transmission sub-agent network of each drone;

[0134] The model training module 103 is used to train the topology reconstruction model based on the training samples to obtain a trained topology reconstruction model.

[0135] Since the present embodiment and the aforementioned method for training a drone cluster topology reconstruction model can be mutually referenced, this is a repeated description, so it will not be repeated here. Since the present system embodiment and the aforementioned method embodiment have the same principle, the present system embodiment also has the corresponding technical effects of the aforementioned method embodiment.

[0136] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0137] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for training a drone cluster topology reconstruction model, characterized in that: The following steps are involved: A topology reconstruction model is constructed, and the topology reconstruction model includes: multiple proxy networks and local hybrid networks, and a global hybrid network in sequence; wherein each proxy network includes a mobile sub-proxy network corresponding to a mobile action and a transmission sub-proxy network corresponding to a data packet transmission selection action, which are respectively used to receive the local state information of the drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment, the local hybrid network is used to nonlinearly add the two action function values ​​of the drone at each moment into a local action value, and the global hybrid network is used to nonlinearly add the local action values ​​of all drones into a joint action value; Initialize the drone cluster network and obtain multiple training samples based on the mobile sub-agent network and transmission sub-agent network of each drone; The topology reconstruction model is trained based on the training samples to obtain a trained topology reconstruction model.

2. The method for training a drone cluster topology reconstruction model according to claim 1, characterized in that: The initialization of the drone cluster network includes: Set each UAV to fly at the same altitude and uniform speed in a square area, and the ground base station is the transmission destination of the data packet; initialize the local state information of each UAV, including the UAV position, queue capacity, energy, data packets to be transmitted and neighbor nodes; the experience pool is initialized to empty.

3. The method for training a drone cluster topology reconstruction model according to claim 2, characterized in that: According to the mobile sub-agent network and the transmission sub-agent network of each drone, a plurality of training samples are obtained, including: In chronological order, the local state information of each drone at each moment is collected, and the mobile actions and data packet transmission selection actions selected by the mobile sub-agent network and the transmission sub-agent network at the previous moment are respectively transmitted to the corresponding mobile sub-agent network and the transmission sub-agent network, and the mobile actions and data packet transmission selection actions at each moment are obtained according to the ε-greedy strategy; each drone performs the action at each moment, calculates the reward value according to the reward function, and collects the corresponding local state information at the next moment to obtain the action at the next moment until the preset time period is reached; The local state information, moving action, data packet transmission selection action, reward value and the corresponding local state information at the next moment are stored in the experience pool as a record; the records of all drones at the same moment constitute a data set, and multiple data sets are randomly sampled from the experience pool to obtain training samples.

4. The method for training a drone cluster topology reconstruction model according to claim 3 is characterized in that: The loss function of the topology reconstruction model is the sum of squares of the joint action value of each training sample at each moment and the target joint action value at the next moment; the target joint action value is obtained by: Construct a target model with the same structure as the topology reconstruction model, obtain the local state information of each drone in each training sample at the next moment and the corresponding action selected at each moment, and pass them into the mobile sub-agent network and transmission sub-agent network in the target model, respectively take out the maximum action function value and pass it into the local hybrid network corresponding to each drone in the target model, and then pass the local action value of each drone into the global hybrid network in the target model to obtain the maximum joint action value; According to the maximum joint action value, reward value and attenuation factor of each training sample, the target joint action value of each training sample is obtained.

5. The method for training a drone cluster topology reconstruction model according to claim 1 or 4, characterized in that: The parameters of the local hybrid network and the global hybrid network are generated by a super network, wherein the input of the super network of the local hybrid network is the local state information of the corresponding drone, and the input of the super network of the global hybrid network is the global state information of the drone cluster.

6. The method for training a drone cluster topology reconstruction model according to claim 3, characterized in that: The moving actions include: eastward, southward, westward, northward and hovering, and the data packet transmission selection action is to select the transmission node of the next hop of the data packet from the neighboring nodes within the communication range of the UAV.

7. The method for training a drone cluster topology reconstruction model according to claim 6, characterized in that: The step of calculating the reward value according to the reward function includes: According to the data packet transmission selection action at each moment and the corresponding previous moment, the data packet transmission time of each hop is calculated, and the inverse of the data packet transmission time is taken as the data packet transmission reward value; if the constraint condition is violated in each hop of data transmission, the preset negative reward value corresponding to the violated constraint condition is taken as the constraint condition reward value; if the transmission node of the next hop of the data packet is a ground base station, the preset positive reward value of the data packet transmission to the ground base station is taken as the transmission end reward value; According to the movement action at each moment, take the preset negative reward value corresponding to the selected movement action as the reconstruction cost reward value; The reward value is obtained by summing up the packet transmission reward value, the constraint condition reward value, the transmission end reward value and the reconstruction cost reward value.

8. The method for training a drone cluster topology reconstruction model according to claim 7, characterized in that: The constraints include: the distance between the positions of the drones at adjacent moments is the product of the flight speed and the time interval; the transmission node of the next hop of the data packet is within the communication range of the drone; the remaining queue capacity of the transmission node of the next hop of the data packet is greater than or equal to the size of the data packet; the sum of the energy consumed by the drone in performing actions at each moment within the time period is less than or equal to the total energy of the drone; the position of the drone at each moment is within the preset flight area.

9. The method for training a drone cluster topology reconstruction model according to claim 7, characterized in that: The step of selecting an action for data packet transmission at each moment and the corresponding last moment to calculate the data packet transmission time at each hop includes: According to the data packet transmission selection action at each moment and the corresponding previous moment, the transmission distance and transmission node type of each hop data packet are obtained; Calculate the path loss of the direct link based on the transmission node type and transmission distance; The signal-to-noise ratio of the direct link is calculated based on the path loss value, the UAV transmission power and the noise power; The transmission rate of the direct link is calculated based on the signal-to-noise ratio and the bandwidth of the drone; The packet transmission time is calculated based on the size and transmission rate of the packet at each hop.

10. A UAV cluster topology reconstruction model training system, characterized in that: include: A model building module is used to build a topology reconstruction model, which includes: multiple proxy networks and local hybrid networks, and a global hybrid network in sequence; wherein each proxy network includes a mobile sub-proxy network corresponding to a mobile action and a transmission sub-proxy network corresponding to a data packet transmission selection action, which are respectively used to receive the local state information of the drone at each moment and the corresponding action at the previous moment and output the action function value of the action at each moment, the local hybrid network is used to nonlinearly add the two action function values ​​of the drone at each moment into a local action value, and the global hybrid network is used to nonlinearly add the local action values ​​of all drones into a joint action value; The sample extraction module is used to initialize the UAV cluster network and obtain multiple training samples according to the mobile sub-agent network and transmission sub-agent network of each UAV; The model training module is used to train the topology reconstruction model based on the training samples to obtain the trained topology reconstruction model.

Images