Unmanned aerial vehicle communication scheduling method and device based on multi-agent reinforcement learning

Abstract

The application provides a kind of unmanned plane communication scheduling method and device based on multi-agent reinforcement learning, it is related to unmanned plane technical field, comprising: obtaining the local input state of the unmanned plane in current time slot, and constructing the one-hop neighbor set of the unmanned plane in the current time slot;The local input state is mapped to coding feature by strategy branch encoder, the coding feature is sent to the neighbor unmanned plane in the one-hop neighbor set, and the coding feature sent by the neighbor unmanned plane in the one-hop neighbor set is received;The received coding feature is aggregated based on attention mechanism, and the neighborhood aggregation message of the current time slot is generated;The local input state and the neighborhood aggregation message are input into shared policy network, and the joint action of the current time slot is obtained;The unmanned plane is controlled to execute the joint action, and the communication scheduling of the current time slot is completed.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV communication scheduling method and apparatus based on multi-agent reinforcement learning. Background Technology

[0002] With the continuous expansion of application scenarios such as logistics delivery, emergency rescue, post-disaster communication, relay coverage, and collaborative inspection, the demand for multiple drones to coordinate missions is increasing. During mission execution, multiple drones need to exchange status information, service data, and control commands under limited wireless spectrum resources. Due to the characteristics of drone nodes, such as high mobility, temporary network deployment, and low dependence on ground infrastructure, multiple drones usually need to establish air-to-air multi-hop communication links in a self-organizing manner. Therefore, the quality of the communication resource scheduling mechanism directly determines the network stability, mission coordination, and service transmission quality.

[0003] Existing UAV communication resource scheduling schemes can be broadly categorized into two types: centralized optimization schemes and distributed learning schemes. Centralized optimization schemes typically rely on a central node to collect network-wide status information and then uniformly solve for resource allocation results.

[0004] However, centralized optimization schemes require each UAV to continuously report its real-time status to the central node. In scenarios with a large number of nodes or high movement speed, the status reporting latency and control signaling overhead will increase rapidly. Furthermore, the overall network status on which the central node is based often changes by the time the transmission is completed, resulting in a mismatch between the scheduling results and the actual topology.

[0005] While distributed learning schemes alleviate the above problems to some extent, they typically assume that the neighborhood relationships between UAVs are approximately stable over a long period of time, making it difficult to adapt to rapid link changes at the time slot scale. At the same time, some distributed learning schemes rely too much on the local independent decision-making of each UAV and lack the ability to exchange information collaboratively with neighboring UAVs, making it easy for adjacent UAVs to have co-channel conflicts and interference coupling when selecting channels and transmission power.

[0006] It is evident that existing resource scheduling schemes for UAV self-organizing communication networks are insufficient to balance real-time performance, collaboration, and adaptability to topology changes in highly dynamic topology scenarios. There is an urgent need for a communication resource scheduling method that can support collaborative scheduling among neighboring UAVs in high-speed, time-varying multi-UAV scenarios without relying on a central node for real-time decision-making. Summary of the Invention

[0007] This invention provides a UAV communication scheduling method and apparatus based on multi-agent reinforcement learning, which solves the problem that the resource scheduling scheme of UAV self-organizing communication network in the prior art is difficult to balance real-time performance, coordination and adaptability to topology changes in highly dynamic topology scenarios.

[0008] This invention provides a UAV communication scheduling method based on multi-agent reinforcement learning, applied to a multi-UAV self-organizing communication network. The method is executed independently by the UAVs in the multi-UAV self-organizing communication network in each time slot, and includes: Obtain the local input state of the UAV in the current time slot, and construct the one-hop neighbor set of the UAV in the current time slot; The local input state is mapped to encoded features by a policy branch encoder, the encoded features are sent to neighbor drones in the one-hop neighbor set, and the encoded features sent by neighbor drones in the one-hop neighbor set are received. The received encoded features are aggregated based on an attention mechanism to generate a neighborhood aggregation message for the current time slot; The local input state and the neighborhood aggregation message are input to the shared policy network to obtain the joint action of the current time slot; wherein, the joint action includes channel selection action, transmit power action and packet scheduling priority action; The drone is controlled to perform the joint action to complete the communication scheduling of the current time slot; wherein, the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

[0009] According to the present invention, a UAV communication scheduling method based on multi-agent reinforcement learning is provided, wherein obtaining the local input state of the UAV in the current time slot includes: The location, velocity, remaining energy, queue status, channel state estimate, and interference measurement of the UAV in the current time slot are collected to obtain the local observation of the current time slot. Extract the temporal aggregation features of the previous time slot from the UAV's local buffer; The local observations of the current time slot and the temporal aggregation features of the previous time slot are used as parallel inputs to form the local input state; wherein, the temporal aggregation features of the previous time slot are the neighborhood temporal summary representation generated and cached on the UAV side in the previous time slot, which is used to supplement the temporal memory.

[0010] According to the present invention, a UAV communication scheduling method based on multi-agent reinforcement learning is provided, wherein constructing the one-hop neighbor set of the UAV in the current time slot includes: The relative distance between the UAV and the detectable node, and the link quality index between the UAV and the detectable node are obtained; wherein, the link quality index includes the signal-to-interference-plus-noise ratio obtained based on pilot estimation; When the relative distance is not greater than a preset distance threshold and the link quality index is not less than a preset link quality threshold, an adjacency relationship for the current time slot is established between the UAV and the detectable node. All nodes that have the adjacency relationship with the drone are included in the one-hop neighbor set.

[0011] According to the present invention, a UAV communication scheduling method based on multi-agent reinforcement learning is provided, wherein the aggregation of the received encoded features based on an attention mechanism to generate a neighborhood aggregation message for the current time slot includes: By implementing the query parameters and key parameters in the policy branch aggregation module of the attention mechanism, the attention weight between the encoded features generated by the UAV itself and the encoded features received from the neighboring UAV is calculated. Based on the attention weights and the value parameters in the policy branch aggregation module, the received encoded features sent by the neighboring drones are weighted and summed to obtain the neighborhood aggregation message of the current time slot.

[0012] According to the present invention, a UAV communication scheduling method based on multi-agent reinforcement learning is provided, wherein controlling the UAV to execute the joint action to complete the communication scheduling of the current time slot includes: Based on the channel selection action, switch the spectrum resources occupied by the UAV data transmission; Based on the aforementioned transmission power action, the transmission power level of the UAV is adjusted to simultaneously change the link transmission reliability, the interference leakage intensity to the one-hop neighbor set, and the local transmission energy consumption. Based on the packet scheduling priority action, the service order of different service queues of the UAV is adjusted; After performing the joint action, the number of successfully transmitted bits, the estimated instantaneous co-channel interference, the transmission energy consumption and service delay statistics of this time slot are recorded as feedback sample data for this time slot.

[0013] According to the present invention, a UAV communication scheduling method based on multi-agent reinforcement learning is provided, the method further comprising: The UAV writes the local input state, the neighborhood aggregation message, the joint action, the instant reward, the local input state of the next time slot, and the constraint statistics composed of the feedback sample data into the UAV's local trajectory buffer to form a trajectory sample; The drone uploads the trajectory samples to the training node so that the training node can update the parameters during the centralized training phase. The UAV receives the updated parameters of the shared policy network from the training node at a preset period, and performs the communication scheduling for subsequent time slots according to the updated parameters of the shared policy network.

[0014] According to the present invention, a UAV communication scheduling method based on multi-agent reinforcement learning is provided, the method further comprising: Receive the trajectory samples uploaded by each UAV in the multi-UAV self-organizing communication network; Based on the local input states, joint actions, and adjacency relationships of all the UAVs in the same time slot, the global state is reconstructed; The global state is feature-aggregated using a critic branch encoder to generate a global aggregated representation. The global state, the global aggregate representation, and the joint action are input into a centralized critic network to obtain a value assessment result. Based on the value assessment results and the final reward feedback generated by the two-layer constraint linkage adjustment mechanism, the parameters of the shared policy network and the parameters of the centralized critic network are updated, and the updated parameters of the shared policy network are distributed to each UAV.

[0015] According to a UAV communication scheduling method based on multi-agent reinforcement learning provided by the present invention, the shared policy network and the centralized critic network are respectively configured with structurally corresponding but parameter-independent message aggregation branches; wherein, the policy message aggregation branch configured by the shared policy network is used to perform weighted aggregation of the encoded features received from the neighboring UAVs during the execution phase of the UAV to generate the neighborhood aggregation message. The critic message aggregation branch configured in the centralized critic network is used to aggregate the global features corresponding to the global state during the centralized training phase of the training node to generate the global aggregate representation. The parameters of the policy message aggregation branch and the parameters of the critic message aggregation branch are trained independently, and the UAV only runs the policy message aggregation branch during the execution phase.

[0016] According to the present invention, a UAV communication scheduling method based on multi-agent reinforcement learning is provided, wherein the step of generating the final reward feedback by the two-layer constraint linkage adjustment mechanism includes: Statistical windows are established for the interference index, latency index, and energy consumption index included in the constraint statistics of the trajectory sample, and the average index value of the target index within the statistical window is calculated; wherein, the target index includes the interference index, the latency index, or the energy consumption index; According to the Lagrange multiplier adjustment layer, when it is determined that the average index value of the target index exceeds the threshold set by the target index, the Lagrange multiplier configured for the target index is increased to obtain the violation penalty item generated by the Lagrange multiplier. Based on the barrier function pre-suppression layer, and the remaining margin between the average index value of the target index and the threshold set by the target index, the risk suppression penalty term of the target index is calculated. The final reward feedback is obtained by subtracting the violation penalty and the risk suppression penalty from the basic reward; wherein the basic reward is obtained based on a weighted combination of the weighted throughput, the latency index, and the energy consumption index of the current time slot.

[0017] According to a multi-agent reinforcement learning-based UAV communication scheduling method provided by the present invention, the step of calculating a risk suppression penalty term for the target indicator based on the margin between the average indicator value of the target indicator and a threshold set for the target indicator, according to a barrier function pre-suppression layer, includes: When the average value of the target indicator exceeds the threshold set by the target indicator in the most recent statistical window, the pre-configured barrier coefficient of the target indicator is maintained or increased. When the average value of the target indicator does not exceed the threshold set by the target indicator within a consecutive preset number of statistical windows, the barrier coefficient is reduced according to a preset attenuation coefficient. Logarithmic operations are performed based on the barrier coefficient and the remaining margin to generate the risk suppression penalty term, so as to pre-suppress the tendency to choose relevant high-risk actions in the shared policy network.

[0018] According to the present invention, a UAV communication scheduling method based on multi-agent reinforcement learning is provided, wherein updating the parameters of the shared policy network includes: After a preset number of time slots, the training node calculates the advantage function based on the value assessment result and the final reward feedback; The parameters of the shared policy network are updated once based on the advantage function and the truncated agent objective function; wherein the truncated agent objective function is used to limit the magnitude of a single policy update; When the preset maximum number of training rounds is reached, or when the parameter change of the shared policy network is lower than a preset threshold for several consecutive rounds, the training node saves the parameters of the shared policy network and sends them to the onboard computing unit of each UAV.

[0019] This invention also provides a UAV communication scheduling device based on multi-agent reinforcement learning, for UAVs deployed in a multi-UAV self-organizing communication network, the device comprising: The observation and acquisition module is used to acquire the local input state of the UAV in the current time slot and construct the one-hop neighbor set of the UAV in the current time slot; The feature encoding and message aggregation module is used to map the local input state into encoded features using a policy branch encoder, send the encoded features to neighboring drones in the one-hop neighbor set, and receive encoded features sent by neighboring drones in the one-hop neighbor set. The aggregation module is used to aggregate the coded features received from the neighboring drones based on an attention mechanism to generate the neighborhood aggregation message for the current time slot. An input module is used to input the local input state and the neighborhood aggregation message into the shared policy network to obtain the joint action of the current time slot; wherein, the joint action includes channel selection action, transmit power action and packet scheduling priority action; The scheduling module is used to control the UAV to execute the joint action and complete the communication scheduling of the current time slot; wherein, the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the UAV communication scheduling method based on multi-agent reinforcement learning as described above.

[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the UAV communication scheduling method based on multi-agent reinforcement learning as described above.

[0022] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the UAV communication scheduling method based on multi-agent reinforcement learning as described above.

[0023] The UAV communication scheduling method and apparatus based on multi-agent reinforcement learning provided by this invention enables each UAV to independently construct a one-hop neighbor set in each time slot and complete the encoding of its local input state and feature exchange with neighbors, so that the neighborhood aggregation message can reflect the dynamic changes in link relationships in a timely manner. At the same time, each UAV can output a joint action including channel selection, transmission power and data packet scheduling priority in one go through the shared policy network based only on its local input state and neighborhood aggregation message. This explicitly handles the coupling relationship between the three types of decisions, and no central node is required to participate in real-time scheduling during the execution phase. Thus, in the scenario of highly dynamic topology and time-varying links in multi-UAV self-organizing communication networks, it effectively improves the adaptability of communication resource scheduling to neighborhood changes and the efficiency of collaborative decision-making among neighboring UAVs, and reduces the adverse effects of state reporting latency and control overhead on the real-time performance of scheduling results. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 is a schematic diagram of the system structure applied to the UAV communication scheduling method based on multi-agent reinforcement learning provided by the present invention; Figure 2 This is a flowchart illustrating the UAV communication scheduling method based on multi-agent reinforcement learning provided by the present invention. Figure 3 is a schematic diagram of the dynamic neighborhood graph construction and neighborhood message aggregation process; Figure 4 A schematic diagram of the overall process of the distributed channel access and resource scheduling method for UAV self-organizing communication networks based on multi-agent reinforcement learning provided by the present invention; Figure 5 is a schematic diagram of the overall structure of the distributed channel access and resource scheduling device for UAV self-organizing communication network based on multi-agent reinforcement learning provided by the present invention. Figure 6 A schematic diagram of the UAV communication scheduling device based on multi-agent reinforcement learning provided by the present invention; Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The UAV communication scheduling method based on multi-agent reinforcement learning provided in this invention may have execution entities including, but not limited to, the UAV's onboard computing unit, onboard controller, edge computing node, or ground training server. For ease of explanation, the onboard computing unit of a single UAV is used as an example to describe the method in the following embodiments, but this does not constitute a limitation of the invention.

[0028] For ease of understanding, the following example uses a temporary multi-drone network in an emergency search and rescue scenario: 20 drones are deployed in a mission area, and the wireless spectrum is divided into 4 equal-bandwidth channels. Each drone's radio frequency front-end supports three transmission power levels: low, medium, and high. Each drone simultaneously carries three types of services: control commands, status feedback, and mission data, with control commands having the highest priority and mission data having the lowest priority. The system's running time is divided into equal-length time slots, and each drone performs a joint scheduling decision once in each time slot.

[0029] Figure 1 is a schematic diagram of the system structure of the UAV communication scheduling method based on multi-agent reinforcement learning provided by the present invention. As shown in Figure 1, the multi-UAV self-organizing communication network includes N UAVs and K available communication channels. Each UAV acts as a scheduling agent to perform a joint scheduling decision in each time slot.

[0030] Figure 2 This is a flowchart illustrating the UAV communication scheduling method based on multi-agent reinforcement learning provided by the present invention. The method provided in this embodiment is executed independently by each UAV in the self-organizing communication network in each time slot, such as... Figure 2 As shown, the method includes the following: Step S110: Obtain the local input state of the drone in the current time slot and construct the one-hop neighbor set of the drone in the current time slot.

[0031] The local input status in this step refers to the set of input signals that the UAV's onboard computing unit can directly obtain or read from its local cache in the current time slot. It can at least reflect the UAV's operating status, current service load, and current wireless channel environment.

[0032] In one possible implementation, the local input state can be obtained by the UAV's onboard computing unit through its own sensors, communication modules, and cache. The specific content and form involved can be further defined in subsequent embodiments.

[0033] A one-hop neighbor set refers to the set of other drones that can establish a direct communication relationship with this drone in the current time slot.

[0034] In one feasible approach, the UAV's onboard computing unit identifies neighboring nodes with direct communication relationships in the current time slot based on one or more criteria that determine whether the detectable nodes and the UAV meet the communication conditions, and then categorizes them into a one-hop neighbor set.

[0035] The discrimination criteria can be any one or a combination of distance-based discrimination criteria, link reachability-based discrimination criteria, and link quality-based discrimination criteria.

[0036] Step S120: Map the local input state to encoded features through the policy branch encoder, send the encoded features to the neighbor drones in the one-hop neighbor set, and receive the encoded features sent by the neighbor drones in the one-hop neighbor set.

[0037] Among them, the strategy branch encoder is a feature encoding module pre-configured in the UAV's onboard computing unit. Its structure can adopt any one or a combination of neural network structures with feature compression capabilities, such as multilayer perceptron, convolutional neural network, and gated recurrent unit.

[0038] The function of the policy branch encoder is to map high-dimensional, heterogeneous local input states into low-dimensional encoded features of a unified dimension, so as to control communication overhead when exchanging these encoded features between nodes through communication channels.

[0039] It should be noted that in this step, the UAVs exchange encoded and compressed feature vectors with each other, instead of exchanging the original perception data or the parameters of the shared policy network itself. This can both support collaborative decision-making among neighbors and reduce the burden of air interface communication.

[0040] Step S130: Aggregate the received encoded features based on the attention mechanism to generate a neighborhood aggregation message for the current time slot.

[0041] Figure 3 is a schematic diagram of the dynamic neighborhood graph construction and neighborhood message aggregation process, illustrating the process of reconstructing adjacency relationships based on distance and link quality, and weighted aggregation of neighbor encoding features based on attention weights. Figure 3The neighborhood aggregation message represents a summary of the overall behavior or state of the one-hop neighbors perceived by the UAV in the current time slot. The attention mechanism assigns learnable, differentiated weights to different neighbors, enabling the UAV to assign different levels of attention to neighbors with different link qualities and service states, thereby avoiding the problem of key information being overwhelmed by simple mean aggregation. In one possible implementation, the attention mechanism can calculate weights based on a similarity metric between encoded features and then weight and summarize the neighbor encoded features; the specific attention calculation method can be further defined in subsequent embodiments.

[0042] Step S140: Input the local input state and neighborhood aggregation message into the shared policy network to obtain the joint action of the current time slot.

[0043] The joint action refers to the multi-dimensional scheduling decision results simultaneously output by the UAV in the current time slot, including at least channel selection action, transmit power action, and packet scheduling priority action. The channel selection action is used to determine the spectrum resources occupied by data transmission in the current time slot; the transmit power action is used to determine the transmit power level of the UAV; and the packet scheduling priority action is used to determine the service category or service queue that is prioritized for service in this time slot.

[0044] This invention incorporates three types of actions into a unified action space, which are output together by a shared policy network in the same inference, enabling explicit handling of the coupling relationship between channel occupancy, link reliability, interference leakage, service waiting delay, and transmission energy consumption.

[0045] The shared policy network is a policy inference network for decentralized execution. Its parameters are updated by the ground training server during the centralized training phase and then distributed to the onboard computing units of each UAV, sharing the same network parameters among all UAVs. The shared policy network includes at least a feature fusion substructure and an action distribution output substructure: the feature fusion substructure takes local input states and neighborhood aggregation messages as parallel inputs and fuses them; the action distribution output substructure outputs corresponding action distributions in the channel selection dimension, transmit power dimension, and packet scheduling priority dimension, and obtains the joint action by sampling or greedily selecting from them.

[0046] Step S150: Control the UAV to perform joint actions and complete the communication scheduling of the current time slot.

[0047] Specifically, the UAV's onboard computing unit switches the channel resources used for data transmission and reception based on channel selection, switches the power level of its transmitter based on transmit power, and selects the service queue for the current time slot based on packet scheduling priority. After these actions are completed, communication scheduling for the current time slot is finished, and the process moves to the next time slot.

[0048] It should be noted that the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

[0049] In this embodiment, a training node refers to a centralized training device that undertakes parameter updates, including but not limited to ground training servers, edge training nodes, or centrally controlled UAVs with sufficient computing power. During the centralized training phase, the training node receives trajectory samples uploaded by each UAV and updates the shared policy network parameters, then distributes the updated parameters to the onboard computing units of each UAV at preset intervals.

[0050] In the aforementioned emergency search and rescue scenario embodiment, each UAV's onboard computing unit independently executes steps S110 to S150 in each time slot: first, it collects the current local input state and identifies the one-hop neighbor of the current time slot; then, it exchanges the encoded feature vector with the one-hop neighbor; then, it forms a neighborhood aggregation message for the current time slot through attention aggregation; it inputs the local input state and the neighborhood aggregation message into the shared policy network to obtain channel selection actions, transmit power actions, and data packet scheduling priority actions, selecting the service that is prioritized for this time slot among the three types of services: control commands, status feedback, and task data; finally, the UAV executes the joint actions to complete the communication scheduling for this time slot.

[0051] This embodiment avoids local optima caused by step-by-step decision-making by incorporating channel selection, transmit power, and packet scheduling priority into the same action space and outputting them together in the same inference by the shared policy network. By exchanging only the coding features without exchanging the original observations and policy parameters, it can support inter-neighbor collaboration while controlling communication overhead. Furthermore, under the centralized training and decentralized execution framework, each UAV does not rely on the central node to participate in real-time scheduling, which improves the real-time performance and robustness in highly dynamic topology scenarios.

[0052] Optionally, obtaining the local input status of the UAV in the current time slot includes: The location, velocity, remaining energy, queue status, channel state estimate, and interference measurement of the UAV in the current time slot are collected to obtain the local observation of the current time slot. Extract the temporal aggregation features of the previous time slot from the UAV's local buffer; The local observations of the current time slot and the temporal aggregation features of the previous time slot are used as parallel inputs to form the local input state; wherein, the temporal aggregation features of the previous time slot are the neighborhood temporal summary representation generated and cached on the UAV side in the previous time slot, which is used to supplement the temporal memory.

[0053] In this application, location information can be obtained from the UAV's onboard satellite positioning module, inertial navigation module, or visual odometry module; speed information can be calculated by the inertial measurement unit, ground speed estimation module, or satellite positioning module; remaining energy information can be obtained from feedback by the onboard battery management system; queue status information includes at least the current queue leader, first packet waiting time, or queue occupancy ratio of each service queue; the channel state estimate is preferably the instantaneous channel gain or instantaneous channel quality index obtained based on pilot signal estimation; the interference measurement value refers to the instantaneous estimate of the intensity of co-channel interference from other nodes perceived by the local unit in the current time slot. The above acquisition results, arranged in a predetermined splicing order, constitute the local observation of the current time slot, denoted as... .

[0054] The temporal aggregation feature of the previous time slot is a neighborhood temporal summary representation that has been formed and written into the local cache of the UAV on the airborne side of the previous time slot, denoted as This temporal aggregation feature is generated by the neighborhood aggregation process of the previous time slot and is persistently stored in the local cache for retrieval in the current time slot. The purpose of this temporal aggregation feature is to supplement temporal memory, enabling the shared policy network to utilize both the instantaneous observations of the current time slot and the accumulated knowledge of the UAV's past neighborhood behavior when making current decisions, thereby improving its adaptability to rapidly changing topologies.

[0055] The local observations of the current time slot and the temporal aggregated features of the previous time slot are used as parallel inputs to form the local input state.

[0056] Specifically, the UAV's onboard computing unit will and The local input status for this time slot is obtained by combining the components according to the predetermined splicing order: ; It is important to emphasize the temporal aggregation characteristics of the previous time slot. The neighborhood aggregation result, derived from the previous time slot's neighborhood aggregation result already completed on the local side, differs from the current time slot's neighborhood aggregation message, which is recalculated in this time slot, in terms of both time source and function: the former is used to supplement temporal memory, serving as input for the local historical state; the latter is used to reflect changes in the current time slot's neighborhood, serving as a real-time neighborhood reference for decision-making. The two are not confused and are separately incorporated into the input of the shared policy network.

[0057] By separating the input of temporal memory and instantaneous observation in this embodiment, the overlapping of historical information and current neighborhood information can be avoided under dynamic topology, thereby improving the ability of the shared policy network to identify environmental changes.

[0058] Optionally, constructing the one-hop neighbor set of the UAV in the current time slot includes: The relative distance between the UAV and the detectable node, and the link quality index between the UAV and the detectable node are obtained; wherein, the link quality index includes the signal-to-interference-plus-noise ratio obtained based on pilot estimation; When the relative distance is not greater than a preset distance threshold and the link quality index is not less than a preset link quality threshold, an adjacency relationship for the current time slot is established between the UAV and the detectable node. All nodes that have the adjacency relationship with the drone are included in the one-hop neighbor set.

[0059] In this application, a detectable node refers to another UAV node that this UAV can detect in the current time slot via broadcast messages, signaling interaction, or pilot signals. The relative distance can be obtained by calculating the position information difference, the time difference of arrival of the ranging signal, or by inverting the received signal strength.

[0060] Link quality metrics include, but are not limited to, any one or a combination of several of the following: signal-to-interference-plus-noise ratio (SINR) obtained based on pilot estimation, received signal-to-noise ratio, received signal strength indication, and link reliability metrics. Among these, the SINR obtained based on pilot estimation is preferred to more accurately reflect link availability under co-channel interference conditions.

[0061] When the relative distance is not greater than a preset distance threshold and the link quality index is not less than a preset link quality threshold, an adjacency relationship for the current time slot is established between the UAV and the detectable node.

[0062] Specifically, the following discriminant can be used: when the drone With drones Satisfy relative distance Not greater than the distance threshold And the link quality estimate Not less than the link quality threshold At that time, the two are considered to be adjacent in the current time slot: in, Preferably obtained from pilot estimation Or its estimated value, it can also be represented by equivalent indicators such as received signal-to-noise ratio and link reliability index.

[0063] Distance threshold The link quality threshold can be determined by a tradeoff between the maximum achievable communication distance and the typical reliable communication distance of the UAV communication module. It can be determined by adding an engineering margin to the lowest demodulation threshold corresponding to the physical layer modulation and coding scheme.

[0064] All nodes that are adjacent to the drone are grouped into a one-hop neighbor set, denoted as . .

[0065] It should be noted that the adjacency matrix It is reconstructed once in each time slot, therefore The changes are dynamic, occurring due to the rapid movement of the drone and the rapid fluctuations in link quality. Subsequent steps, such as coded feature exchange and neighborhood message aggregation, are based on the current time slot. This ensures that the aggregation process can reflect the actual changes in the link relationships in a timely manner.

[0066] In the aforementioned emergency search and rescue scenario, when a drone's maneuvering causes the link quality between it and its neighbor to drop below a certain level... At that time, the original neighbor will be automatically removed from the list in the next time slot. Remove from the list; when a drone enters a new nearest neighbor range and the link quality meets the threshold, it is included in the next time slot. .

[0067] This embodiment avoids the problem of neighborhood information mismatch caused by a single threshold by jointly judging distance threshold and link quality threshold: judging by distance alone may include nodes with extremely poor link quality in the neighborhood, while judging by link quality alone may cause neighborhood oscillation due to instantaneous interference fluctuations.

[0068] Optionally, the step of aggregating the received encoded features based on an attention mechanism to generate a neighborhood aggregation message for the current time slot includes: By implementing the query parameters and key parameters in the policy branch aggregation module of the attention mechanism, the attention weight between the encoded features generated by the UAV itself and the encoded features received from the neighboring UAV is calculated. Based on the attention weights and the value parameters in the policy branch aggregation module, the received encoded features sent by the neighboring drones are weighted and summed to obtain the neighborhood aggregation message of the current time slot.

[0069] The strategy branch aggregation module is a pre-configured attention aggregation substructure within the shared policy network, which contains learnable query parameters. Key parameters and value parameters The UAV's onboard computing unit first processes the locally encoded features. After querying the parameters Mapped to a query vector, each neighbor's encoded feature. Key parameters The mapping is done as a key vector, and then the attention weights are obtained by scaling the dot product and normalizing with softmax. : ; in, The scaling factor is the dimension of the encoded feature. This is used to avoid gradient saturation caused by excessively large dot product values.

[0070] In this application, specifically, each neighbor's encoded features are... Longitude parameter After mapping, according to the corresponding Weighted summation: ; From this, we obtain This refers to the neighborhood aggregation message of this UAV in the current time slot. The attention mechanism enables the shared policy network to adaptively amplify the information weight provided by important neighbors (such as those located at critical link conflict positions or carrying high-priority services) and suppress the information of weakly related neighbors, thereby improving the effectiveness of neighborhood collaboration.

[0071] This embodiment uses a learnable attention aggregation method based on query-key-value three parameters, enabling neighborhood aggregation messages to adaptively reflect the importance of different neighbors to the UAV's decision-making as training progresses.

[0072] Optionally, controlling the UAV to perform the joint action and complete the communication scheduling of the current time slot includes: According to the channel selection action, switch the spectrum resources occupied by the UAV data transmission; according to the transmit power action, adjust the transmit power level of the UAV to synchronously change the link transmission reliability, the interference leakage intensity to the one-hop neighbor set, and the local transmit power consumption. Based on the packet scheduling priority action, the service order of different service queues of the UAV is adjusted; After performing the joint action, the number of successfully transmitted bits, the estimated instantaneous co-channel interference, the transmission energy consumption and service delay statistics of this time slot are recorded as feedback sample data for this time slot.

[0073] In this application, the UAV onboard computing unit performs channel selection actions based on the shared policy network output. One of the K available channels is selected as the data transmission channel for this time slot, and the local oscillator switching is completed by the radio frequency front end. Channel selection changes the co-channel occupancy relationship and spectrum reuse method, thus directly affecting the level of co-channel interference and available throughput.

[0074] Specifically, the UAV's onboard computing unit operates based on the transmit power output by the shared policy network. The power amplifier gain is set by selecting one from a preset set of discrete power levels (e.g., low, medium, and high). Increasing the transmit power will improve the link success rate, but at the same time, it will increase the interference leakage intensity to the one-hop neighbor set and increase the local transmit power consumption; decreasing the transmit power has the opposite effect.

[0075] Specifically, the UAV's onboard computing unit schedules actions based on the data packet priority output by the shared policy network. The current time slot's transmission window will be preferentially allocated to a specified service queue or service category. For example, in an emergency search and rescue scenario, when... When sending control command packets, the current time slot transmission window is prioritized for sending control command packets, thereby reducing the end-to-end latency of critical services.

[0076] Among them, the number of bits successfully sent The instantaneous co-channel interference estimate is obtained from MAC layer ACK feedback or physical layer successful decoding statistics; the instantaneous co-channel interference estimate can be obtained by inverting the pilot measurement results at the receiver; the transmit power consumption in this time slot is determined by the transmit power. The computational energy consumption generated by the summation of the product of the time slot length Δt and the in-house inference and control processing. Obtain; Business latency statistics The latency can be estimated using packet timestamps, queue first packet waiting time, or sliding window average latency. These metrics together constitute the feedback sample data for this time slot, providing a constrained statistical basis for subsequent training phases.

[0077] This embodiment combines joint action execution with multi-dimensional feedback sample collection, enabling the communication scheduling process in each time slot to not only drive service transmission but also simultaneously generate indicator data for training and constraint adjustment.

[0078] Optionally, the method further includes: The UAV writes the local input state, the neighborhood aggregation message, the joint action, the instant reward, the local input state of the next time slot, and the constraint statistics composed of the feedback sample data into the UAV's local trajectory buffer to form a trajectory sample; The drone uploads the trajectory samples to the training node so that the training node can update the parameters during the centralized training phase. The UAV receives the updated parameters of the shared policy network from the training node at a preset period, and performs the communication scheduling for subsequent time slots according to the updated parameters of the shared policy network.

[0079] In this application, after the UAV onboard computing unit completes steps S110 to S150 in each time slot, it will process the data for that time slot. , , = ( , Instant rewards The next time slot and by The constraint statistics, consisting of the instantaneous co-channel interference estimate, the transmission energy consumption in this time slot, and the service delay statistics, are written into the local trajectory buffer in chronological order. The trajectory buffer can adopt a circular buffer or a first-in-first-out queue structure, and the buffer capacity is determined based on the onboard storage resources and the parameter update cycle.

[0080] Specifically, each UAV uploads trajectory samples from its local trajectory buffer to the ground training server via a control channel or dedicated backhaul link according to a preset reporting cycle. It should be noted that only encoded or summarized trajectory samples are reported; the original onboard sensing data is not uploaded, thereby controlling backhaul overhead and protecting local information.

[0081] Specifically, after receiving the parameter package from the ground training server, the UAV onboard computing unit loads the parameters into its shared policy network and executes steps S110 to S150 according to the new parameters starting from the next time slot.

[0082] This embodiment enables each UAV to aggregate its own execution experience to the training node through local trajectory caching and periodic reporting mechanisms, and allows all UAVs in the network to share the latest strategies through parameter distribution mechanisms, thereby achieving continuous strategy optimization while retaining the advantages of decentralized execution.

[0083] Optionally, the method further includes: Receive the trajectory samples uploaded by each UAV in the multi-UAV self-organizing communication network; Based on the local input states, joint actions, and adjacency relationships of all the UAVs in the same time slot, the global state is reconstructed; The global state is feature-aggregated using a critic branch encoder to generate a global aggregated representation. The global state, the global aggregate representation, and the joint action are input into a centralized critic network to obtain a value assessment result. Based on the value assessment results and the final reward feedback generated by the two-layer constraint linkage adjustment mechanism, the parameters of the shared policy network and the parameters of the centralized critic network are updated, and the updated parameters of the shared policy network are distributed to each UAV.

[0084] In this application, specifically, the ground training server enables the training sample receiving service, gathers trajectory samples from each UAV according to a preset reporting cycle, and establishes an index according to the UAV identifier and time slot number for subsequent time slot alignment.

[0085] Specifically, the ground training server aligns and stitches together the samples uploaded by N drones within the same time slot t to obtain the global state of that time slot. It includes at least all local observations, queue states, remaining energy, link quality estimates, and adjacency matrices. .

[0086] It is important to emphasize that the global state is not obtained directly by a single node during the execution phase, but rather by the ground training server integrating the samples reported by each node during the training phase, thereby avoiding dependence on the central node for real-time scheduling.

[0087] The critic branch encoder is a feature aggregation substructure configured in the centralized critic network, denoted as... The ground training server will enter The global aggregate representation is obtained. : ; Specifically, the ground training server will and and the joint actions of the current time slot ( A centralized network of critics, through shared input, yields state value or action value: The dual-layer constraint linkage adjustment mechanism refers to a processing mechanism that incorporates interference, time delay, and energy consumption constraints into the reward signal generation process. Its specific implementation method can be found in the further details of the subsequent embodiments.

[0088] In this step, after the ground training server completes the parameter update, it sends the updated shared policy network parameters to the onboard computing units of each UAV through the control channel according to the preset cycle. Each UAV then loads the parameters and puts them into real-time scheduling in subsequent time slots.

[0089] This embodiment enables the training process to utilize global information for high-quality value assessment without compromising the decentralized nature of the execution phase, thus balancing the global assessment capabilities of the training phase with the real-time performance of the execution phase.

[0090] Optionally, the shared policy network and the centralized critic network are respectively configured with structurally corresponding but parameter-independent message aggregation branches; wherein, the policy message aggregation branch configured in the shared policy network is used to perform weighted aggregation of the encoded features received from the neighboring drones during the execution phase of the drone to generate the neighborhood aggregation message; The critic message aggregation branch configured in the centralized critic network is used to aggregate the global features corresponding to the global state during the centralized training phase of the training node to generate the global aggregate representation. The parameters of the policy message aggregation branch and the parameters of the critic message aggregation branch are trained independently, and the UAV only runs the policy message aggregation branch during the execution phase.

[0091] In this embodiment of the invention, the shared policy network and the centralized critic network are respectively configured with structurally corresponding but parameter-independent message aggregation branches. Specifically: The shared policy network is configured with a policy message aggregation branch, denoted as the policy branch, which includes the aforementioned policy branch encoder. With the strategy branch aggregation module (including parameters) , , ).

[0092] The strategy branch is used during the drone's execution phase to perform weighted aggregation of the encoded features received from neighboring drones, generating a neighborhood aggregation message for the current time slot. and local input status The action distribution output substructure of the shared policy network is input together to obtain the joint action.

[0093] The centralized critic network is configured with a critic message aggregation branch, denoted as the critic branch, which includes a critic branch encoder. And the corresponding critic branch aggregation module.

[0094] The critic branch is used to aggregate global features corresponding to the global state during the centralized training phase of the training nodes, generating a global aggregated representation. and with global state The value assessment substructure of the centralized critic network is input together through joint actions.

[0095] The parameters of the policy message aggregation branch and the critic message aggregation branch are updated independently during training and are not shared with each other; furthermore, the UAV onboard computing unit only runs the policy message aggregation branch during the execution phase and does not need to run the critic message aggregation branch.

[0096] It is important to note that the branching message aggregation scheme described in this invention does not refer to an abstract forking concept within a single network. Rather, it means that the policy network and the critic network each possess an independent neighborhood aggregation branch. While their structures may be identical, their parameters are trained separately, and their functions are dedicated to execution decision-making and value assessment, respectively. This design allows the two branches to optimize independently based on their respective tasks (execution decision-making focuses more on the quality of local action selection; value assessment focuses more on predicting long-term global returns), avoiding mutual interference between them.

[0097] The branched message aggregation scheme presented in this embodiment can provide functionally matched feature aggregation paths for execution decision-making and value evaluation under decentralized execution and centralized training frameworks, respectively, thereby improving training stability and execution accuracy.

[0098] Optionally, the step of generating the final reward feedback by the dual-layer constraint linkage adjustment mechanism includes: Statistical windows are established for the interference index, latency index, and energy consumption index included in the constraint statistics of the trajectory sample, and the average index value of the target index within the statistical window is calculated; wherein, the target index includes the interference index, the latency index, or the energy consumption index; According to the Lagrange multiplier adjustment layer, when it is determined that the average index value of the target index exceeds the threshold set by the target index, the Lagrange multiplier configured for the target index is increased to obtain the violation penalty item generated by the Lagrange multiplier. Based on the barrier function pre-suppression layer, and the remaining margin between the average index value of the target index and the threshold set by the target index, the risk suppression penalty term of the target index is calculated. The final reward feedback is obtained by subtracting the violation penalty and the risk suppression penalty from the basic reward; wherein the basic reward is obtained based on a weighted combination of the weighted throughput, the latency index, and the energy consumption index of the current time slot.

[0099] In this application, the target indicators include interference indicators, latency indicators, or energy consumption indicators. In one feasible approach, the interference indicator can be defined as: The latency metric can be defined as: Energy consumption index can be defined as: in, Indicates the first A drone in a time slot The number of bits successfully sent. Indicates business weight. This represents the set of links where potential interference coupling exists. Indicates the first The drone was used to attack the first Interference gain caused by a single receiving link This represents a latency statistic that can be estimated from the first packet waiting time, the actual transmission delay of completed services, or the average latency of a sliding window. This indicates the computational energy consumption of the machine in this time slot for inference and control processing.

[0100] The ground training server sets a statistical window of length W for each type of constraint index, and averages the index values ​​within the nth window to obtain the average index value of the target index within that statistical window. Specifically, the Lagrange multipliers are updated according to the statistical window: in, To determine the length of the statistical window, a sliding window or an update window consisting of several consecutive time slots is preferred. This is the update step size corresponding to the constraint. If the most recent statistical window still exceeds the threshold, then maintain or increase it. If continuous If a statistical window meets the threshold condition, the decay coefficient will be gradually reduced. To free up space for exploration.

[0101] Based on the updated multipliers, construct the violation penalty term: ,in, Indicates to Take the non-negative part. Indicates the first The Lagrange multipliers corresponding to class constraints Indicates the first The obstacle coefficient corresponding to the class constraint, To prevent positive integers with a denominator of zero, this item comes into play to provide violation correction feedback to the strategy when the indicator has gone out of bounds.

[0102] Specifically, the remaining margin is defined as And construct a risk suppression penalty term in the form of a logarithmic barrier: ;in, ε is the barrier coefficient, and ε is a positive constant to prevent the denominator from being zero.

[0103] when Gradually approaching When the remaining margin decreases, this value increases significantly, thereby suppressing the tendency to choose high-risk actions in advance and playing a role in boundary early warning.

[0104] The basic reward is derived from a weighted combination of the weighted throughput, latency, and energy consumption metrics for this time slot. The basic reward can be written as: The final reward feedback can be written as: in, Indicates to Take the non-negative part. Indicates the first The Lagrange multipliers corresponding to class constraints Indicates the first The obstacle coefficient corresponding to the class constraint, To prevent positive numbers with a denominator of zero. The second term is a violation penalty term, which penalizes actions that have already exceeded the limit; the third term is a risk suppression penalty term, which preemptively suppresses actions that gradually approach the limit; the sign direction is clearly deducted from the basic reward.

[0105] This embodiment uses a two-layer linkage mechanism of boundary warning and violation correction to enable the shared policy network to avoid frequent boundary crossings in the later stages of training and to avoid blind exploration in the early stages of training, thereby significantly improving the stability of handling scenarios where multiple constraints approach the boundary simultaneously.

[0106] Optionally, the step of calculating the risk suppression penalty term for the target indicator based on the residual margin between the average indicator value of the target indicator and the threshold set for the target indicator, according to the barrier function pre-suppression layer, includes: When the average value of the target indicator exceeds the threshold set by the target indicator in the most recent statistical window, the pre-configured barrier coefficient of the target indicator is maintained or increased. When the average value of the target indicator does not exceed the threshold set by the target indicator within a consecutive preset number of statistical windows, the barrier coefficient is reduced according to a preset attenuation coefficient. Logarithmic operations are performed based on the barrier coefficient and the remaining margin to generate the risk suppression penalty term, so as to pre-suppress the tendency to choose relevant high-risk actions in the shared policy network.

[0107] In this application, the ground training server completes each training session. After calculation, it is compared with the corresponding threshold. Comparison: If > Then maintain the current obstacle coefficient. It remains unchanged, or it is amplified according to a preset growth coefficient to strengthen the pre-suppression force in the direction of the constraint.

[0108] Specifically, if M consecutive statistical windows (M is a pre-defined integer, for example, M = 3 to 5) all satisfy... < Then, it will gradually decrease according to the preset attenuation coefficient (e.g., 0.8 or 0.9). This allows for a more appropriate release of the exploration space of the shared policy network in this constraint direction, avoiding excessive suppression of policy improvement.

[0109] Specifically, the current obstacle coefficient Substituting the remaining margin into the logarithmic barrier function form: ; As can be seen from this form, when the residual margin is close to 0, the denominator tends to ε, and the value of the logarithmic term increases sharply, thereby significantly reducing the tendency of the shared policy network to choose high-risk actions that will further approach the threshold; while when the residual margin is large, the value of the logarithmic term is close to zero, and it has almost no constraint on the policy.

[0110] This embodiment utilizes a mechanism that dynamically adjusts the obstacle coefficient based on window statistics, enabling the risk suppression penalty term to provide differentiated suppression strength under different training stages and different boundary crossing situations, thereby further enhancing the stability and policy flexibility of constraint processing.

[0111] Optionally, updating the parameters of the shared policy network includes: After a preset number of time slots, the training node calculates the advantage function based on the value assessment result and the final reward feedback; The parameters of the shared policy network are updated once based on the advantage function and the truncated agent objective function; wherein the truncated agent objective function is used to limit the magnitude of a single policy update; When the preset maximum number of training rounds is reached, or when the parameter change of the shared policy network is lower than a preset threshold for several consecutive rounds, the training node saves the parameters of the shared policy network and sends them to the onboard computing unit of each UAV.

[0112] In this application, the ground training server triggers a parameter update process every preset number of time slots, first based on the value assessment results output by the centralized critic network and the final reward feedback. Calculate the advantage function using the generalized advantage estimation method. .

[0113] The advantage function measures the relative quality of the current joint action compared to the average level and serves as a guide for subsequent policy updates.

[0114] Specifically, the truncated proxy objective function takes the following form: in, The generalized dominance estimation method is preferred to obtain the estimated value of the dominance function. This is the pruning factor, used to limit the magnitude of a single update.

[0115] in, This represents the ratio of the probabilities of the new and old strategies. The clip function is used to clip values ​​(e.g., values ​​between 0.1 and 0.2) to the specified value. Limited to [ Within the range.

[0116] Ground training server to maximize To achieve the goal, the parameters θ of the shared policy network are updated using gradient ascent. The magnitude of the policy change in a single update is thus limited to a reasonable range by the truncated agent objective function.

[0117] Specifically, the ground training server records the norm of policy parameter changes after each round of parameter updates. When the number of training rounds reaches the preset maximum number of rounds, or the norm of parameter changes in consecutive rounds is lower than a preset threshold, the shared policy network is considered to have converged. The server then saves the current parameters and sends them to each UAV's onboard computing unit via the control channel. After loading these parameters, each UAV's onboard computing unit can independently complete the real-time scheduling described in steps S110 to S150 during the runtime phase, without the need for a central node.

[0118] This embodiment, by truncating the proxy objective function and using a convergence discrimination mechanism based on parameter stability, can ensure training stability while timely policy delivery, thereby supporting the stable operation of the decentralized execution phase.

[0119] In one alternative embodiment, Figure 4 Figure 4 is a schematic diagram of the overall process of the distributed channel access and resource scheduling method for UAV self-organizing communication networks based on multi-agent reinforcement learning provided by this invention. As shown in Figure 4, the method provided by this embodiment of the invention is executed in a closed loop of environment modeling—state acquisition—neighborhood reconstruction—feature aggregation—joint decision-making—action execution and feedback—centralized update, and specifically includes the following steps: Step S1: Construct a multi-UAV self-organizing communication network environment model.

[0120] In this step, the ground training server or airborne initialization unit establishes an environment model for a multi-UAV self-organizing communication network based on parameters such as the mission area size, the number of UAVs N, the number of available channels K, the number of discrete transmission power levels, the number of service categories, and the time slot length Δt. This environment model includes at least a set of UAV nodes, a set of channel resources, a service queue model, and a link propagation model. It provides an interactive simulation environment for reinforcement learning algorithms during the training phase and a unified state space and action space definition for each UAV's airborne computing unit during the deployment phase. In the aforementioned emergency search and rescue scenario embodiment, the environment model corresponds to N=20, K=4, 3 transmission power levels, and three service categories: control commands, status feedback, and mission data.

[0121] Step S2: Obtain local observation information and link status information for each UAV.

[0122] In this step, each UAV's onboard computing unit collects local observation information and link status information at the beginning of each time slot. The local observation information includes at least local state variables such as position, velocity, remaining energy, and queue status; the link status information includes at least the instantaneous channel state estimate obtained based on pilot estimation and the current interference measurement.

[0123] Step S3: Construct a dynamic neighborhood graph based on distance and link quality.

[0124] In this step, each UAV's onboard computing unit determines the adjacency relationship of the current time slot based on the relative distance between itself and the detectable nodes and the link quality index, according to the dual discrimination criteria of distance threshold and link quality threshold, and obtains the dynamic neighborhood graph of the current time slot.

[0125] Step S4: Each UAV exchanges neighborhood encoding features and performs attention aggregation.

[0126] In this step, each UAV onboard computing unit first maps the local input state into encoded features through a policy branch encoder, then exchanges encoded features with the current time slot's one-hop neighbor, and performs weighted aggregation of the received neighbor encoded features based on an attention mechanism to obtain the neighborhood aggregation message for the current time slot.

[0127] Step S5: Input the local observations and aggregation results into the policy network and output the joint scheduling action.

[0128] In this step, each UAV's onboard computing unit inputs its local input state and the current time slot neighborhood aggregation message into the shared policy network. The shared policy network then outputs a joint scheduling action that includes channel selection, transmit power, and packet scheduling priority actions during the same inference process. This step incorporates the three types of scheduling actions into a unified action space, enabling explicit handling of the coupling relationships between them.

[0129] Step S6: Perform channel access, transmit power, and packet priority scheduling, and calculate rewards and constraint violations.

[0130] In this step, each UAV's onboard computing unit performs channel switching, transmit power adjustment, and service queue service within its time slot according to joint scheduling actions. It also collects the number of successfully transmitted bits, the estimated instantaneous co-channel interference, the transmit energy consumption within the time slot, and service delay statistics as feedback sample data. Based on the feedback sample data, the instantaneous reward for the time slot is calculated, and the constraint violations for the time slot are statistically analyzed according to three categories: interference constraints, delay constraints, and energy consumption constraints.

[0131] Step S7: Update the policy network, value network, Lagrange multipliers, and obstacle coefficients using a centralized training and decentralized execution framework.

[0132] In this step, the ground training server aggregates trajectory samples uploaded by each UAV during the centralized training phase, reconstructs the global state, completes value evaluation using a centralized critic network, and generates final reward feedback based on a two-layer constraint linkage adjustment mechanism. This updates the parameters of the shared policy network, the centralized critic network (value network), the Lagrange multipliers, and the obstacle coefficients. In a preferred implementation, the parameter update algorithm employs the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm. On the execution side, each UAV independently and in a distributed manner performs action reasoning, while on the training side, the ground training server centrally updates the parameters, thus aligning with the centralized training and decentralized execution framework of this invention.

[0133] After completing step S7, the ground training server will send the updated shared policy network parameters to the onboard computing units of each UAV. After each UAV loads the new parameters, it will return to step S2 and enter the loop execution of the next time slot until the preset maximum number of training rounds is reached or the parameter change converges to the preset threshold.

[0134] This process couples four core components—dynamic neighborhood reconstruction, attention aggregation, unified action decision-making, and two-layer constraint linkage adjustment—into a complete closed loop. This enables continuous optimization of the shared policy network in highly dynamic multi-UAV scenarios, allowing it to balance multi-objective optimization of channel access, power control, and service scheduling even under decentralized execution conditions.

[0135] Figure 5 is a schematic diagram of the overall structure of the distributed channel access and resource scheduling device for UAV self-organizing communication network based on multi-agent reinforcement learning provided by the present invention. As shown in Figure 5, the device provided in this embodiment of the present invention is physically divided into two parts: a distributed resource scheduling device on the execution side and a centralized training module on the training side. The execution side is deployed in a decentralized manner on the onboard computing units of each UAV, while the training side is deployed in a centralized manner on the ground training server. The two are connected through a control channel to achieve trajectory sample reporting and policy parameter distribution, thus conforming to the overall framework of decentralized execution side and centralized training side.

[0136] The distributed resource scheduling device is deployed on one side of the UAV's onboard computing unit and, in order of data flow, includes at least the following sub-modules: The environmental perception and status acquisition module is used to acquire local observations, link status, and queue status. At the start of each time slot, this module collects the aforementioned information through the airborne sensor interface, communication module interface, and local buffer interface, and assembles it into the local input status for that time slot. Its specific functions correspond to the observation acquisition module in the aforementioned embodiments.

[0137] The dynamic neighborhood construction module is used to determine the neighborhood set of this UAV based on relative distance and link quality indicators in each time slot.

[0138] After obtaining the link status information from the environment perception and status acquisition module, this module outputs the adjacency relationship of the current time slot and the set of one-hop neighbors, and then passes it to the downstream neighborhood message encoding and exchange module.

[0139] The neighborhood message encoding and exchange module maps local input states to encoded features using a policy branch encoder and exchanges these encoded features with neighboring drones in the one-hop neighbor set. The module's output includes locally generated encoded features and collected neighbor encoded features.

[0140] The attention aggregation module is used to perform weighted aggregation of neighbor encoded features to generate collaborative features (i.e., neighborhood aggregation messages) for the current time slot. This module calculates attention weights and performs weighted summation based on the query, key, and value parameters in the policy branch aggregation module.

[0141] The joint policy decision module is used to output joint scheduling actions based on local observations and aggregation results. This module inputs the local input state and cooperative characteristics into the shared policy network to obtain channel selection actions, transmit power actions, and packet scheduling priority actions.

[0142] The resource scheduling execution module is used to perform channel selection, power control, and queue scheduling. Based on the joint scheduling actions output by the joint policy decision module, this module drives the RF front-end, power control circuit, and queue service scheduler to complete the actual communication scheduling operations.

[0143] The constraint evaluation and penalty update module is used to evaluate the constraint violations in this time slot and update the Lagrange multipliers and obstacle coefficients. This module obtains feedback sample data for this time slot from the resource scheduling execution module, performs statistics on the three types of constraints: interference, delay, and energy consumption, and generates the final reward feedback and updates the multipliers and obstacle coefficients according to the two-layer constraint linkage adjustment mechanism.

[0144] The centralized training module is deployed on the ground training server side and interacts with the distributed resource scheduling device through two logical channels: one is a channel for providing policy / value information, which is used to provide the value assessment results and reward feedback required for training to the joint policy decision module and constraint evaluation and penalty update module in the distributed resource scheduling device during the training phase; the other is a channel for updating policy network parameters, which is used to send the latest parameters of the shared policy network to each UAV onboard computing unit after each round of parameter update.

[0145] The outputs of the environment perception and state acquisition module are fed into the dynamic neighborhood construction module and the neighborhood message encoding and exchange module, respectively. The dynamic neighborhood construction module outputs the adjacency relationship to the neighborhood message encoding and exchange module. The neighborhood message encoding and exchange module sends the encoded features to the attention aggregation module. The attention aggregation module sends the collaborative features to the joint policy decision module. The joint policy decision module outputs the joint scheduling action to the resource scheduling execution module. The resource scheduling execution module outputs feedback sample data to the constraint evaluation and penalty update module. The output of the constraint evaluation and penalty update module interacts with the centralized training module via the control channel. The parameter distribution output of the centralized training module is finally fed back to the joint policy decision module to complete the parameter loading of the shared policy network.

[0146] This device, through the collaboration of seven sub-modules on the execution side and the cooperation of the centralized training module on the training side, can achieve real-time scheduling under decentralized execution conditions and parameter optimization under centralized training conditions, thereby supporting the complete closed-loop implementation of the method described in this invention.

[0147] The UAV communication scheduling device based on multi-agent reinforcement learning provided by the present invention will be described below. The UAV communication scheduling device based on multi-agent reinforcement learning described below can be referred to in correspondence with the UAV communication scheduling method based on multi-agent reinforcement learning described above.

[0148] Figure 6This is a schematic diagram of the UAV communication scheduling device based on multi-agent reinforcement learning provided by the present invention, as shown below. Figure 6 As shown, it includes: The observation and acquisition module 610 is used to acquire the local input state of the UAV in the current time slot and construct the one-hop neighbor set of the UAV in the current time slot; The feature encoding and message aggregation module 620 is used to map the local input state into encoded features using a policy branch encoder, send the encoded features to neighboring drones in the one-hop neighbor set, and receive encoded features sent by neighboring drones in the one-hop neighbor set. The aggregation module 630 is used to aggregate the encoded features received from the neighboring drones based on an attention mechanism to generate the neighborhood aggregation message for the current time slot. The input module 640 is used to input the local input state and the neighborhood aggregation message into the shared policy network to obtain the joint action of the current time slot; wherein, the joint action includes channel selection action, transmit power action and packet scheduling priority action; The scheduling module 650 is used to control the UAV to perform the joint action and complete the communication scheduling of the current time slot; wherein, the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

[0149] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a UAV communication scheduling method based on multi-agent reinforcement learning. This method includes obtaining the local input state of the UAV in the current time slot and constructing a one-hop neighbor set of the UAV in the current time slot. The local input state is mapped to encoded features by a policy branch encoder, the encoded features are sent to neighbor drones in the one-hop neighbor set, and the encoded features sent by neighbor drones in the one-hop neighbor set are received. The received encoded features are aggregated based on an attention mechanism to generate a neighborhood aggregation message for the current time slot; The local input state and the neighborhood aggregation message are input to the shared policy network to obtain the joint action of the current time slot; wherein, the joint action includes channel selection action, transmit power action and packet scheduling priority action; The drone is controlled to perform the joint action to complete the communication scheduling of the current time slot; wherein, the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

[0150] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the UAV communication scheduling method based on multi-agent reinforcement learning provided by the above methods, the method including: obtaining the local input state of the UAV in the current time slot, and constructing a one-hop neighbor set of the UAV in the current time slot; The local input state is mapped to encoded features by a policy branch encoder, the encoded features are sent to neighbor drones in the one-hop neighbor set, and the encoded features sent by neighbor drones in the one-hop neighbor set are received. The received encoded features are aggregated based on an attention mechanism to generate a neighborhood aggregation message for the current time slot; The local input state and the neighborhood aggregation message are input to the shared policy network to obtain the joint action of the current time slot; wherein, the joint action includes channel selection action, transmit power action and packet scheduling priority action; The drone is controlled to perform the joint action to complete the communication scheduling of the current time slot; wherein, the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

[0152] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the UAV communication scheduling method based on multi-agent reinforcement learning provided by the above methods, the method comprising: obtaining the local input state of the UAV in the current time slot, and constructing a one-hop neighbor set of the UAV in the current time slot; The local input state is mapped to encoded features by a policy branch encoder, the encoded features are sent to neighbor drones in the one-hop neighbor set, and the encoded features sent by neighbor drones in the one-hop neighbor set are received. The received encoded features are aggregated based on an attention mechanism to generate a neighborhood aggregation message for the current time slot; The local input state and the neighborhood aggregation message are input to the shared policy network to obtain the joint action of the current time slot; wherein, the joint action includes channel selection action, transmit power action and packet scheduling priority action; The drone is controlled to perform the joint action to complete the communication scheduling of the current time slot; wherein, the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

[0153] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A UAV communication scheduling method based on multi-agent reinforcement learning, applied to a multi-UAV self-organizing communication network, characterized in that, The method is executed independently by the drones in the multi-drone self-organizing communication network in each time slot, including: Obtain the local input state of the UAV in the current time slot, and construct the one-hop neighbor set of the UAV in the current time slot; The local input state is mapped to encoded features by a policy branch encoder, the encoded features are sent to neighbor drones in the one-hop neighbor set, and the encoded features sent by neighbor drones in the one-hop neighbor set are received. The received encoded features are aggregated based on an attention mechanism to generate a neighborhood aggregation message for the current time slot; The local input state and the neighborhood aggregation message are input to the shared policy network to obtain the joint action of the current time slot; wherein, the joint action includes channel selection action, transmit power action and packet scheduling priority action; The drone is controlled to perform the joint action to complete the communication scheduling of the current time slot; wherein, the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

2. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The step of obtaining the local input status of the UAV in the current time slot includes: The location, velocity, remaining energy, queue status, channel state estimate, and interference measurement of the UAV in the current time slot are collected to obtain the local observation of the current time slot. Extract the temporal aggregation features of the previous time slot from the UAV's local buffer; The local observations of the current time slot and the temporal aggregation features of the previous time slot are used as parallel inputs to form the local input state; wherein, the temporal aggregation features of the previous time slot are the neighborhood temporal summary representation generated and cached on the UAV side in the previous time slot, which is used to supplement the temporal memory.

3. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The construction of the one-hop neighbor set of the UAV in the current time slot includes: The relative distance between the UAV and the detectable node, and the link quality index between the UAV and the detectable node are obtained; wherein, the link quality index includes the signal-to-interference-plus-noise ratio obtained based on pilot estimation; When the relative distance is not greater than a preset distance threshold and the link quality index is not less than a preset link quality threshold, an adjacency relationship for the current time slot is established between the UAV and the detectable node. All nodes that have the adjacency relationship with the drone are included in the one-hop neighbor set.

4. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The aggregation of the received encoded features based on the attention mechanism to generate the neighborhood aggregation message of the current time slot includes: By implementing the query parameters and key parameters in the policy branch aggregation module of the attention mechanism, the attention weight between the encoded features generated by the UAV itself and the encoded features received from the neighboring UAV is calculated. Based on the attention weights and the value parameters in the policy branch aggregation module, the received encoded features sent by the neighboring drones are weighted and summed to obtain the neighborhood aggregation message of the current time slot.

5. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, The control of the UAV to execute the joint action and complete the communication scheduling of the current time slot includes: Based on the channel selection action, switch the spectrum resources occupied by the UAV data transmission; Based on the aforementioned transmission power action, the transmission power level of the UAV is adjusted to simultaneously change the link transmission reliability, the interference leakage intensity to the one-hop neighbor set, and the local transmission energy consumption. Based on the packet scheduling priority action, the service order of different service queues of the UAV is adjusted; After performing the joint action, the number of successfully transmitted bits, the estimated instantaneous co-channel interference, the transmission energy consumption and service delay statistics of this time slot are recorded as feedback sample data for this time slot.

6. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 5, characterized in that, The method further includes: The UAV writes the local input state, the neighborhood aggregation message, the joint action, the instant reward, the local input state of the next time slot, and the constraint statistics composed of the feedback sample data into the UAV's local trajectory buffer to form a trajectory sample; The drone uploads the trajectory samples to the training node so that the training node can update the parameters during the centralized training phase. The UAV receives the updated parameters of the shared policy network from the training node at a preset period, and performs the communication scheduling for subsequent time slots according to the updated parameters of the shared policy network.

7. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 6, characterized in that, The method further includes: Receive the trajectory samples uploaded by each UAV in the multi-UAV self-organizing communication network; Based on the local input states, joint actions, and adjacency relationships of all the UAVs in the same time slot, the global state is reconstructed; The global state is feature-aggregated using a critic branch encoder to generate a global aggregated representation. The global state, the global aggregate representation, and the joint action are input into a centralized critic network to obtain a value assessment result. Based on the value assessment results and the final reward feedback generated by the two-layer constraint linkage adjustment mechanism, the parameters of the shared policy network and the parameters of the centralized critic network are updated, and the updated parameters of the shared policy network are distributed to each UAV.

8. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 7, characterized in that, The shared policy network and the centralized critic network are respectively configured with structurally corresponding but parameter-independent message aggregation branches; wherein, the policy message aggregation branch configured in the shared policy network is used to perform weighted aggregation of the encoded features received from the neighboring drones during the execution phase of the drone to generate the neighborhood aggregation message. The critic message aggregation branch configured in the centralized critic network is used to aggregate the global features corresponding to the global state during the centralized training phase of the training node to generate the global aggregate representation. The parameters of the policy message aggregation branch and the parameters of the critic message aggregation branch are trained independently, and the UAV only runs the policy message aggregation branch during the execution phase.

9. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 7, characterized in that, The steps of generating the final reward feedback by the dual-layer constraint linkage adjustment mechanism include: Statistical windows are established for the interference index, latency index, and energy consumption index included in the constraint statistics of the trajectory sample, and the average index value of the target index within the statistical window is calculated; wherein, the target index includes the interference index, the latency index, or the energy consumption index; According to the Lagrange multiplier adjustment layer, when it is determined that the average index value of the target index exceeds the threshold set by the target index, the Lagrange multiplier configured for the target index is increased to obtain the violation penalty item generated by the Lagrange multiplier. Based on the barrier function pre-suppression layer, and the remaining margin between the average index value of the target index and the threshold set by the target index, the risk suppression penalty term of the target index is calculated. The final reward feedback is obtained by subtracting the violation penalty and the risk suppression penalty from the basic reward; wherein the basic reward is obtained based on a weighted combination of the weighted throughput, the latency index, and the energy consumption index of the current time slot.

10. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 9, characterized in that, The step of calculating the risk suppression penalty term for the target indicator based on the residual margin between the average indicator value and the threshold set for the target indicator, according to the barrier function pre-suppression layer, includes: When the average value of the target indicator exceeds the threshold set by the target indicator in the most recent statistical window, the pre-configured barrier coefficient of the target indicator is maintained or increased. When the average value of the target indicator does not exceed the threshold set by the target indicator within a consecutive preset number of statistical windows, the barrier coefficient is reduced according to a preset attenuation coefficient. Logarithmic operations are performed based on the barrier coefficient and the remaining margin to generate the risk suppression penalty term, so as to pre-suppress the tendency to choose relevant high-risk actions in the shared policy network.

11. The UAV communication scheduling method based on multi-agent reinforcement learning according to claim 7, characterized in that, The updating of the parameters of the shared policy network includes: After a preset number of time slots, the training node calculates the advantage function based on the value assessment result and the final reward feedback; The parameters of the shared policy network are updated once based on the advantage function and the truncated agent objective function; wherein the truncated agent objective function is used to limit the magnitude of a single policy update; When the preset maximum number of training rounds is reached, or when the parameter change of the shared policy network is lower than a preset threshold for several consecutive rounds, the training node saves the parameters of the shared policy network and sends them to the onboard computing unit of each UAV.

12. A UAV communication scheduling device based on multi-agent reinforcement learning, deployed in a multi-UAV self-organizing communication network, characterized in that, The device includes: The observation and acquisition module is used to acquire the local input state of the UAV in the current time slot and construct the one-hop neighbor set of the UAV in the current time slot; The feature encoding and message aggregation module is used to map the local input state into encoded features using the policy branch encoder, send the encoded features to neighbor drones in the one-hop neighbor set, and receive encoded features sent by neighbor drones in the one-hop neighbor set. The aggregation module is used to aggregate the coded features received from the neighboring drones based on an attention mechanism to generate the neighborhood aggregation message for the current time slot. An input module is used to input the local input state and the neighborhood aggregation message into the shared policy network to obtain the joint action of the current time slot; wherein, the joint action includes channel selection action, transmit power action and packet scheduling priority action; The scheduling module is used to control the UAV to execute the joint action and complete the communication scheduling of the current time slot; wherein, the shared policy network is a network formed after loading the updated parameters issued by the training nodes during the centralized training phase.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the UAV communication scheduling method based on multi-agent reinforcement learning as described in any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the UAV communication scheduling method based on multi-agent reinforcement learning as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the UAV communication scheduling method based on multi-agent reinforcement learning as described in any one of claims 1 to 11.

Images