A method for jointly dynamic allocation of spectrum and power based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's spectrum and power joint dynamic allocation method.The method includes: randomly deploying communication equipment node and available communication channel, generates environment data;Based on the interference coupling relationship between the interference of power quantization equipment, constructs edge set;Extract the state information of each node, generate network graph;Network graph is input to GAT-MAPPO joint decision framework, and deep node characteristics are obtained;The action strategy of each node is generated and the global state value is evaluated;The joint action corresponding to each communication equipment is generated;The parameters of GAT-MAPPO joint decision framework are updated until convergence, and a distributed spectrum and power joint allocation strategy is generated;The application solves the technical problems in the prior art of wireless communication resource allocation method in high dynamic complex network such as unmanned aerial vehicle and tactical communication, which cannot accurately model the dynamic change of the topological interference relationship between nodes, resulting in the inability to optimize the spectrum and power, low system throughput and high power consumption.

Description

Technical Field

[0001] This invention relates to the technical field of wireless communication and intelligent decision-making, and more specifically, to a method for joint dynamic allocation of spectrum and power based on reinforcement learning. Background Technology

[0002] With the explosive growth of wireless communication devices, spectrum resources are becoming increasingly scarce. Co-channel interference generated when multiple devices share limited spectrum resources has become a core bottleneck restricting the communication performance of the system. In the aforementioned high-dynamic communication scenarios, the high mobility of nodes, the rapid time-varying network topology, and the large fluctuations in service load place far more stringent requirements on the real-time performance and adaptive capabilities of resource allocation than on traditional cellular networks.

[0003] Existing solutions to the spectrum resource allocation problem have significant limitations: traditional static / semi-static and centralized heuristic algorithms (such as game theory and convex optimization) suffer from a surge in computational complexity in highly maneuverable and topology-changing scenarios, making it difficult to adapt to environmental changes in real time and resulting in weak interference coordination capabilities; single-agent reinforcement learning methods face non-stationarity and state space explosion problems in multi-node environments, and traditional vector inputs are insufficient to characterize topological interference relationships, leading to insufficient decision-making accuracy; existing graph neural network improvement schemes mostly optimize spectrum allocation only, lacking joint optimization with power control, and have insufficient integration of graph coding and multi-agent decision-making frameworks, and rely on central node decisions, making them difficult to adapt to decentralized communication scenarios.

[0004] In summary, existing wireless communication resource allocation methods struggle to simultaneously meet the demands of dynamic topology adaptation, accurate interference modeling, joint spectrum and power optimization, and distributed deployment. They generally suffer from low system throughput, high power consumption, and insufficient service QoS guarantees. Therefore, a joint dynamic spectrum and power allocation method adapted to highly dynamic and complex communication scenarios is urgently needed. Summary of the Invention

[0005] This invention provides a reinforcement learning-based method for joint dynamic allocation of spectrum and power, which at least addresses the technical problems of existing wireless communication resource allocation methods in highly dynamic and complex networks such as UAVs and tactical communications, where it is difficult to accurately model the dynamically changing topological interference relationships between nodes, resulting in the inability to coordinate spectrum and power optimization, low system throughput, and high power overhead.

[0006] According to one aspect of the present invention, a reinforcement learning-based method for joint dynamic allocation of spectrum and power is provided. The method may include: randomly deploying communication device nodes and available communication channels within a preset communication area; calculating channel gain based on the relative positions of the communication device nodes; and generating environmental data including channel gain, co-channel interference power, and communication rate by combining path loss; mapping the communication device nodes to a set of nodes in a graph structure; quantifying the interference coupling relationship between devices based on the co-channel interference power in the environmental data; and constructing an edge set representing potential interference relationships; extracting state information of each node based on the environmental data; generating node features; constructing an initial feature matrix based on the node features; and generating a network graph composed of node features and an edge set, wherein the node features include the current link channel gain, co-channel interference power, currently occupied channel number, current transmit power, and minimum rate requirement. The network graph is input into the GAT-MAPPO joint decision framework. The node features are encoded using the topological interference relationships contained in the edge set to obtain deep node features that fuse topological interference information. Based on the deep node features, action policies for each node are generated and the global state value is evaluated. Based on the action policies, joint actions corresponding to each communication device are generated. The generated joint actions, the deep node features of the current node, and the global state value are associated and encapsulated as an interaction sample and stored in the experience replay pool. The global state space required for training is defined, a multi-objective reward function is designed, and the parameters of the GAT-MAPPO joint decision framework are iteratively updated using interaction samples until convergence, generating a distributed spectrum and power joint allocation strategy that can be deployed on various communication devices.

[0007] Optionally, the node features are encoded using the topological interference relationships contained in the edge set to obtain deep node features that fuse topological interference information. This includes: performing a linear transformation on the node features to obtain intermediate features; determining the connection relationships between nodes based on the edge set and calculating the association weights between connected nodes using an attention mechanism; and weighting and aggregating the intermediate features of neighboring nodes with the association weights to obtain deep node features.

[0008] Optionally, constructing a set of edges representing potential interference relationships includes: traversing any two communication device nodes, and if the first node and the second node occupy the same communication channel and the interference power between them is greater than a preset threshold, then constructing an edge between the first node and the second node.

[0009] Optionally, the GAT-MAPPO joint decision-making framework specifically includes a graph attention network encoder, a distributed actor network, and a centralized critic network: the graph attention network encoder is used to receive the network graph, encode the node features, and output deep node features; the distributed actor network is used to receive deep node features, generate action policies for each node, and output joint actions; the centralized critic network is used to receive deep node features of all nodes, evaluate the global state value, and guide policy updates.

[0010] Optionally, the joint action includes channel selection action and transmit power control action; the distributed Actor network also includes: for each communication device node, based on the input deep node characteristics, outputting the corresponding channel number and discretized power level respectively.

[0011] Optionally, the parameters of the GAT-MAPPO joint decision-making framework are updated iteratively, including: calculating the advantage function based on interaction samples; updating the parameters of the distributed Actor network by pruning the policy objective function; updating the parameters of the centralized Critic network by minimizing the value loss function; and synchronously updating the parameters of the graph attention network encoder.

[0012] The beneficial effects of this invention are: This invention proposes a joint dynamic allocation method for spectrum and power based on reinforcement learning. By constructing a GAT-MAPPO joint decision-making framework and utilizing the topological reasoning capability of Graph Attention Networks (GAT), it accurately quantifies the implicit interference coupling relationships between nodes in highly dynamic and complex networks, overcoming the deficiency of traditional vector inputs that cannot perceive network structure. On this basis, relying on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm, it realizes the joint decision-making of spectrum channel selection and transmit power control, breaking through the limitation of existing schemes that can only optimize resources in a single dimension. In addition, the "centralized training, distributed execution" architecture adopted in this invention enables the trained model to be deployed without the need for a central node to manage it. Each communication node can make independent decisions based on local observations, which greatly improves the system's resilience and deployment flexibility in non-steady-state environments such as UAV swarms and tactical mobile communications, perfectly balancing the multiple design goals of high throughput, low power consumption, and strong real-time performance. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of a reinforcement learning-based joint dynamic allocation method for spectrum and power according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a network topology according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a GAT-MAPPO algorithm according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a GAT-MAPPO algorithm flow according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a reward curve according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a system throughput curve according to an embodiment of the present invention; Figure 7 This is a schematic diagram of an average transmit power curve according to an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example According to an embodiment of the present invention, a method for joint dynamic allocation of spectrum and power based on reinforcement learning is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system containing at least one set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0017] Figure 1This is a flowchart of a reinforcement learning-based joint dynamic allocation method for spectrum and power according to an embodiment of the present invention. The proposed GAT-MAPPO joint decision framework includes: a graph attention network (GAT) encoder, a distributed actor network, and a centralized critic network. The GAT encoder is responsible for receiving a network graph containing node features and interference edges, and extracting deep features that fuse topological interference information; the distributed actor network outputs the action policies of each node based on the deep features, realizing decentralized execution; the centralized critic network evaluates the state value based on global features and guides policy updates; the three work together end-to-end, forming the core architecture of the present invention; as follows. Figure 1 As shown, the method may include the following steps: Step S101: Randomly deploy communication device nodes and available communication channels within a preset communication area, calculate channel gain based on the relative positions between communication device nodes, and generate environmental data including channel gain, co-channel interference power, and communication rate by combining path loss.

[0018] In the technical solution provided by step S101 of the present invention, N communication device nodes are randomly deployed and K available orthogonal channels are predefined within a preset communication area (e.g., a 500m×500m area, which is only an example and does not limit the specific size of the communication area); based on the Euclidean distance between the nodes... Calculate large-scale path loss In conjunction with small-scale fading characteristics, based on the following computing equipment i To the equipment j Channel gain between :

[0019] in, As a random variable following a Rayleigh distribution, it is used to simulate the fast signal fading effect caused by multipath propagation. The introduction of this parameter enhances the model's robustness against complex electromagnetic environments.

[0020] Rayleigh Rayleigh Let the channel amplitude random variable follow a Rayleigh distribution. Based on the channel gain, the device amplitude is further calculated according to the following formula. i Received co-channel interference power:

[0021] This formula quantifies the impact of other devices on the device. iThe resulting cumulative interference forms the physical basis for constructing the network topology edge set; after obtaining the channel gain matrix, based on the following computing devices... i Signal-to-interference-plus-noise ratio at the receiver :

[0022] in, Indicates device i The transmission power; For equipment i This link channel gain; For equipment j The transmission power; , respectively equipment i , j Selected channel number; This is a co-channel indication function; it takes the value 1 if the condition is met, and 0 otherwise. The receiver noise power; based on Further calculation equipment i instantaneous communication rate : ) The total throughput is finally obtained by summing the results according to the following formula. : .

[0023] Step S102: Map the communication device nodes to a set of nodes in a graph structure. Quantify the interference coupling relationship between devices based on the co-channel interference power in the environmental data, and construct an edge set representing the potential interference relationship. Extract the state information of each node based on the environmental data to generate node features. Construct an initial feature matrix based on the node features to generate a network graph composed of node features and edge set. The node features include the current link channel gain, co-channel interference power, currently occupied channel number, current transmit power, and minimum rate requirement.

[0024] In the technical solution provided by step S102 of the present invention, the N communication devices deployed in step S101 are mapped as a graph structure. G=(V,E) The set of nodes V; for each node i Multidimensional state information is extracted based on the following formula to construct node features. x i :

[0025] in, For equipment i The local link device gain reflects the device's... i The signal quality at its receiving end, For equipment i Received co-channel interference power, For equipment i The channel number used, For equipment i The transmission power, For equipment i The minimum rate requirement; the node features of all nodes are stacked row by row to form an initial feature matrix, which serves as the node attribute input for the graph structure.

[0026] Figure 2 This is a schematic diagram of a network topology according to an embodiment of the present invention, such as... Figure 2 As shown, based on the interference coupling relationship between devices in the co-channel interference power quantization of environmental data, an edge set E is constructed. All node pairs (i,j) are traversed. If both occupy the same communication channel (i.e., co-channel indicator function),... If the value is 1, then a potential interference is determined, and an undirected edge is constructed between the two nodes. Figure 2 In this model, nodes represent communication devices, and lines represent interference edges. The degree of a node reflects the complexity of its interference, and the distribution density of edges intuitively reflects the complexity of the local interference environment. Compared with traditional static topology assumptions, this model is more adaptable to the topological change characteristics of highly dynamic networks. Finally, the initial feature matrix and edge set are coupled to generate a complete network graph. This network graph contains the multidimensional state features of nodes and explicitly encodes the interference interaction relationships between nodes.

[0027] Step S103: Input the network graph into the GAT-MAPPO joint decision framework, and use the topological interference relationship contained in the edge set to encode the node features to obtain deep node features that fuse topological interference information.

[0028] In the technical solution provided in step S103 of the present invention, the graph attention network encoder in the GAT-MAPPO joint decision framework inputs the network graph, and determines the range of neighboring nodes of each node based on the edge set constructed in step S102. Here, the edge set serves as a mask for topological interference relationships, allowing only node pairs with interference coupling to participate in subsequent feature interactions. This aligns with the physical law that only devices on the same channel have interference in communication scenarios, thus avoiding invalid calculations of irrelevant nodes.

[0029] The graph attention network encoder first processes the input node features. x i A linear transformation is performed, and the original node features of dimension 5 are mapped to a high-dimensional latent space using a trainable weight matrix W according to the following formula, to obtain intermediate features. h i :

[0030] This operation can unify the numerical scale of heterogeneous physical features, providing a suitable input basis for subsequent attention coefficient calculation; based on the neighbor range defined by the edge set, for each node... i and all its neighboring nodes j The attention weight calculation process is as follows: based on the following formula using the trainable parameter vector... With the LeakyReLU linear unit activation function with leakage correction, the computation nodes i with neighbors j Unnormalized attention coefficient : ) in, Represents a node i with neighbors j The intermediate features are concatenated, and LeakyReLU(·) is the linear unit activation function with leakage correction. The parameter vector is a trainable vector; then, according to the following formula, the softmax function is applied to the same node. i All unnormalized attention coefficients are normalized to obtain normalized attention weights with values ​​ranging from [0,1]. :

[0031] in, For nodes i Assigned to neighboring nodes j Normalized attention weights; It is an exponential function; Represents a node i The set of neighbors; k Index the neighboring nodes; For nodes i with neighboring nodes k The unnormalized attention coefficients between nodes; the physical meaning of this weight is the node... i When perceiving the environment, neighbors j The degree of importance attached to it: if the neighbor j For nodes i The higher the co-channel interference power, the better. The larger the value, the more adaptive the quantification of the interference effect is achieved, eliminating the need for manually preset interference priority rules; the final node representation is obtained by weighted aggregation of neighbor features. i Deep node features :

[0032] in, As a non-linear activation function, this deep node feature overcomes the limitation of the original node features in step S102, which only contain local single-hop states. It realizes the transmission of multi-hop interference information through the topological relationship carried by the edge set: for example, although a node has no direct co-channel connection with the distant interference source, it can perceive the indirect interference impact through the feature transmission of intermediate neighbor nodes. Figure 3 This is a schematic diagram of a GAT-MAPPO algorithm according to an embodiment of the present invention, as shown below. Figure 3 As shown, the obtained deep node features will be used as a unified input and fed to the distributed Actor network and the centralized Critic network respectively: the former generates the action policy of each node based on the features, and the latter evaluates the global state value based on the deep node features of all nodes, providing a basis for subsequent policy updates.

[0033] Step S104: Based on the deep node features, generate the action strategy for each node and evaluate the global state value.

[0034] In the technical solution provided by step S104 of the present invention, Figure 4 This is a schematic diagram of a GAT-MAPPO algorithm flow according to an embodiment of the present invention, as shown below. Figure 4 As shown, deep node features serve as the input to the distributed Actor network. Each communication device node corresponds to an independent Actor branch, receiving only its own deep node features as the basis for decision-making. It can make decisions independently without obtaining global network information, thus adapting to the operational requirements of decentralized scenarios such as drone swarms and communications.

[0035] Suppose that the state of agent i at time t is The action is The Actor network outputs the probability distribution of joint actions through the following policy function:

[0036] in, For policy network parameters, That is, the current time node. i The deep node features, For nodes i The pending actions, ( ) represents the policy network parameters The policy network, with its joint actions, comprises two parts: channel selection and transmit power control, namely:

[0037] in, Indicates the selected first i There are communication channels, if the system has K If there are 1 communication channel, then:

[0038] Transmit power of device i The discretized power level is:

[0039] Where L is the preset power level, and the final motion space is the Cartesian product of the two: ×

[0040] Outputting probability distributions instead of deterministic actions can provide a basis for exploration and sampling in subsequent training phases, avoiding the policy from getting trapped in local optima.

[0041] The evaluation of the global state value is performed only during the training phase. During the deployment phase, the Critic network does not participate in decision-making; only the distributed Actor network is retained. This avoids the inflexibility deficiency of existing solutions that rely on real-time decision-making by a central node, while also improving training stability by leveraging global information. The centralized Critic network receives the global state representation after aggregating the deep node features of all nodes and outputs the expected cumulative reward under the current global state, i.e., the global state value V(s). t The global state value is the subsequent calculation of the advantage function A. t The benchmark, combined with the objective function of the pruning strategy, can effectively limit the policy update amplitude, avoid training oscillations, and optimize the corresponding objective. :

[0042] in, For the dominant function, This is the cutting factor. The ratio of the probabilities of the new and old strategies:

[0043] in, This is the current strategy; The old strategy before the update, the global assessment of the Critic network, covers the entire network's interference topology information. Compared to single-node value assessment, it reflects the overall system operating status better, providing accurate guiding signals for strategy updates. Figure 3 The global value evaluation logic of the Critic network. Action policy and global state value together constitute the core elements of subsequent interaction samples. After sampling, the action policy yields deterministic joint actions. After execution, these actions, combined with the reward signal from environmental feedback, are used together with the global state value to calculate the advantage function, supporting the iterative update of network parameters.

[0044] Step S105: Based on the action strategy, generate joint actions corresponding to each communication device, and encapsulate the generated joint actions, the deep node features of the current node, and the global state value as an interaction sample and store them in the experience replay pool.

[0045] In the technical solution provided in step S105 of the present invention, corresponding to Figure 4 The core nodes in the algorithm flow shown are the environment executing actions and storing interaction samples. The action policy output by the distributed Actor network is the Softmax probability distribution of the joint actions of each node. Sampling is required based on this distribution to generate deterministic joint actions. That is, for each node i, the action policy is determined from its strategy. Mid-sampling yields specific actions The joint actions of all nodes constitute the system-level action vector. The joint decision-making logic of this process is expressed as follows: ) in, The joint action of all agents at time t; Let be the initial system state at time t; The topological state representation encoded by the graph attention network is the deep node feature set output in step S103. The sampling operation introduces exploration noise to avoid the policy getting trapped in local optima and to ensure the diversity of policies during the training phase.

[0046] After the joint action is generated, it will be executed and interact with the wireless communication environment to obtain an immediate reward from the environment. The reward calculation is based on the following multi-objective reward function:

[0047] in, , , It is a constant used to balance system performance and power consumption. The nodes calculated in step S101 i The current rate, This is the current transmission power. To ensure communication quality and meet minimum speed requirements, a penalty is imposed if the device speed falls below the minimum requirement. This is addressed by introducing the following QoS constraint:

[0048] This reward function simultaneously addresses the three objectives of increasing total system throughput, QoS violation penalties, and reducing power consumption, providing multi-dimensional optimization guidance for policy updates.

[0049] After determining the reward function, correlation encapsulation is performed, which encapsulates the current network state (based on the deep node feature set). The combined action of representation and sampling Instant rewards for environmental feedback r t The set of deep node features at the next moment after the action is performed. and the global state value V output in step S104. s t ) is bound to a structured interaction sample: ( , , r t, , V( s t This encapsulation logic ensures that the sample contains a complete causal chain of state transition, action execution, reward feedback, and value assessment, providing complete data support for subsequent advantage function calculation.

[0050] The encapsulated samples will be stored in the experience replay pool. The experience replay pool uses a uniform random sampling or priority sampling mechanism to extract historical samples in batches during the training phase for network parameter updates, breaking the correlation of time series and improving training stability.

[0051] Step S106: Define the global state space required for training, design a multi-objective reward function, and use interactive samples to iteratively update the parameters of the GAT-MAPPO joint decision framework until convergence, thereby generating a distributed spectrum and power joint allocation strategy that can be deployed on various communication devices.

[0052] In the technical solution provided in step S106 of the present invention, the global state is composed of the node features of all nodes and the edge set. The node features cover five dimensions: channel gain of the current link, co-channel interference power, currently occupied channel number, current transmit power, and minimum rate requirement. The edge set represents the co-channel interference coupling relationship between devices. Together, they constitute the topological state representation of the input of the adaptation graph attention network. The dimensions can be dynamically adjusted with the number of network nodes, which meets the needs of scenarios with dynamically changing network scale, such as UAV swarms and tactical communications. Figure 2 The network topology diagram contains complete information on node attributes and interference connections.

[0053] The design employs a multi-objective reward function with weighted multi-objectives. The first objective incentivizes an increase in the system's total throughput. The second objective imposes QoS violation penalties on nodes that fail to meet the minimum rate requirement. The third objective suppresses the total transmit power consumption of the entire network. The weights of these three objectives can be flexibly adjusted according to the performance preferences of the scenario.

[0054] Subsequently, iterative parameter updates are performed: interaction samples are batch-extracted from the experience replay pool. First, the advantage function is calculated to measure the relative merits of the current joint action, serving as a guide for policy updates. Then, the parameters of the distributed Actor network are updated based on the MAPPO pruning mechanism. By limiting the magnitude of single-step policy updates, training oscillations are avoided, ensuring convergence stability. Simultaneously, the parameters of the centralized Critic network are updated to minimize value loss, improving the accuracy of global state value evaluation. The above update process is simultaneously applied to the parameters of the graph attention network encoder, achieving synergistic optimization of GAT feature extraction capabilities and MAPPO decision-making capabilities. Figure 3 The linkage training logic of the three components.

[0055] The iterative process continues until the termination conditions are met: either the preset maximum number of training rounds is reached, or the reward value converges to a stable range; finally, a deployable distributed spectrum and power joint allocation strategy is generated: after training, the centralized Critic network is stripped away, and only the distributed Actor network is retained. Each communication device node loads its own Actor subnetwork. The Actor network includes multiple Actor subnetworks, and each communication device node corresponds to one Actor subnetwork. During the deployment phase, the joint action of channel selection and power control can be independently output based on the locally observed node characteristics. There is no need for global topology information and central node management. It can adapt to the millisecond-level adaptive interference coordination requirements of non-steady-state scenarios such as decentralized communication and highly mobile UAV swarms, and takes into account multiple design goals such as high throughput, low power consumption and strong deployment flexibility.

[0056] The method described in this embodiment will be further described below.

[0057] As an optional embodiment, step S103 involves encoding the node features using the topological interference relationships contained in the edge set to obtain deep node features that fuse topological interference information. This includes: performing a linear transformation on the node features to obtain intermediate features; determining the connection relationships between nodes based on the edge set and calculating the association weights between connected nodes using an attention mechanism; and weighting and aggregating the intermediate features of neighboring nodes using the association weights to obtain deep node features.

[0058] In this embodiment, a linear transformation is performed on the node features, and the 5-dimensional node features obtained in step S102 are mapped to a high-dimensional latent space through a trainable weight matrix to obtain intermediate features. This operation can unify physical features with different dimensions (such as channel gain, interference power, rate requirements, etc.) into standardized features suitable for attention calculation, providing a compliant input basis for subsequent weight calculation.

[0059] The connection relationship between nodes is determined based on the edge set. Only node pairs with interfering edges are included in the calculation scope, which conforms to the physical law that only co-channel devices interfere in wireless communication scenarios and avoids invalid calculations of irrelevant nodes. On this basis, the association weight between connected nodes is calculated using an attention mechanism. The normalized association weight is obtained by calculating the attention coefficient and normalizing the process in sequence. This weight adaptively reflects the importance that a node attaches to the interference from its neighbors: if the co-channel interference power of the neighbor node to the current node is higher, the corresponding association weight value is larger. The quantification of interference impact can be achieved without manually pre-setting interference priority rules.

[0060] Finally, the intermediate features of neighboring nodes are weighted and aggregated using association weights. After processing by a non-linear activation function, a deep node feature that integrates topological interference information is output. This feature breaks through the limitation that the original node features only contain local single-hop states. It can integrate multi-hop interference information through topological propagation. For example, distant interference sources without direct connections can be perceived by the current node through the feature transmission of intermediate neighbors.

[0061] As an optional implementation, the GAT-MAPPO joint decision-making framework specifically includes a graph attention network encoder, a distributed actor network, and a centralized critic network: the graph attention network encoder is used to receive the network graph, encode the node features, and output deep node features; the distributed actor network is used to receive deep node features, generate action policies for each node, and output joint actions; the centralized critic network is used to receive deep node features of all nodes, evaluate the global state value, and guide policy updates.

[0062] In this embodiment, corresponding Figure 3 The diagram of the GAT-MAPPO algorithm shows that the three components adopt an end-to-end collaborative training mode, with clear feature flow and functional division: The graph attention network encoder, as the front-end feature extraction module of the framework, receives the network graph composed of node features and edge sets output from step S102. Based on the GAT encoding logic, it performs topology enhancement processing on the node features, transforming the original features containing only the local state of a single node into deep node features that integrate multi-hop interference associations. This not only preserves the single-node attributes such as the local link gain and interference power, but also embeds global interference coupling information through the topological relationships carried by the edge sets. From the feature level, it solves the defect that traditional deep reinforcement learning cannot characterize the network topology structure by using flat vector input.

[0063] As the decision-making and execution branch of the framework, the distributed Actor network receives the deep node features of a single node output by the encoder, and outputs the probability distribution of the joint actions of each node based on the policy function logic. The joint actions cover two types of decisions: channel selection and discretized power control. The characteristic of being able to complete the decision based solely on local features is in line with the requirements of the distributed execution architecture. It can be adapted to the deployment needs of high-dynamic scenarios such as decentralized tactical communication and UAV swarms. The sampling based on the probability distribution during the training phase can also introduce exploration, avoiding the policy from getting trapped in local optima.

[0064] The centralized Critic network, serving as a training auxiliary branch of the framework, is only activated during the training phase. It receives a global state representation aggregated from the deep node features of all nodes and outputs the expected cumulative reward of the current state based on global value evaluation logic. This global state value serves as the benchmark for calculating the advantage function. Combined with the MAPPO pruning mechanism, it can effectively limit the policy update magnitude and significantly reduce training oscillations caused by environmental non-stationarity in multi-agent environments. During the deployment phase, this network will be completely decoupled, and only the distributed Actor network will be retained to run independently, balancing the stability of the training phase with the flexibility of the deployment phase.

[0065] As an optional implementation, the parameters of the GAT-MAPPO joint decision-making framework are iteratively updated, including: calculating the advantage function based on interaction samples; updating the parameters of the distributed Actor network by pruning the policy objective function; updating the parameters of the centralized Critic network by minimizing the value loss function; and synchronously updating the parameters of the graph attention network encoder.

[0066] In this embodiment, corresponding Figure 4 The core node in the policy update phase of the algorithm flow shown relies on the structured interactive samples stored in the experience replay pool in step S105 to carry out full-component collaborative optimization of the GAT-MAPPO joint decision-making framework; the advantage function is calculated based on the interactive samples. This function takes the immediate reward, the current global state value, and the next global state value recorded in the sample as inputs, quantifies the profit deviation of the current joint action relative to the average policy, avoids the global optimization deviation caused by the local reward of a single node, and provides accurate guiding signals for parameter updates.

[0067] The parameters of the distributed Actor network are updated by pruning the policy objective function. The MAPPO pruning mechanism is used to limit the fluctuation range of the probability ratio between the old and new policies, avoiding training oscillations caused by excessive single-step policy updates and supporting stable policy convergence. At the same time, the parameters of the centralized Critic network are updated by minimizing the value loss function, reducing the error between the global state value output by the Critic network and the actual cumulative reward, improving the accuracy of global value assessment, and providing a more reliable benchmark signal for Actor network updates. The synergy of these two methods effectively solves the convergence difficulty caused by environmental non-stationarity in multi-agent reinforcement learning.

[0068] Finally, the parameters of the graph attention network encoder are updated synchronously. Since the deep node features output by the encoder are the common input of the Actor and Critic networks, synchronous updates ensure the co-evolution of feature extraction and decision evaluation capabilities, allowing deep node features to more accurately integrate topological interference information and gradually adapt to the dynamically changing network environment. The entire update process is executed iteratively until the termination condition is met.

[0069] Experimental section: To verify the effectiveness of this invention in the field of wireless communication and intelligent decision-making technology, a simulation design was performed using a Python 3.7 + PyTorch environment, and the simulation results were analyzed. Table 1 shows the simulation parameter settings.

[0070] Table 1 Simulation Parameter Settings Multiple communication device nodes, such as 12 communication devices, are deployed within a 500m*500m communication area, and 4 available communication channels are set up. The nodes are randomly distributed within the area, and a wireless channel model is established by calculating the distance between the nodes. The channel gain is determined by both the path loss model and the Rayleigh fading model. The calculation method has been explained above and will not be repeated here. Figure 5 This is a schematic diagram of a reward curve according to an embodiment of the present invention, corresponding to... Figure 5 As shown, the initial training phase (first 1000 rounds) is the policy exploration phase, with the average reward fluctuating and increasing from about 20. The phase from 1000 to 6000 rounds is the rapid convergence phase, with the reward value climbing from 30 to about 75, without significant training oscillations. After 6000 rounds, the phase enters a stable convergence range, with the average reward fluctuating slightly within the range of 70 to 85. This verifies the effect of the aforementioned pruning policy objective function on improving the stability of multi-agent training and proves that the method of this invention can effectively avoid the training divergence problem caused by environmental non-stationarity.

[0071] Figure 6 This is a schematic diagram of a system throughput curve according to an embodiment of the present invention, corresponding to... Figure 6As shown, the total system throughput under the untrained initial policy is approximately 6 Mbps. As training progresses, the GAT encoder gradually and accurately extracts topological interference features, and the MAPPO policy network learns an efficient spectrum reuse mode, resulting in a continuous increase in throughput. After 6000 rounds, the throughput stabilizes in the range of 14-16 Mbps, an increase of 133%-167% compared to the initial value, with a fluctuation range of less than 5%. This demonstrates that the method of this invention can fully tap the potential of spectrum resources and effectively improve the system transmission efficiency under complex interference environments.

[0072] Figure 7 This is a schematic diagram of an average transmit power curve according to an embodiment of the present invention, corresponding to Figure 7 As shown, under the initial strategy, the average transmission power is approximately 0.65W to maintain basic communication. During the training process, the multi-objective reward function guides the strategy to prioritize the low-power transmission mode, and the average transmission power continues to decrease. After 6000 rounds, it stabilizes at around 0.48W, which is about 26% lower than the initial value. Moreover, there is no problem of increased QoS default rate due to excessively low power. This proves that the method of the present invention can achieve optimal energy efficiency while ensuring communication quality, and solves the performance contradiction of the traditional method that "increasing throughput must increase power".

[0073] In summary, the simulation results fully verify the comprehensive advantages of the method of the present invention in improving system throughput, reducing power consumption, adapting to dynamic topology, and ensuring training convergence, which can meet the resource allocation requirements of complex wireless communication scenarios.

[0074] In this embodiment of the invention, communication device nodes and available communication channels are randomly deployed within a preset communication area. Channel gain is calculated based on the relative positions of the communication device nodes, and path loss is combined to generate environmental data containing channel gain, co-channel interference power, and communication rate. The communication device nodes are mapped to a set of nodes in a graph structure. The interference coupling relationship between devices is quantified based on the co-channel interference power in the environmental data, and an edge set representing potential interference relationships is constructed. State information of each node is extracted based on the environmental data to generate node features. An initial feature matrix is ​​constructed based on the node features, generating a network graph composed of node features and the edge set. The network graph is input into the GAT-MAPPO joint decision framework, and the node features are encoded using the topological interference relationships contained in the edge set to obtain deep node features that fuse topological interference information. Based on the deep node features, action strategies for each node are generated, and the global state value is evaluated. Based on the action strategies, action strategies for each communication device are generated. The joint action corresponding to the communication device is associated and encapsulated with the generated joint action, the deep node features of the current node, and the global state value, and stored as an interaction sample in the experience replay pool. The global state space required for training is defined, a multi-objective reward function is designed, and the parameters of the GAT-MAPPO joint decision framework are iteratively updated using interaction samples until convergence, generating a distributed spectrum and power joint allocation strategy that can be deployed on various communication devices. The interference coupling relationship between devices is quantified by using the edge set through the graph attention mechanism, and the node features containing only local state are encoded into deep features that fuse topological interference information, solving the technical problem that traditional vector input cannot perceive the network structure. Relying on the centralized training and distributed execution architecture, the global state value is used to guide the policy update during the training phase, ensuring the convergence stability of multi-agent training. During deployment, only a distributed Actor network is needed to achieve millisecond-level independent decision-making, which has the advantages of high throughput and low power consumption.

[0075] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0076] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a first processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0080] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for joint dynamic allocation of spectrum and power based on reinforcement learning, characterized in that, include: Within a preset communication area, communication device nodes and available communication channels are randomly deployed. Channel gain is calculated based on the relative positions of the communication device nodes. Combined with path loss, environmental data including channel gain, co-channel interference power, and communication rate are generated. The communication device nodes are mapped to a set of nodes in a graph structure. The interference coupling relationship between devices is quantified based on the co-channel interference power in the environmental data, and an edge set representing the potential interference relationship is constructed. The state information of each node is extracted based on the environmental data to generate node features. An initial feature matrix is ​​constructed based on the node features, and a network graph composed of the node features and the edge set is generated. The node features include the current link channel gain, co-channel interference power, currently occupied channel number, current transmit power, and minimum rate requirement. The network graph is input into the GAT-MAPPO joint decision framework, and the node features are encoded using the topological interference relationships contained in the edge set to obtain deep node features that fuse topological interference information. Based on the deep node features, action strategies for each node are generated and the global state value is evaluated. Based on the action strategy, a joint action corresponding to each communication device is generated. The generated joint action, the deep node features of the current node, and the global state value are associated and encapsulated, and stored as an interaction sample in the experience replay pool. Define the global state space required for training, design a multi-objective reward function, and use the interactive samples to iteratively update the parameters of the GAT-MAPPO joint decision framework until convergence, thereby generating a distributed spectrum and power joint allocation strategy that can be deployed on various communication devices.

2. The method according to claim 1, characterized in that, The process of encoding the node features using the topological interference relationships contained in the edge set to obtain deep node features that fuse topological interference information includes: Perform a linear transformation on the node features to obtain intermediate features; Based on the set of edges, the connection relationships between nodes are determined, and the association weights between connected nodes are calculated using an attention mechanism. The intermediate features of neighboring nodes are weighted and aggregated using the association weights to obtain the deep node features.

3. The method according to claim 1, characterized in that, The construction of the edge set representing potential interference relationships includes: If any two communication device nodes are traversed, and the first node and the second node occupy the same communication channel and the interference power between them is greater than a preset threshold, then an edge is constructed between the first node and the second node.

4. The method according to claim 1, characterized in that, The GAT-MAPPO joint decision-making framework specifically includes a graph attention network encoder, a distributed actor network, and a centralized critic network. The graph attention network encoder is used to receive the network graph, encode the node features, and output the deep node features; The distributed Actor network is used to receive the deep node features, generate action strategies for each node, and output the joint action; The centralized Critic network is used to receive the deep node features of all nodes, evaluate the global state value, and guide policy updates.

5. The method according to claim 4, characterized in that, The joint action includes channel selection and transmit power control actions; the distributed Actor network also includes: For each communication device node, based on the input deep node characteristics, the corresponding channel number and discretized power level are output respectively.

6. The method according to claim 4, characterized in that, The iterative update of the parameters of the GAT-MAPPO joint decision-making framework includes: Based on the interaction samples, calculate the dominance function; The parameters of the distributed Actor network are updated by the objective function of the pruning strategy; The parameters of the centralized Critic network are updated by minimizing the value loss function; The parameters of the graph attention network encoder are updated synchronously.

Images