A beam hopping control method and training method based on deep reinforcement learning

Abstract

The application provides a beam hopping control method and a training method based on deep reinforcement learning, and relates to the technical field of satellite communication; including obtaining the geographical position topological relationship between each service cell in the simulation environment and the global environment state at the current moment; using a graph neural network module to perform graph convolution operation on the geographical position topological relationship and the global environment state to extract the spatial interference features between each beam; based on the spatial interference features, dynamically determining the fusion weight parameters of the Mixing network at the current moment through a preset mapping function; using the fusion weight parameters to weight and fuse the local Q values output by each Agent network to obtain the global joint Q value at the current moment; based on the loss between the global joint Q value and the target Q value, iteratively updating the network parameters of the neural network until the preset iteration end condition is met. The application reduces the same frequency interference between beams in the full frequency multiplexing scene.

Description

Technical Field

[0001] This invention relates to the field of satellite communication technology, and more specifically, to a beam hopping control method and training method based on deep reinforcement learning. Background Technology

[0002] With the development of low-orbit satellites, although full-frequency reuse technology can greatly improve spectrum efficiency, it has caused serious co-channel interference problems.

[0003] In existing distributed beam control schemes based on deep reinforcement learning, agents typically rely solely on local observations for independent decision-making, lacking awareness of spatial interference between adjacent beams. This leads to inaccurate collaborative weights when aggregating global Q-values, making it prone to getting trapped in local optima and hindering the achievement of cooperative interference avoidance among multiple beams.

[0004] Therefore, a beam hopping control method is urgently needed to solve the problem of strong interference coupling under full-frequency reuse. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a beam hopping control method and training method based on deep reinforcement learning, so as to reduce co-channel interference between beams in full-frequency reuse scenarios and improve the accuracy of beam hopping pointing.

[0006] Firstly, a training method for a neural network to control beam hopping is provided, applicable to a low-Earth orbit (LEO) satellite full-frequency reuse scenario; the neural network includes several agent networks and a mixing network with embedded graph neural network modules, each agent network controlling the beam of a LEO satellite; the method includes: Obtain the geographical location topology between each serving cell in the simulation environment and the current global environment state; A graph neural network module is used to perform graph convolution operations on the topological relationships of geographical locations and the global environmental state in order to extract spatial interference features between beams; Based on spatial interference characteristics, the fusion weight parameters of the Mixing network at the current time are dynamically determined through a preset mapping function; The local Q-values ​​output by each Agent network are weighted and fused using fusion weight parameters to obtain the global joint Q-value at the current time. The local Q-value is the expected cumulative reward obtained by each Agent network in selecting the target serving cell based on the local observation information of the corresponding beam. The global joint Q-value represents the overall expected cumulative reward of all beams in collaborative decision-making. Based on the loss between the global joint Q-value and the target Q-value, the network parameters of the neural network are iteratively updated until the preset iteration termination condition is met.

[0007] Optionally, the method also includes: Spatial clustering is performed on all serving cells in the simulation environment to obtain the beam service cluster corresponding to each beam, so that each agent network can select the target serving cell within its corresponding beam service cluster.

[0008] Optionally, spatial clustering is performed on all serving cells in the simulation environment to obtain the beam service cluster corresponding to each beam, including: Based on the coverage of each beam, all serving cells are divided into dedicated cells and shared cells; a dedicated cell corresponds to one candidate beam; a shared cell corresponds to multiple candidate beams. Dedicated cells are assigned to a single beam, while shared cells are assigned to the beam with the lowest spatial density, forming an initial cluster set; The connectivity of the initial cluster set is verified by utilizing the topological relationships of geographical locations, thereby obtaining an intermediate cluster set that satisfies connectivity. Traverse the edge cells in each intermediate cluster, calculate the optimized gain of the pre-constructed joint index after migrating to the neighboring cluster, select the migration operation that maximizes the optimized gain to update the cluster state, and obtain the iterative beam service cluster; the joint index includes the load variance, size variance and spatial clustering degree of the beam cluster. Repeat the migration steps for the intermediate clusters until the preset convergence condition is met to obtain the final optimized beam service cluster for each beam.

[0009] Optionally, a graph neural network module can be used to perform graph convolution operations on the geographic location topology and global environmental state to extract spatial interference features that represent the geographic adjacency relationships between beams, including: Using all serving cells as nodes, the global environment state is mapped to an initial node feature vector; where the initial node feature vector includes at least the current load of services in each cell, the average latency tolerance, and the number of packets about to fail. Based on the topological relationships of geographical locations, determine the adjacency matrix and degree matrix for adding self-loops; The graph neural network module is used to perform multi-layer graph convolution aggregation operation on the initial node feature vector, the adjacency matrix with self-loops, and the degree matrix to obtain the fused node feature representation; The node feature representation is mapped to the spatial interference features between beams.

[0010] Optionally, the neural network includes an online network and an offline network. Both the online and offline networks include an agent network and a mixing network with embedded graph neural network modules. The offline network periodically replicates the model parameters of the online network. The local Q-values ​​output by each agent network are weighted and fused using fusion weight parameters to obtain the global joint Q-value at the current time step, including: A batch of data samples is randomly sampled from a pre-built experience pool; the data sample includes the current global environment state, the current local observation information, the beam-jump action, the reward value for performing the beam-jump action, the next global environment state, and the next local observation information. The local observation information and beam-jumping action at the current moment are input into the Agent network in the online network, and the local Q value is output. At the same time, the fusion weight parameters of the Mixing network in the online network are used to aggregate all the local Q values ​​to generate a global joint Q value. The global environment state and local observation information of the next time step are input into the offline network, and the offline network outputs the initial Q value of the next time step. Based on the reward value and the preset discount factor, the initial Q value is optimized to obtain the target Q value at the next time step.

[0011] Optionally, iteratively updating the network parameters of the neural network based on the loss between the global joint Q-value and the target Q-value includes: Calculate the timing error between the global joint Q-value and the target Q-value; The time series error is corrected based on asymmetric weighting coefficients, and the corrected time series error is used as the loss function. The parameters of the online network are updated using gradient descent based on the loss function to simultaneously adjust the parameters of the Agent network and the Mixing network. This gradient descent update is repeated until the preset iteration termination condition is met.

[0012] Optionally, the reward value is calculated using a pre-built reward function, which includes an effective throughput term, an average latency tolerance term, a packet loss penalty term, and an interference penalty term.

[0013] Secondly, a beam hopping control method based on deep reinforcement learning is provided, applied to a low-Earth orbit satellite full-frequency multiplexing communication system. This system includes multiple beam agents deployed on the satellite payload, each beam agent independently deploying an agent network pre-trained using any of the methods in the first aspect. The method includes: Obtain local observation information of each beam agent at the current moment. The local observation information includes at least the load of the service in the current covered cell of this beam, the average latency tolerance, and the number of failed packets. Local observation information is input into the Agent network in the corresponding beam intelligent body, and features are extracted and the local Q value of each candidate action is calculated through the Agent network. Based on a preset action selection strategy, the optimal action at the current moment is determined from the local Q value. The optimal action is represented by the target cell index or beam pointing instruction that the current beam should serve. The antenna array is driven to perform beam switching operations based on the target cell index or beam pointing command that the current beam should serve.

[0014] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements any of the methods of the first aspect.

[0015] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the methods of the first aspect.

[0016] This invention provides a beam hopping control method and training method based on deep reinforcement learning. It acquires the geographical location topology between serving cells in a simulation environment and the current global environmental state. A graph neural network module is used to perform graph convolution operations on the geographical location topology and global environmental state to extract spatial interference features between beams. Based on these spatial interference features, the fusion weight parameters of the mixing network at the current moment are dynamically determined through a preset mapping function. The local Q-values ​​output by each agent network are weighted and fused using the fusion weight parameters to obtain the current global joint Q-value. Based on the loss between the global joint Q-value and the target Q-value, the network parameters of the neural network are iteratively updated until a preset iteration termination condition is met. This invention extracts spatial interference features between beams through a graph neural network module and accurately aggregates local Q-values ​​using dynamic fusion weights, effectively solving the problem of inaccurate cooperation caused by the lack of global interference perception in distributed decision-making. It enables beam agents to have collaborative anti-interference capabilities based on independent observation, avoiding getting trapped in local optima and reducing co-channel interference between beams in full-frequency reuse scenarios.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1A flowchart of a training method for a neural network for controlling beam hopping, provided in an embodiment of the present invention, is shown. Figure 2 A schematic diagram of a beam hopping control architecture provided by an embodiment of the present invention is shown; Figure 3 A flowchart illustrating a beam hopping control method based on deep reinforcement learning provided in an embodiment of the present invention is shown. Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] This invention provides a training method for a neural network to control beam hopping, applicable to a low-Earth orbit satellite full-frequency reuse scenario; the neural network includes several agent networks and a mixing network with embedded graph neural network modules, each agent network controlling the beam of a low-Earth orbit satellite.

[0022] The beam control architecture for the full-frequency reuse scenario of low-orbit satellites is as follows: Figure 2 As shown: The beam control architecture for the low-orbit satellite full-frequency reuse scenario is a complex system based on multi-agent reinforcement learning, and its workflow begins with the extraction of environmental features on the left.

[0023] In this stage, the system utilizes a GNN (Graph Neural Network) located in the top left corner to process unstructured spatial data. The GNN receives the current global state. and the adjacency matrix representing the geographical topological relationships of the serving cells. As input, computation is performed through the internal graph convolutional layer (GCN) and max pooling layer (Maxpool). The core objective of this process is to encode complex spatial interference relationships and geographic topology into a high-dimensional feature vector. This allows the system to perceive the local environment in which each beam is located and its spatial interactions with other beams, providing a feature base containing spatial context information for subsequent decision-making.

[0024] After acquiring environmental characteristics, the information flow enters the agent network, with each agent specifically responsible for controlling a particular beam.

[0025] Taking agent 1 in the diagram as an example, its input observation vector ,in Represents a local observation at the current moment. Representing the action taken at the previous moment, this design incorporates action history information to aid decision-making.

[0026] This agent network employs a structure combining temporal memory and feature mapping: First, a multilayer perceptron (MLP) processes the input observation vector; then, the data is fed into a gate-controlled recurrent unit (GRU), whose output hidden state is denoted as... The role of GRU is to capture dependencies in time series, enabling the agent to remember historical states (such as the impact of previous beam pointing on current interference). After being mapped again by the MLP, the input is fed into the policy network. In addition, it combines the exploration of noise. Generate local value This represents the parameter... Below, based on observation Take action The expected return obtained. Similarly, for the Kth agent, its input is... The corresponding local value is These local Q-values ​​reflect the expected reward that a single agent believes it will obtain by taking a certain action.

[0027] To achieve global collaborative optimization, the local decision information of all agents ultimately converges to the processing layer on the right. This layer mainly consists of a hybrid network and an asymmetric weighting mechanism. The hybrid network receives information from all... The local Q-value of a beam agent (e.g.) ,…, and graph-level embedding vectors Unlike simple numerical averaging, hybrid networks contain a complex MLP structure that dynamically adjusts the weights of each agent's local Q-value based on the current global state, outputting a global joint value function. This value reflects the overall expected cumulative reward after all beams make collaborative decisions.

[0028] During training, the system also uses the target network to calculate the global Q-value at the next time step. ,in The parameters representing the target network, This represents the optimal action for the next moment. Furthermore, by employing an asymmetric weighting mechanism to calculate the TD error and generate asymmetric weighting coefficients, the calculation of the loss function (i.e., weighted TDLoss) is further refined, and the parameters of the hybrid network are updated through gradient backpropagation. The entire system guides parameter updates and policy optimization in this way, ensuring efficient beam control in complex low-Earth orbit satellite communication environments.

[0029] The asymmetric weighting mechanism will be explained in detail later, and will not be repeated here.

[0030] like Figure 1 As shown, the method includes the following steps: Step S101: Obtain the geographical location topology between each serving cell in the simulation environment and the current global environment status.

[0031] In this embodiment of the invention, a high-fidelity low-Earth orbit (LEO) high-throughput satellite full-frequency multiplexing communication simulation environment is first constructed. This environment not only includes physical layer beam coverage and interference models, but also deeply integrates the queue state evolution mechanism for multi-service concurrent transmission. The specific construction process is as follows: 1. Satellite beam and ground cell topology modeling Configure K independently switchable dynamic beams, and denote the set of beams to be assigned as . The ground area within the satellite's coverage area is divided into N hexagonal grid cells, denoted as the set of ground cells. .

[0032] For any cell Its geographical location features are determined by three-dimensional coordinate vectors in a geocentric fixed coordinate system (ECEF). Represented as: ,in These represent the coordinates of the center of the cell in the ECEF coordinate system along the three axes.

[0033] This position vector Used to construct the physical distance matrix between cells in the entire network. Any two cells in the matrix and Euclidean distance between The calculation formula is as follows:

[0034] The distance matrix This is the core foundation for subsequent calculations of the average internal distance within a cluster.

[0035] Meanwhile, in order to quantify the spatial propagation characteristics of co-channel interference under full-frequency reuse, a cell geographical adjacency relation matrix A is constructed, which can be represented as an N×N square matrix:

[0036] Among them, matrix elements The definition is as follows: if and only if the cell With the community When geographically adjacent ;otherwise .

[0037] 2. QoS queue modeling for multi-service concurrent operations To simulate the QoS requirements of multi-service concurrent operation in LEO satellite communications, each cell M service queues are maintained independently at the satellite end; for example, M=3. Define the set of service types. ,in Corresponding interactive games, Corresponding to video calls, Corresponding to voice calls.

[0038] The three services have a strict priority order. and the maximum survival time of differentiation . Time Community The Middle The queue state of a business function can be represented as follows: .element Indicates in time slot The queue has been stuck for a long time. Number of data packets per time slot.

[0039] This structure can accurately record the service wait time of each data packet, providing fine-grained basis for scheduling latency-sensitive services.

[0040] 3. Strict priority scheduling and state evolution mechanism In each decision-making time slot, the agent network will prioritize... The corresponding service queues are scanned and served in sequence, meaning that the second-highest priority service queue is only allowed to be served when the high-priority queue is cleared or the current beam capacity is exhausted.

[0041] Within queues of the same business type, services follow the FIFO principle, and the scheduler strictly selects from the end of the queue ( ) to the front of the queue ( It provides services. Its state updates follow a dynamic evolution mechanism of "service-shift-new arrival".

[0042] First, calculate the remaining queue after service. .in For time slot t, in the service queue m of cell n, the index The number of data packets being served is then determined. Subsequently, aging shift and new service injection are performed to obtain the state at the next time step:

[0043] in, For the community The Middle The number of newly arriving data packets for this type of service follows an average value. The Poisson business flow.

[0044] During this process, the original index And unserved data packets It will be removed from the queue boundary and judged as a packet loss.

[0045] Through the above modeling, the simulation environment can realistically reflect the differentiated latency and reliability requirements of different services, providing reinforcement learning agents with a global environment state vector that includes complex queuing states. .

[0046] Step S102: Use the graph neural network module to perform graph convolution operation on the topological relationship of geographical location and the global environmental state to extract the spatial interference features between beams.

[0047] In one feasible implementation, a graph neural network module is used to perform graph convolution operations on the geographic location topology and global environmental state to extract spatial interference features between beams, including: Step S102A: Using all serving cells as nodes, map the global environment state to the initial node feature vector.

[0048] Dimension is global environment state vector Reconstructed into a node feature matrix , where N is the total number of ground cells (i.e., the number of nodes). The initial feature dimension for each node.

[0049] The initial node feature vector includes at least the current load of services in each cell, the average latency tolerance, and the number of packets about to fail.

[0050] Step S102B: Based on the geographic location topology, determine the adjacency matrix and degree matrix to be added to the self-loop.

[0051] Step S102C: Use the graph neural network module to perform multi-layer graph convolution aggregation operation on the initial node feature vector, the adjacency matrix with added self-loops, and the degree matrix to obtain the fused node feature representation.

[0052] definition To add a self-loop to the adjacency matrix, where Let be an N×N identity matrix. The purpose of adding self-loops is to ensure that the original features of a node are not completely smoothed out when aggregating neighbor information, thus preserving the integrity of local information.

[0053] Based on adjacency matrix Calculate the degree matrix This is a diagonal matrix, and its diagonal elements are... equal No. The sum of all elements in the row is used for Laplace normalization to prevent numerical overflow.

[0054] By utilizing a graph neural network module, a multi-layer message passing mechanism is implemented on the initial features, adjacency matrix, and degree matrix to gradually fuse information from multi-hop neighbors.

[0055] Specifically, perform L layers of convolution to... Taking layers as an example, the propagation rule of graph convolution in each layer is as follows:

[0056]

[0057] in, Indicates the activation function; Indicates the first The learnable parameter matrix in a layered graph convolutional network.

[0058] As the number of layers increases, the features of each node gradually incorporate information from its more distant neighbors, forming a globally perceptive feature representation. For example, after... After layer propagation, the features of each node It not only includes the condition of its own community, but also incorporates the surrounding area. L The interference propagation path information of all cells within the jump range is used to form a node feature representation with a global perspective.

[0059] Step S102D: Map the node feature representation to the spatial interference features between beams.

[0060] The node features obtained from multi-layer convolution are transformed into a vector that can represent the overall network interference state.

[0061] Continuing from the previous example, for the features of the last layer of nodes... Perform global max pooling on each row to generate graph-level embedding vectors. : .

[0062] Generated graph-level embedding vectors This refers to spatial interference characteristics, which encapsulate the essential information about the interaction between beams in the entire simulation environment and serve as the input for subsequent dynamic weight generation.

[0063] Step S103: Based on spatial interference characteristics, dynamically determine the fusion weight parameters of the Mixing network at the current moment through a preset mapping function.

[0064] Using the hypernetwork in Mixing networks Graph-level embedding vectors Mapped to the non-negative weights W and bias b of the Mixing network:

[0065] For example, if graph embedding This indicates that the interference characteristic value between two beams is extremely high (e.g., there is strong co-channel interference). The supernetwork will automatically adjust the mapped weight coefficients to reduce the contribution weight of one of the agents in the network, preventing it from dominating the global decision due to local perspective bias, which would lead to a decrease in system performance.

[0066] This dynamic mechanism enables the network to adaptively adjust the contribution of each agent based on real-time interference, effectively avoiding the rigidity problem caused by fixed weights.

[0067] Step S104: Use the fusion weight parameter to perform weighted fusion of the local Q values ​​output by each Agent network to obtain the global joint Q value at the current time.

[0068] The local Q-value represents the expected cumulative reward obtained by each agent network in selecting the target serving cell based on the local observation information of the corresponding beam; the global joint Q-value represents the overall expected cumulative reward of all beams in collaborative decision-making.

[0069] For example, calculate the global joint Q-value. :

[0070] in, This represents the first layer of hybrid weight matrix generated by the hypernetwork, which is responsible for mapping local Q-values ​​to a high-dimensional latent space and capturing the nonlinear interaction between local Q-values. This represents the second-layer weight vector (usually a scalar or row vector) generated by the hypernetwork. It is responsible for compressing the high-dimensional latent space features back to a single global Q value and is usually constrained to be non-negative to meet the monotonicity constraint required by the QMIX algorithm (i.e., increasing the Q value of a single agent does not lead to a decrease in the global Q value). This represents the first layer bias vector generated by the hypernetwork, used to adjust the reference point of the mapping; This represents the second-layer bias scalar generated by the hypernetwork, which serves as a benchmark for fine-tuning the global Q-value; This represents a column vector consisting of the local Q-values ​​output by the K agent networks.

[0071] Through the aforementioned weighted fusion, not only is the distributed decision-making flexibility of each agent network preserved, but also... , The introduction of dynamic parameters enables global interference perception, ensuring that the final output global joint Q value accurately reflects the true value of all beam cooperative actions, and providing a reliable feedback signal for subsequent reinforcement learning parameter updates.

[0072] Step S105: Based on the loss between the global joint Q-value and the target Q-value, iteratively update the network parameters of the neural network until the preset iteration termination condition is met.

[0073] This embodiment extracts inter-beam spatial interference features through a graph neural network module and accurately aggregates local Q-values ​​using dynamic fusion weights, effectively solving the problem of inaccurate collaboration caused by the lack of global interference perception in distributed decision-making. This enables the beam agent to possess collaborative anti-interference capabilities based on independent observation, avoiding getting trapped in local optima and reducing co-channel interference between beams in full-frequency reuse scenarios.

[0074] In a full-frequency reuse scenario, a single beam can theoretically cover the entire simulation area, meaning that a single beam can select the target serving cell within all ground cell ranges. However, this results in an excessively large action space, making convergence difficult.

[0075] Therefore, based on the above embodiments, the method further includes: Step S106: Spatial clustering is performed on all serving cells in the simulation environment to obtain the beam service cluster corresponding to each beam, so that each agent network can select the target serving cell within its corresponding beam service cluster.

[0076] By spatial clustering, the global problem is decomposed into multiple local subproblems, restricting each agent to make decisions only within its associated cluster.

[0077] In one feasible implementation, spatial clustering is performed on all serving cells in the simulation environment to obtain the beam service cluster corresponding to each beam, including: Step S106A: Based on the coverage of each beam, all serving cells are divided into dedicated cells and shared cells; a dedicated cell corresponds to one candidate beam; a shared cell corresponds to multiple candidate beams.

[0078] The beam consists of a main lobe and side lobes; the geometric coverage of each beam is analyzed. If a cell falls entirely within the main lobe of a particular beam and is not significantly affected by other beams, it is designated as an exclusive cell, and the set of exclusive cells is defined as such. If a cell is located at the edge of overlapping multiple beams, it is marked as a shared cell, and the set of shared cells is marked as... .

[0079] Step S106B: Assign the dedicated cell to a unique beam and assign the shared cell to the beam with the lowest spatial clustering to form an initial cluster set.

[0080] Specifically, dedicated cells will be directly assigned to the only feasible beam cluster. .

[0081] For the remaining shared cell set First, count the number of candidate beams for each shared cell, and then sort them in ascending order of this number. The cells are sorted and prioritized for those with limited available resources.

[0082] During the assignment process, a greedy strategy is used to traverse the sorted shared cells and attempt to temporarily add them to each candidate beam cluster that meets the coverage conditions. middle.

[0083] Define the temporary cluster structure as follows The number of communities it includes is .

[0084] Average internal distance within the temporary cluster Characterizes the degree of spatial aggregation.

[0085] when When, the average distance is defined as 0; when At that time, the average internal distance of the temporary cluster The calculation formula is as follows:

[0086] In the formula, the summation term The coefficient represents the sum of the Euclidean distances between all unique pairs of subclusters within a cluster. This is the reciprocal of the total number of node pairs, used to calculate the average.

[0087] Compare the calculation results after adding each candidate cluster, and select the one that makes the cluster the most suitable for the next step. Minimum cluster As the final assigned target, perform the update operation.

[0088] Step S106C: Use the geographic location topology to perform connectivity verification on the initial cluster set to obtain an intermediate cluster set that satisfies connectivity.

[0089] If nodes within a cluster are geographically disconnected (i.e. isolated by other clusters), it may cause path discontinuities during beam transitions.

[0090] Therefore, by traversing the initial clusters, the adjacency relationships between nodes are checked. If a disconnection is found, adjacent small clusters are merged or boundary nodes are redistributed to ensure that the clusters served by each beam are topologically connected.

[0091] Specifically, a breadth-first search (BFS) algorithm can be used to verify the connectivity of each generated initial cluster, ensuring that the clustering scheme is geographically connected.

[0092] Step S106D: Traverse the edge cells in each intermediate cluster, calculate the optimized gain of the pre-constructed joint index after migrating to the neighboring cluster, select the migration operation that maximizes the optimized gain to update the cluster state, and obtain the iterated beam service cluster.

[0093] Joint metrics include the load variance of the beam cluster. Size variance and spatial concentration .

[0094] Load variance Characterizes the degree of balance in traffic volume between beams; scale variance Characterizes the uniformity of beam coverage.

[0095] Specifically, the calculation formulas for each indicator are as follows:

[0096]

[0097]

[0098] In the formula, Indicates the total number of dynamic beams that can be switched independently; Indicates assignment to the first A collection of service cells for each beam; For clusters The average mixed traffic volume of all communities within the area; That is, the average traffic volume of all beam clusters; for Cluster size, i.e., the number of cells within a cluster; This represents the average cluster size, which is the total number of cells divided by the number of clusters. The internal average distance is calculated by referring to the calculation process in step S106B above.

[0099] In each iteration, all clusters are traversed. peripheral communities And identify the neighbor clusters that can be covered by the cell. For potential migration operations that satisfy connectivity constraints Calculate the reduction in load variance caused by this movement. , size variance reduction and average compactness optimization amount And based on the weighting coefficients of each indicator Construct the following comprehensive scoring function:

[0100] Select the migration operation that maximizes the positive score and execute it, while simultaneously updating the entire network clustering scheme. And various real-time indicators.

[0101] Step S106E: Repeat the steps of migrating the intermediate clusters until the preset convergence condition is met, and obtain the final optimized beam service cluster corresponding to each beam.

[0102] The process of step S106D is iterated repeatedly until the maximum overall score gain is reached. Below the preset convergence threshold Or, after reaching the maximum number of iterations, the optimized beam service cluster partitioning scheme will be output.

[0103] Based on the above embodiments, in order to improve training stability and solve the Q-value overestimation problem, this embodiment introduces a dual-network architecture of online and offline networks. Figure 2 As can be seen, both the agent network and the hybrid network have a dual-network structure, meaning the neural network includes an online network and an offline network. Both the online and offline networks include an agent network and a mixing network with embedded graph neural network modules; the offline network periodically replicates the model parameters of the online network.

[0104] By weighting and fusing the local Q-values ​​output by each agent network using fusion weight parameters, the global joint Q-value at the current time step is obtained, including: Step S104A: Randomly sample a batch of data from the pre-built experience pool.

[0105] The data sample is a tuple , This represents the global environment state vector at the current moment. This represents local observation information at the current moment. Indicates beam switching action. This represents the reward value for performing the beam-switching action. Indicates the global environment state at the next moment; This indicates local observation information at the next moment.

[0106] The initial sample size in the experience pool is 0. After the beam agent interacts with the simulation environment for the first time, it selects a beam pointing action within a valid beam service cluster and generates an experience data point through interaction with the environment. The experience data is stored in the replay pool. After multiple interactions, once the number of samples in the experience pool reaches a certain training requirement, it is used for subsequent random sampling training.

[0107] Step S104B: Input the local observation information and beam-jumping action at the current moment into the Agent network in the online network, output the local Q value, and at the same time aggregate all the local Q values ​​through the fusion weight parameters of the Mixing network in the online network to generate a global joint Q value.

[0108] The agent network employs a recurrent neural network architecture, with the network input being local observations of the current beam. Actions at the previous moment The network output is the local Q-value of the agent in the current time slot for selecting each cell, using a greedy strategy or... - Greedy strategy selects the action with the highest Q value That is, to determine the specific cell to which the beam is pointing.

[0109] Specifically, for each beam agent The recurrent neural network (RNN) structure of the Agent network is used to process time-series observation data. The formulas for hidden layer state update and local Q-value calculation are as follows:

[0110]

[0111] in, for The hidden state at any given moment. For the parameters of the Agent network, Represents beam intelligent agent Local observations Represents beam intelligence exist The action performed at any given moment; For beam-based intelligent agents in The hidden state at any given moment; It is the hidden state A nonlinear mapping function that maps to local Q-values.

[0112] The Mixing network is responsible for nonlinearly fusing the local Q-values ​​of all agents into a global joint Q-value. In order to evaluate the merits and demerits of the overall coordinated action.

[0113] Step S104C: Input the global environment state and local observation information of the next time step into the offline network, and output the initial Q value of the next time step from the offline network.

[0114] The offline network (TargetNetwork) periodically copies the parameters of the online network but keeps them frozen for each training cycle. Its purpose is to provide a stable target value reference and avoid oscillations during training. It uses the same network structure as the online network, but its parameters lag behind the online network.

[0115] Specifically, first utilize online network parameters State at the next moment Next choice greedy combined action Secondly, utilize offline network parameters. Calculate the value of this joint action, i.e., the initial Q value. .

[0116] Step S104D: Based on the reward value and the preset discount factor, optimize the initial Q value to obtain the target Q value at the next moment.

[0117] Specifically, the target Q value The calculation formula is as follows:

[0118] in, For offline network parameters, As a termination marker, As a discount factor, This is the reward value.

[0119] This embodiment effectively suppresses the Q-value divergence problem common in deep reinforcement learning by introducing an offline network and a soft update mechanism for the target Q-value, thereby improving the convergence speed and stability of the algorithm under dynamic disturbance environments.

[0120] In multi-agent collaborative scenarios, to ensure consistency in multi-agent collaboration, a global joint value function is needed. Must satisfy local value Monotonicity constraints.

[0121] In traditional multi-agent scenarios, agents typically cooperate, and increased individual gains usually translate to increased overall gains. However, in LEO satellite full-frequency reuse networks, intense co-channel interference competition exists between beams. When a certain beam... When a high-gain channel or high-density user area is selected to maximize its throughput, its local Q value increases, which may cause severe co-channel interference to its neighboring beams.

[0122] Such interference can cause a sharp drop in the signal-to-interference-plus-noise ratio (SIR) of adjacent beams, leading to a severe deterioration in their Quality of Service (QoS) and even significant packet loss. At this point, although the beam... While the local performance is excellent, the overall throughput and user experience drop significantly due to the surge in interference, resulting in non-monotonic returns.

[0123] To address the non-monotonic reward problem caused by interference, this embodiment introduces asymmetric weighting coefficients to assign higher fitting priority to underestimated high-quality actions during the projection process.

[0124] Therefore, based on the above embodiments, the iterative update of the neural network parameters based on the loss between the global joint Q-value and the target Q-value includes: Step S105A: Calculate the timing error between the global joint Q value and the target Q value.

[0125] Specifically, timing error The calculation formula is as follows:

[0126] in, For the first The target Q value for each sample.

[0127] Step S105B: Correct the timing error based on the asymmetric weighting coefficients, and use the corrected timing error as the loss function.

[0128] Specifically, asymmetric weighting coefficients Represented as:

[0129] in, This represents the magnification factor, which is usually set to be greater than 1.

[0130] The ultimate optimization objective is to minimize the weighted mean squared error loss function. :

[0131] in, The total number of samples drawn from the experience pool; For sample index.

[0132] Step S105C: Update the network parameters of the online network using gradient descent based on the loss function to simultaneously adjust the parameters of the Agent network and the Mixing network.

[0133] Specifically, first, the loss function is calculated with respect to the network parameters. gradient Then use an optimizer (such as Adam) to update the parameters: .

[0134] It should be noted that since the Agent network and Mixing network share some of the underlying feature extraction logic, gradient descent will optimize the parameters of both parts simultaneously, achieving end-to-end co-evolution.

[0135] Step S105D: Repeat gradient descent updates until the preset iteration termination condition is met.

[0136] Preset iteration termination conditions, such as reaching a preset number of iterations or satisfying convergence conditions.

[0137] The trained agent network is deployed on satellite beam agents, and each beam agent independently makes optimal scheduling decisions based on local observations.

[0138] This embodiment introduces asymmetric weighting coefficients, breaking the constraints of the strict monotonicity assumption in traditional multi-agent reinforcement learning on the training process. When the network discovers that certain actions are underestimated in a disturbed environment (i.e., the local Q-value may not fully reflect the global contribution), it assigns higher penalty weights, guiding the network to learn a more accurate nonlinear value mapping, thereby achieving stable beam cooperative control in a dynamically changing full-frequency reuse environment.

[0139] Based on the above embodiments, the reward value is calculated through a pre-constructed reward function, which includes a system throughput term, an average latency tolerance term, a packet loss penalty term, and an interference penalty term.

[0140] This embodiment no longer focuses solely on maximizing total throughput, but instead constructs a comprehensive reward function that includes effective throughput gain, average latency tolerance gain, packet loss cost, and interference penalty.

[0141] The formula for calculating the reward function is as follows: in, For effective throughput; The normalization coefficient representing the effective throughput; This represents the average value of the delay tolerance. Normalized coefficient representing the average delay tolerance; Packet loss rate; Indicates the interference penalty value; Indicates the weight of the effective throughput term; This indicates the weight of the average delay tolerance term; This indicates the weight of the interference penalty term.

[0142] Specifically, the physical meaning and calculation of each sub-reward: First item: Total throughput in time slot t The sum of the effective service volume of all activated beams .

[0143] In this regard, the FIFO service principle described in the above steps is as follows: The actual effective throughput provided by beam k to cell n in time slot t. : In the formula, For the size of a single data packet, The time slot length, Let w be the backlog of the m-th type of service in cell n at index w; Provide the physical link capacity for beam k to cell n.

[0144] The physical link capacity can be calculated based on Shannon's formula: .

[0145] In the full-frequency reuse mode, the received signal-to-interference-plus-noise ratio (SINR) of beam k to cell n can be calculated as follows:

[0146] in, This indicates the transmit power of each beam; This represents the transmit antenna gain of beam k transmitted to target cell n in that time slot; This indicates the transmit antenna gain of other active beams j that leak into the target cell n in this time slot; Indicates the antenna gain for ground reception; Indicates free space path loss; Indicates noise power.

[0147] The summation term in the denominator of the formula explicitly characterizes the co-channel interference (CCI) caused by other active beams j to the target cell n.

[0148] To accurately characterize the interference characteristics of beam sidelobes on co-frequency neighboring cells, the transmit antenna gain... The calculation is performed using an analytical model based on Bessel functions of the first kind. The off-axis angle of the user relative to the beam center is considered. The gain calculation formula is:

[0149] in, . For half-power beamwidth, and These are first-order and third-order Bessel functions of the first kind, respectively; This indicates the maximum transmit antenna gain.

[0150] For ground receiving antenna gain The model is based on a reference radiation pattern, and it is assumed that the ground receiving antenna is always precisely aligned with the satellite direction. Therefore, in this problem... .

[0151] Second term: Average time delay tolerance in time slot t The average delay tolerance for all cells .

[0152] in, This represents the cell-level latency tolerance.

[0153] Furthermore, in order to assess the overall urgency of cell n, priority masking logic is performed to determine cell-level tolerance. The overall tolerance status of a cell is determined by the exclusive access of the currently backlogged highest priority services. If the cell is completely empty, it is reset to the system's maximum lifetime. This indicates the safest state. The mathematical expression is as follows:

[0154] In order to quantify the differentiated requirements of mixed services for scheduling timeliness, the multi-dimensional queue state established by the above steps is used. An evaluation is conducted. The absolute average remaining lifetime of the m-th service type in cell n is determined. The calculation formula is:

[0155] in, This indicates the remaining lifespan of the data packet at index w. The physical meaning of this indicator is the average number of remaining lifespan slots of the data packets currently in the queue.

[0156] Third item: Packet loss rate At the end of time slot t, located at the end of the aging index of each logical queue ( Furthermore, packets not allocated capacity will be forcibly overflowed and judged as dropped. Therefore, at time slot t, the total number of packets lost for type m services is... It can be calculated as follows:

[0157] Since the traffic follows a Poisson distribution, the average traffic of type m traffic per unit time slot can be calculated as follows: , This represents the average value of the Poisson distribution for type m services in cell n.

[0158] Therefore, at time slot t, the formula for calculating the packet loss rate of type m service is:

[0159] Fourth item: Interference penalty: To reduce co-channel interference between beams, beams should avoid illuminating adjacent cells as much as possible when illuminating a cell; therefore, we define an interference penalty term. This penalty is proportional to the number of conflicting beams illuminating geographically neighboring cells. The calculation formula is as follows:

[0160] in, Let represent the cell ID illuminated by beam k at time t. By using the previously defined adjacency matrix A, we can determine the relationship between beam k and the cell ID at that time. Whether the cells illuminated at that moment are adjacent is used to calculate the value of the co-channel interference penalty term in the current time slot system.

[0161] The reward function aims to guide the agent network to find the optimal balance between "high throughput" and "low latency / low packet loss" and minimize co-channel interference between beams, especially in scenarios with non-monotonic rewards (i.e., high local throughput leads to high global interference), by suppressing destructive actions through a negative feedback mechanism.

[0162] Another embodiment of the present invention provides a beam hopping control method based on deep reinforcement learning, applied to a low-Earth orbit satellite full-frequency reuse communication system. This system includes multiple beam agents deployed on the satellite payload, each beam agent independently deploying an agent network pre-trained using any of the methods described in the first aspect; such as Figure 3 As shown, the method includes: Step S301: Obtain the local observation information of each beam agent at the current time.

[0163] Local observation information includes at least the load of services in the cell currently covered by this beam, the average delay tolerance, and the number of packets about to fail. Step S302: Input the local observation information into the Agent network in the corresponding beam intelligent body, extract features through the Agent network and calculate the local Q value of each candidate action.

[0164] Step S303: Based on the preset action selection strategy, determine the optimal action at the current moment from the local Q value. The optimal action is represented by the target cell index or beam pointing instruction that the current beam should serve.

[0165] The preset action selection has been tested, for example, selecting the action with the largest Q value.

[0166] Step S304: Drive the antenna array to perform beam switching operation according to the target cell index or beam pointing command that the current beam should serve.

[0167] This embodiment extracts inter-beam spatial interference features through a graph neural network module and accurately aggregates local Q-values ​​using dynamic fusion weights, effectively solving the problem of inaccurate collaboration caused by the lack of global interference perception in distributed decision-making. This enables the beam agent to possess collaborative anti-interference capabilities based on independent observation, avoiding getting trapped in local optima and reducing co-channel interference between beams in full-frequency reuse scenarios.

[0168] Based on the same technical concept, embodiments of the present invention also provide an electronic device, such as... Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0169] Memory 403 is used to store computer programs; When the processor 401 executes the program stored in the memory 403, it implements the steps of the above method embodiment.

[0170] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0171] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0172] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0173] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0174] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method embodiments. Specific implementation details can be found in the method embodiments and will not be repeated here.

[0175] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0178] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0180] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A training method for a neural network for controlling beam hopping, characterized in that, This method is applied to low-Earth orbit (LEO) satellite full-frequency reuse scenarios. The neural network includes several agent networks and a mixing network with embedded graph neural network modules. Each agent network controls the beam of a LEO satellite. Obtain the geographical location topology between each serving cell in the simulation environment and the current global environment state; The graph neural network module is used to perform graph convolution operations on the geographic location topology and global environmental state to extract spatial interference features between beams; Based on the spatial interference characteristics, the fusion weight parameters of the Mixing network at the current moment are dynamically determined through a preset mapping function; The local Q-values ​​output by each Agent network are weighted and fused using the fusion weight parameters to obtain the global joint Q-value at the current time. The local Q-value is the expected cumulative reward obtained by each Agent network in selecting the target serving cell based on the local observation information of the corresponding beam. The global joint Q-value represents the overall expected cumulative reward of all beams in collaborative decision-making. Based on the loss between the global joint Q-value and the target Q-value, the network parameters of the neural network are iteratively updated until the preset iteration termination condition is met.

2. The method according to claim 1, characterized in that, The method further includes: Spatial clustering is performed on all serving cells in the simulation environment to obtain the beam service cluster corresponding to each beam, so that each agent network can select the target serving cell within its corresponding beam service cluster.

3. The method according to claim 2, characterized in that, The step of spatially clustering all serving cells in the simulation environment to obtain the beam service cluster corresponding to each beam includes: Based on the coverage of each beam, all serving cells are divided into dedicated cells and shared cells; each dedicated cell corresponds to one candidate beam; each shared cell corresponds to multiple candidate beams. The dedicated cells are assigned to a single beam, and the shared cells are assigned to the beam with the lowest spatial clustering, forming an initial cluster set; The initial cluster set is tested for connectivity using the geographical location topology to obtain an intermediate cluster set that satisfies connectivity. Traverse the edge cells in each intermediate cluster, calculate the optimized gain of the pre-constructed joint index after migrating to the neighboring cluster, select the migration operation that maximizes the optimized gain to update the cluster state, and obtain the iterative beam service cluster; the joint index includes the load variance, size variance and spatial clustering degree of the beam cluster. Repeat the migration steps for the intermediate clusters until the preset convergence condition is met to obtain the final optimized beam service cluster for each beam.

4. The method according to claim 1, characterized in that, The step of using the graph neural network module to perform graph convolution operations on the geographic location topology and global environmental state to extract spatial interference features representing the geographic adjacency relationships between beams includes: Using all serving cells as nodes, the global environment state is mapped to an initial node feature vector; wherein the initial node feature vector includes at least the current load of services in each cell, the average latency tolerance, and the number of packets about to fail. Based on the aforementioned geographic location topology, determine the adjacency matrix and degree matrix for adding self-loops; The graph neural network module is used to perform multi-layer graph convolution aggregation operation on the initial node feature vector, the adjacency matrix with self-loops, and the degree matrix to obtain the fused node feature representation; The node features are mapped to spatial interference features between beams.

5. The method according to claim 1, characterized in that, The neural network includes an online network and an offline network, both of which include an agent network and a Mixing network with embedded graph neural network modules. The offline network periodically replicates the model parameters of the online network. The weighted fusion of the local Q-values ​​output by each agent network using the fusion weight parameters to obtain the global joint Q-value at the current time includes: A batch of data samples is randomly sampled from a pre-built experience pool; the data samples include the current global environment state, the current local observation information, the beam-jumping action, the reward value for performing the beam-jumping action, the global environment state at the next time step, and the local observation information at the next time step. The local observation information and beam-jumping action at the current moment are input into the Agent network in the online network, and the local Q value is output. At the same time, the fusion weight parameters of the Mixing network in the online network are used to aggregate all the local Q values ​​to generate a global joint Q value. The global environment state and local observation information of the next time step are input into the offline network, and the offline network outputs the initial Q value of the next time step. Based on the reward value and the preset discount factor, the initial Q value is optimized to obtain the target Q value at the next moment.

6. The method according to claim 5, characterized in that, The iterative update of the network parameters of the neural network based on the loss between the global joint Q-value and the target Q-value includes: Calculate the timing error between the global joint Q-value and the target Q-value; The timing error is corrected based on asymmetric weighting coefficients, and the corrected timing error is used as a loss function. The network parameters of the online network are updated using gradient descent based on the loss function to synchronously adjust the parameters of the Agent network and the Mixing network. The gradient descent update is repeated until the preset iteration termination condition is met.

7. The method according to claim 5, characterized in that, The reward value is calculated using a pre-constructed reward function, which includes an effective throughput term, an average latency tolerance term, a packet loss penalty term, and an interference penalty term.

8. A beam hopping control method based on deep reinforcement learning, characterized in that, An application to a low-Earth orbit satellite full-frequency multiplexing communication system, the system comprising multiple beam agents deployed on a satellite payload, each beam agent independently deploying an agent network pre-trained by the method described in any one of claims 1-7; the method comprising: Obtain local observation information of each beam agent at the current time. The local observation information includes at least the load of the service in the current covered cell of this beam, the average delay tolerance, and the number of packets about to fail. The local observation information is input into the Agent network of the corresponding beam intelligent body, and features are extracted and the local Q value of each candidate action is calculated through the Agent network. Based on a preset action selection strategy, the optimal action at the current moment is determined from the local Q value. The optimal action is represented as the target cell index or beam pointing instruction that the current beam should serve. The antenna array is driven to perform beam switching operations according to the target cell index or beam pointing command that the current beam should serve.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images