A method and device for skip-beam resource allocation based on deep reinforcement learning

Abstract

This application provides a method and apparatus for beam hopping resource allocation based on deep reinforcement learning. The method includes: acquiring the state information of all arriving data packets in each data packet buffer zone equipped with onboard equipment; reconstructing the state of the data packets based on the state information to obtain a time delay matrix corresponding to each data packet buffer zone; inputting the time delay matrix into a trained resource allocation model to predict different action vectors and corresponding expected reward values; predicting the expected reward values ​​based on the updated resource allocation model; arranging the predicted reward values ​​in descending order; determining the corresponding beam based on the action vectors corresponding to the first preset number of expected reward values; and illuminating the beam. The beam hopping resource allocation method and apparatus based on deep reinforcement learning provided in this application are used to improve the adaptability to the dynamic environment of low-Earth orbit satellites and achieve efficient beam hopping scheduling.

Description

Technical Field

[0001] This application relates to the field of resource allocation technology, and in particular to a method and apparatus for beam hopping resource allocation based on deep reinforcement learning. Background Technology

[0002] With the continuous growth of user demand for satellite communication, traditional single-beam satellite resource scheduling methods are no longer sufficient to meet the ever-changing service requirements. Multi-beam technology has become an effective means to improve the resource utilization of satellite systems. Although multi-beam technology can provide services simultaneously on multiple beams, it cannot address the problem of uneven service demand in different areas. For example, beam resources are scarce in high-demand areas, while beams are abundant in low-demand areas, leading to resource waste and a decrease in overall system throughput. To solve these problems, beam hopping technology has emerged. Its core idea is to illuminate only a portion of the beams in each time slot and dynamically adjust the beam hopping schedule according to changes in user demand to improve system resource utilization. Low Earth Orbit (LEO) satellite communication systems occupy an important position in the future communication market due to their advantages such as low latency, high speed, and wide coverage. However, LEO satellites have characteristics such as short visibility time, rapid orbital changes, and dynamic changes in channel conditions. Therefore, how to efficiently allocate beam hopping resources has become a key issue.

[0003] Currently, beam hopping resource allocation mainly relies on iterative optimization methods, such as greedy algorithms and dynamic programming. These methods typically optimize based on the current network state, but they struggle to adapt to the rapidly changing channel conditions of low-Earth orbit (LEO) satellites. Furthermore, due to the short visibility time of LEO satellites to users, efficient resource scheduling strategies are needed to avoid resource waste and excessive data transmission latency. Therefore, traditional methods have limitations when facing complex and dynamic environments, making it difficult to achieve long-term optimal resource allocation.

[0004] Therefore, there is an urgent need for a method to improve the adaptability to the dynamic environment of low-Earth orbit satellites and achieve efficient beam skipping scheduling. Summary of the Invention

[0005] In view of this, this application provides a method and apparatus for beam hopping resource allocation based on deep reinforcement learning, which can improve the adaptability to the dynamic environment of low-Earth orbit satellites and achieve efficient beam hopping scheduling.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a beam-hopping resource allocation method based on deep reinforcement learning, the method comprising:

[0008] Based on the data packet buffers equipped on the spaceborne equipment, obtain the status information of all arriving data packets in each data packet buffer;

[0009] Based on the state information, the state of the data packets is reconstructed to obtain the delay matrix corresponding to each data packet buffer; wherein, the data packets are normalized according to the arrival time of the data packets under different data packet buffers; the delay matrix represents the number of data packets arriving at the data packet buffer at different times;

[0010] The time delay matrix is ​​input into the trained resource allocation model to predict different action vectors and their corresponding expected reward values. Specifically, an experience pool for reinforcement learning is constructed, and the exploration probability is dynamically adjusted based on the training cycle. The action vector in the current state is determined based on the exploration probability. The action vector is executed, beam resource allocation is adjusted, and the state of the data packet buffer is updated based on the scheduling result. The reward value of the action vector in the current state is calculated, and the current state, the action vector, the reward value, and the next state are stored in the experience pool. When the experience pool is full, a random sampling mechanism is used to select training samples from the experience pool. The objective function value is calculated based on the training samples, a loss function is constructed based on the objective function value, and the weight parameters of the resource allocation model are updated based on the loss function.

[0011] Based on the updated resource allocation model, the expected return value is predicted, and the predicted return value is sorted in descending order. The corresponding beam is determined based on the action vector corresponding to the first preset number of expected return values, and the beam is lit up.

[0012] A second aspect of this application provides a beam-hopping resource allocation device based on deep reinforcement learning, the device comprising an acquisition module, a reconstruction module, a prediction module, and a determination module;

[0013] The acquisition module is used to acquire the status information of all arriving data packets in each data packet buffer based on the data packet buffer provided by the spaceborne equipment.

[0014] The reconstruction module is used to reconstruct the state of the data packets based on the state information to obtain a delay matrix corresponding to each data packet buffer; wherein, the data packets are normalized according to the arrival time of the data packets under different data packet buffers; the delay matrix represents the number of data packets arriving at the data packet buffer at different times;

[0015] The prediction module is used to input the time delay matrix into the trained resource allocation model to predict different action vectors and their corresponding expected reward values. Specifically, it involves constructing a reinforcement learning experience pool, dynamically adjusting the exploration probability based on the training cycle, determining the action vector in the current state based on the exploration probability, executing the action vector, adjusting beam resource allocation, and updating the state of the data packet buffer based on the scheduling result, calculating the reward value of the action vector in the current state, and storing the current state, the action vector, the reward value, and the next state in the experience pool. When the experience pool is full, a random sampling mechanism is used to select training samples from the experience pool. An objective function value is calculated based on the training samples, a loss function is constructed based on the objective function value, and the weight parameters of the resource allocation model are updated based on the loss function.

[0016] The determining module is used to predict the expected return value based on the updated resource allocation model, sort the predicted return values ​​in descending order, determine the corresponding beam based on the action vector corresponding to the first preset number of expected return values, and light up the beam.

[0017] This application provides a method and apparatus for beam hopping resource allocation based on deep reinforcement learning. By optimizing beam hopping resource allocation through deep reinforcement learning, it adapts to the rapid channel changes caused by the high-speed movement of low-Earth orbit satellites, ensuring efficient resource scheduling, reducing resource waste, and lowering data transmission latency. First, this application acquires the state information of all arriving data packets based on the data packet buffer of the onboard equipment and constructs a delay matrix through state reconstruction. Since data packets arrive at different buffers at different times, directly using the raw data may lead to uneven distribution of input data, affecting the convergence speed and generalization ability of reinforcement learning. Therefore, this application employs a normalization operation to keep the data input within a stable numerical range, reducing the interference of time characteristics on the learning process and enabling the model to better learn the dynamic characteristics of data packet changes. The method of constructing the delay matrix effectively represents the traffic changes of data packets at different times, providing reasonable input for subsequent resource allocation.

[0018] Secondly, during the reinforcement learning training phase, this application utilizes an experience pool to store interaction data and dynamically adjusts the exploration probability based on the training cycle to balance exploration and utilization. In reinforcement learning tasks, if the exploration rate is too high, the model may frequently attempt random actions, making it difficult to converge to the optimal policy; if the exploration rate is too low, it may converge to a local optimum prematurely, causing the resource allocation scheme to be unable to adapt to changes in the environment. By dynamically adjusting the exploration probability, the model can extensively explore different resource allocation strategies in the early stages of training and gradually utilize the learned optimal policy in the later stages, improving the efficiency and stability of resource scheduling. Furthermore, the experience pool storage mechanism allows training data to be reused, and the random sampling mechanism avoids the problem of excessive data correlation, improving training stability and generalization ability.

[0019] During the resource allocation decision-making phase, the model predicts expected returns and sorts them in descending order to ensure that the resource allocation scheme with the highest return is prioritized. The Q-value (i.e., expected return value) under the reinforcement learning framework can dynamically evaluate the long-term benefits of different actions, thus making beam resource allocation more intelligent. Compared to traditional heuristic methods or scheduling strategies based on fixed rules, this application can adjust the allocation scheme in real time according to changes in the environment, thereby achieving better resource utilization efficiency under different business needs and network conditions. Attached Figure Description

[0020] Figure 1 A flowchart of the beam hopping resource allocation method based on deep reinforcement learning provided in Embodiment 1 of this application;

[0021] Figure 2 A schematic diagram of the resource allocation model provided in this application;

[0022] Figure 3 A schematic diagram illustrating the training process of the resource allocation model provided in this application;

[0023] Figure 4 A schematic diagram illustrating the change of exploration probability over the training period provided in this application;

[0024] Figure 5 This is a schematic diagram of the structure of the beam hopping resource allocation device based on deep reinforcement learning provided in Embodiment 2 of this application. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0026] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0028] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0029] Figure 1 A flowchart illustrating the deep reinforcement learning-based beam hopping resource allocation method provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0030] S101. Based on the data packet buffers provided by the onboard equipment, obtain the status information of all arriving data packets in each data packet buffer.

[0031] Specifically, spaceborne equipment refers to communication, computing, or sensing devices installed on a satellite, typically used to provide functions such as satellite communication, data transmission, and remote sensing monitoring. In this application, spaceborne equipment refers to a communication terminal or processing unit on a satellite, used for on-orbit storage and management of data packets, and for optimizing beam resource allocation.

[0032] Furthermore, the packet buffer refers to a data buffer storage area in onboard equipment, used to temporarily store received data packets for subsequent scheduling and transmission. Due to the unique nature of satellite communication, data packets cannot be immediately transmitted to the target user and therefore need to be queued in the packet buffer. It should be noted that the buffer capacity may vary between different satellite devices, limited by hardware resources. The storage time of data packets in the packet buffer affects end-to-end communication latency, especially in real-time communication applications. As data packets continuously arrive and leave the packet buffer, the buffer state changes over time, requiring dynamic adjustments to resource allocation strategies.

[0033] The status information records the buffering status, network status, and packet characteristics of data packets within the corresponding packet buffer. The buffering status includes the number of packets currently in the buffer, the distribution of packets with different priorities, and the queuing delay. The network status includes the link quality between the satellite and the ground station or terminal (e.g., signal-to-noise ratio), available beam resources (bandwidth, power allocation, etc.), and transmission channel status (congestion, interference). Packet characteristics include arrival rate (number of packets received per unit time), departure rate (number of packets successfully transmitted per unit time), and queuing delay (waiting time for packets within the packet buffer). Furthermore, the status information also includes timestamps (recording the time packets arrived in the packet buffer), priority (packets for different applications may have different priorities), data size, target terminal, and quality of service requirements.

[0034] In practical implementation, multiple packet buffers are initialized in the onboard equipment, each storing packets for different link or service types. The communication module of the onboard equipment monitors and records packet arrival events for each packet buffer. Whenever a new packet arrives in a packet buffer, its status information is extracted and stored, and the status of the packet buffer is updated based on this information. If no new packets arrive in a packet buffer within a preset time period, the status information of all packets in the current packet buffer is normalized and integrated to obtain the status information of all arriving packets in the current packet buffer. Using the same method, the status information of all arriving packets in each packet buffer is obtained.

[0035] Optionally, the size of data packets for different link or service types is predicted. This prediction includes obtaining a distribution map of the number of communication objects under the current operating environment of the onboard equipment, determining the communication type of each communication object in the distribution map, establishing a matching relationship in the distribution map (the matching relationship being the two communicating parties), predicting the data packet size of each communication object in the distribution map based on the communication type and the current operating environment, and determining the total data packet size of the distribution map based on the matching relationship and the data packet size. Specifically, if a matching relationship exists, the largest data packet among the two is selected as the data size of this group of communication objects; for communication objects without a matching relationship, whose communication subject may be other objects outside the operating environment, their predicted data sizes are summed to the total data size.

[0036] The method provided by this invention, to avoid redundant calculations, considers the entire working environment and accurately assesses the amount of data within a region for deduplication. Based on this accurate assessment, it accurately allocates storage areas, preventing rapid overflow and frequent transfers to other storage locations, thus improving data storage accuracy. Data packets are clustered, and the ratio of the total size of data packets in each cluster is determined based on the predicted size. The size of the data packet buffer area is then determined for each clustered category based on this ratio. Within each clustered category, the size allocated to the corresponding buffer area is determined according to the ratio of the predicted sizes of each data packet. This method, when buffering data packets, stores similar and related data in nearby areas according to clustering, effectively improving the search time for subsequent related data retrieval.

[0037] S102. Based on the state information, the state of the data packet is reconstructed to obtain the delay matrix corresponding to each data packet buffer.

[0038] Specifically, the delay matrix describes the number and distribution of data packets entering the packet buffer at different time points. The delay matrix is ​​typically organized by time steps as rows and packet buffer number or category as columns. Each element of the delay matrix represents the number of data packets received by a specific packet buffer within a given time step. The delay matrix corresponding to each packet buffer can be represented as:

[0039]

[0040] Wherein, the S t The time delay matrix; This represents the number of data packets arriving in the packet buffer N and added to the buffer queue in time slot t.

[0041] It's important to note that directly inputting packet state information (such as arrival time, size, and priority) leads to complex and imbalanced data, making it difficult to learn effective strategies. Furthermore, directly processing the state information of all packets results in high-dimensional input. Constructing a delay matrix based on state information allows the resource allocation model to more clearly identify congestion in the packet buffer, thereby optimizing resource allocation strategies. The delay matrix is ​​a time-series representation method that compresses data into low-dimensional features, improving computational efficiency and capturing the load on the buffer over time. This helps reinforcement learning models dynamically adjust beam resource allocation to meet actual business needs. Since the number of packets and delay distribution may differ in different packet buffers, normalization eliminates the influence of data scale, making the resource allocation model more stable and better generalizable.

[0042] In specific implementation, the step of reconstructing the state of the data packets based on the state information to obtain the delay matrix corresponding to each data packet buffer includes: obtaining the state information of each data packet buffer; the state information includes the arrival time of different data packets arriving at the data packet buffer; setting a time window size based on different arrival times, sliding the window based on the time window size to normalize data packets with different arrival times to a unified time window; calculating the number of data packets in each data packet buffer within different time windows; and constructing a delay matrix with the data packet buffer as the row index, the time window as the column index, and the matrix element being the number of data packets in the data packet buffer within the time window.

[0043] Specifically, based on the status information recorded in each data packet buffer, the arrival times of all data packets under each data packet buffer are read, and the data packet buffer number is obtained. Further, all data packets under the current data packet buffer are traversed to determine the earliest and latest arrival times. The time window size is calculated based on the distribution of data packet arrival times, and the standard deviation σ of the data packet arrival times is calculated. T Then set W = k·σ T Wherein, W is the time window size; k is a hyperparameter, taking a value from 0.5 to 2; and σ T This represents the standard deviation. Furthermore, based on the set time window size, the time interval is divided starting from the earliest arrival time: [T] min ,T min +W),[T min +W,T min +2W),...,[T max -W,T max ]; wherein, the T min The earliest arrival time; the T max The latest arrival time is T; W is the time window size. For each data packet's arrival time T... i Find the index of the time window to which it belongs by traversing: Wherein, the t index Time window index for each data packet; the T i The arrival time for each data packet; the T minThe earliest arrival time is defined as W, and the time window size is defined as W. Data packets are assigned to their corresponding time windows based on the determined time window index. After normalization for each data packet buffer, the number of data packets in each buffer within different time windows is calculated, obtained by counting the total number of data packets arriving within each time window. Buffer occupancy is calculated as the ratio of the total storage capacity of the current data packet buffer to the maximum storage capacity. A delay matrix is ​​constructed using the data packet buffer number as the row index, the time window number as the column index, and the calculated number of data packets as matrix elements.

[0044] Specifically, setting a time window includes: determining the time distribution gap of data packets received in the current buffering period, where the time distribution gap is the difference between the earliest and latest arrival times; determining a time window adjustment coefficient based on the ratio of the time distribution gap in the current buffering period to the time distribution gap in the previous buffering period; and multiplying the endpoint values ​​of each time window in the previous buffering period by the time window adjustment coefficient to obtain the final time window for the current buffering period. In the method provided by this invention, considering that the time states of data packets received by the spaceborne equipment differ at different times, and their distribution gaps also vary, normalizing data time according to a completely uniform time window is not conducive to data segmentation and may result in uncovered data packets. This invention provides an adaptive, dynamically changing time window that, while adaptively changing, maintains a good proportional relationship with the time window of the previous period, thereby simplifying the problem of synchronizing different time windows to a unified time scale and improving the efficiency of time management.

[0045] Furthermore, after constructing the latency matrix, it is also necessary to determine the average latency, which is obtained by calculating the latency of all data packets from generation to arrival in the buffer within different time windows. The average latency can be expressed as: Wherein, the delay t The average delay; The number of packets arriving in packet buffer i that are added to the buffer queue in time slot j.

[0046] The method provided in this embodiment significantly improves the training effect of the resource allocation model by reconstructing the state of data packets at different arrival times and constructing a delay matrix. First, the arrival times of the original data packets are discrete; directly using these timestamps as input leads to inconsistent data dimensions, affecting the model's training performance. However, by setting a time window and performing sliding normalization, data packets at different times are mapped to a unified time scale. This not only eliminates data format inconsistencies but also enhances the model's generalization ability, making it applicable to resource allocation needs in different scenarios. Second, directly using the original arrival times of data packets may lead to data sparsity issues, with some time points having less data and others having excessively dense data, resulting in an uneven data distribution. Reconstructing data through a sliding window smooths the data flow, ensuring that each time window contains sufficient effective information, thereby reducing data sparsity and improving the model's ability to learn data change trends. Furthermore, the delay matrix not only represents the current load of the data packet buffer but also provides historical information, enabling the model to learn the trend of data changes over time. For example, if the data volume in a certain buffer continues to grow, the model can predict potential congestion in the future, thereby optimizing beam resource allocation strategies in advance and avoiding problems such as wasted communication resources and excessive transmission latency. Compared to using data at the current moment as input alone, the time delay matrix can more comprehensively reflect the dynamic changes in the data flow, improving the model's decision-making ability. On the other hand, in the rapidly changing channel environment of LEO satellites, resource scheduling needs to adapt quickly to dynamic changes, and the time delay matrix can clearly characterize the data traffic situation of each buffer in different time windows, enabling reinforcement learning models to combine historical data traffic distribution to predict future data demand, thereby optimizing resource scheduling strategies and making them better suited to the time-varying characteristics of LEO satellite networks. In addition, sliding window normalization can reduce the dimensionality of input data, reduce computational complexity, and improve the convergence speed of model training, making the reinforcement learning process more efficient. Therefore, by reconstructing the state of data packets and constructing a time delay matrix, not only can the model's ability to learn temporal features be improved and its ability to perceive changes in data traffic be enhanced, but resource allocation decisions can also be optimized, enabling LEO satellite communication systems to achieve more efficient resource utilization and lower communication latency in dynamic environments.

[0047] S103. Input the time delay matrix into the trained resource allocation model to predict different action vectors and corresponding expected reward values.

[0048] The process involves: constructing an experience pool for reinforcement learning; dynamically adjusting the exploration probability based on the training cycle; determining the action vector in the current state based on the exploration probability; executing the action vector to adjust beam resource allocation; updating the state of the data packet buffer based on the scheduling result; calculating the reward value of the action vector in the current state; storing the current state, the action vector, the reward value, and the next state in the experience pool; when the experience pool is full, selecting training samples from the experience pool using a random sampling mechanism; calculating the target Q value based on the training samples; constructing a loss function based on the target Q value; and updating the weight parameters of the resource allocation model based on the loss function.

[0049] Specifically, the resource allocation model is a deep reinforcement learning-based decision-making model designed to optimize beam resource allocation strategies based on the input system state (such as the delay matrix). The core of the resource allocation model is a neural network model, typically trained using reinforcement learning algorithms such as deep Q-networks, deep deterministic policy gradients, or proximal policy optimization. The resource allocation model takes the current system state (such as the delay matrix of each data packet buffer) as input and outputs multiple possible action vectors. Each action vector corresponds to a beam resource allocation scheme, and its expected reward value is calculated. The expected reward value reflects the improvement in system performance after executing the action vector, and is typically calculated based on metrics such as throughput, average latency, and buffer occupancy. During the training process, the resource allocation model continuously adjusts its weight parameters to maximize the cumulative reward of the selected action vectors, thereby gradually learning the optimal beam resource allocation strategy.

[0050] Furthermore, Figure 2 A schematic diagram of the resource allocation model provided in this application. Please refer to... Figure 2 The resource allocation model includes a Q-network, which is composed of a convolutional neural network and a deep neural network. The step of inputting the time delay matrix into the trained resource allocation model to predict different action vectors and corresponding expected reward values ​​includes: the convolutional neural network extracting spatial features from the time delay matrix based on convolutional layers to obtain the distribution of data packets at different times and beam positions; concatenating the spatial features with the transmission distance matrix and the current beam allocation pattern to obtain a concatenated feature vector; and the deep neural network processing the concatenated feature vector based on multiple hidden layers to fit the nonlinear relationship between the features and the action vector, determining the corresponding action vector and the corresponding expected reward value.

[0051] As an optional embodiment, the step of dynamically adjusting the exploration probability based on the training cycle includes: determining the exploration probability of the previous training cycle as the initial exploration probability; calculating the empty proportion of the current experience pool as the initial adjustment coefficient; using the initial exploration probability as input, estimating the repetition between the next training result and the existing stored information in the current experience pool based on the current training cycle, and determining the current rate of change by multiplying the repetition by the rate of change of the exploration probability of the previous training cycle; determining the candidate exploration probability of the current training cycle by multiplying the initial exploration probability by the current rate of change; and determining the final exploration probability value based on the product of the candidate exploration probability and the initial adjustment coefficient.

[0052] The method provided by this invention takes into account the limited storage capacity of the experience pool and the need to solve for the optimal value within limited resources. At this time, the exploration probability is adjusted according to the rate of change of probability, the size of the remaining experience pool, and the repetition rate of prediction results. While ensuring continuous optimization, it uses the current resources to perform differentiated solutions, avoiding spending too much resources and solution time on similar parameters, and effectively improving the quality of the solution results.

[0053] Specifically, the transmission distance matrix represents the physical distance from the data packet buffer to the receiver. The current pattern represents the beam resource allocation at the current moment. The Q-network is used to estimate the Q-value (i.e., the cumulative reward that may be obtained in the future) for taking different actions in the current state. In reinforcement learning, the Q-value represents the expected reward that the agent can obtain in the future after performing a certain action in a certain state. The goal is to make the Q-value estimation more accurate through training, thereby selecting the optimal strategy. A convolutional neural network is a neural network specifically designed to process grid-structured data (such as images, time series, matrix data, etc.). In this application, the convolutional neural network is mainly used to process the time delay matrix and extract the distribution features of data packets at different times and beam positions. The main components of the convolutional neural network include: Convolutional layers: extracting local features, such as data traffic distribution patterns, by sliding convolutional kernels on the time delay matrix. Activation functions: such as the ReLU function, used to introduce nonlinearity, enabling the network to learn complex mapping relationships. Pooling layers: reducing feature dimensionality, reducing computational complexity, and improving generalization ability through max pooling or average pooling. Fully connected layer: After feature extraction, the high-dimensional features output by the convolutional neural network are mapped to the next deep neural network. A deep neural network is a multilayer perceptron that processes input features through multiple hidden layers to achieve complex nonlinear mappings. The deep neural network is the decision core of the Q-network, primarily used to learn resource allocation strategies, i.e., generating different action vectors (beam allocation schemes) based on the features extracted by the convolutional neural network. The main structure of a deep neural network includes: Input layer: Receives features extracted by the convolutional neural network, as well as information such as the distance matrix and the current beam allocation pattern. Hidden layer: Composed of multiple neurons, each layer performs a weighted summation of the input data and performs nonlinear transformations through activation functions (such as ReLU, Sigmoid, etc.) to gradually learn the relationship between features and actions. Output layer: Generates different action vectors and their corresponding expected reward values ​​(Q-values) for selecting the optimal resource allocation scheme.

[0054] In its implementation, the convolutional neural network (CNN) employs multiple convolutional layers to extract features from the delay matrix. These layers capture temporal and spatial patterns within the delay matrix through convolution operations, extracting the delay distribution features of each data packet buffer under different time windows. Pooling layers use max pooling or average pooling operations to further reduce the dimensionality of the delay distribution features extracted by the convolutional layers, extracting more representative delay features, reducing computational cost, and enhancing the model's generalization ability. Furthermore, the spatial features extracted by the CNN, the transmission distance matrix, and the current beam allocation pattern are concatenated into a complete feature vector. This feature vector contains complete network state information and is input into the deep neural network. The deep neural network processes the input data layer by layer, learning the nonlinear relationship between input features and resource allocation strategies, generating possible action vectors (i.e., beam resource allocation strategies). The deep neural network ultimately outputs multiple action vectors and their corresponding expected reward values ​​to evaluate the merits of different resource allocation strategies. Based on the calculation results, the resource allocation model outputs different action vectors and their expected reward values. Action vectors with higher expected reward values ​​represent potentially better resource allocation schemes. Optimize beam resource allocation based on prediction results.

[0055] Optional, Figure 3 This is a schematic diagram illustrating the training process of the resource allocation model provided in this application. Please refer to... Figure 3 The training process of the resource allocation model includes: obtaining the current state based on the time delay matrix, transmission distance matrix, and current beam allocation pattern; interacting the resource allocation model with the environment, determining the exploration probability based on the training period, and determining the action vector in the current state based on the exploration probability; storing the current state, the action vector, the reward value, and the next state as an experience bar in an experience pool; randomly selecting a preset number of experience bars from the experience pool when the experience pool is full; performing prediction using the target network for each experience bar to calculate the target value of the resource allocation model; evaluating the prediction loss based on the target Q value and the actual Q value corresponding to all experience bars, and updating the weight parameters of the current resource allocation model.

[0056] In practice, the delay matrix, transmission distance matrix, and current beam allocation pattern are normalized to unify the data scale. Row concatenation is used to flatten and merge these matrices into a single feature vector, which is then used as the current state and input into the resource allocation model. Upon receiving the current state, the resource allocation model begins training. During training, the exploration probability is adjusted according to the training cycle to determine whether to perform random exploration or select the optimal action using the current model. If random exploration is chosen, a random action vector is selected from all possible action vectors. If the optimal action is selected using the current model, the current state vector is input into the resource allocation model, and the Q-network predicts the optimal action vector. After determining the optimal action vector, beam resource allocation is adjusted based on it, and scheduling is executed. The reward value, r, after scheduling is calculated. t =-delay t ; wherein, the r t The delay is the reward value; t To average latency, the data packet buffer state is updated based on the reward value to obtain the next state information. The current state, action vector, reward value, and next state are recorded and stored in the experience pool. When the experience pool capacity reaches a set value, a preset number of experience bars are randomly selected as training samples. For each experience bar, the target network predicts the Q-value of the next state, and the target Q-value is calculated by combining it with the reward value. By calculating the error between the target Q-value and the current Q-value, the resource allocation model is trained through backpropagation and gradient descent, and the weight parameters of the Q-network are updated. The target Q-value is typically calculated by the target network using the following formula: Q target =R+γmaxQ next (s′,a′;θ - ); where Q target The target Q value; R is the current reward value; γ is the discount factor; Q next (s′,a′;θ - Let Q be the Q-value of the optimal action a' in the next state s'. The current Q-value is calculated by the main Q-network based on the current state and the current action: Q current =Q(s, a; θ); where Q current Let be the current Q-value; s be the current state; a be the current action; and θ be the parameters of the current Q-network. Further, the mean squared error is used to calculate the error between the target Q-value and the current Q-value: Wherein, L(θ) is the mean square error; N is the size of the training samples; and Q... target,i The target Q value; the Q current,i This is the current Q value. The gradient of the mean squared error with respect to the model weight parameters is calculated using the error backpropagation algorithm: Among them, the Let L(θ) be the gradient of the mean squared error with respect to the weight parameters; L(θ) is the mean squared error; and θ is the weight parameter. Update the weight parameters using either gradient descent or adaptive optimization algorithms. Wherein, θ is the weight parameter; α is the learning rate, which controls the update step size; Let be the gradient of the mean squared error with respect to the weight parameters. Repeat the above process, continuously calculating the loss, backpropagating the gradient, and updating the weight parameters, until convergence or the maximum number of training steps is reached.

[0057] Optionally, when the idle ratio of the experience pool is greater than a preset value, the step of determining the exploration probability based on the training cycle includes: determining the termination exploration probability and the initial exploration probability; calculating a first difference between the termination exploration probability and the initial exploration probability; calculating the quotient of the first difference and the training cycle; and determining a second difference between the termination exploration probability and the quotient as the exploration probability; wherein the exploration probability under the first training cycle is greater than the exploration probability under the second training cycle, and the difference between the first training cycle and the second training cycle is greater than a preset threshold. Figure 4 This diagram illustrates the change in exploration probability over the training period, as provided in this application. Please refer to... Figure 4 It can be seen that the exploration probability is larger when the training period is short, and gradually decreases until it stabilizes as the training period increases.

[0058] The method provided by this invention uses a large variation in exploration probability to optimize the model when the experience pool is relatively empty, and uses a small variation in exploration probability to optimize the model when the experience pool is relatively full. The adjustment of the rate of change effectively speeds up the model training and improves the matching relationship between model training efficiency, training quality and experience pool resources.

[0059] Specifically, the initial exploration probability typically refers to the early stages of reinforcement learning. To ensure the Q-network can fully explore the environment, a high exploration probability is adopted, usually close to 1, such as 1.0 or 0.9. At the beginning of training, the resource allocation model tends to choose random actions rather than the best action predicted by the current Q-network. The termination exploration probability refers to the probability that needs to be reduced as the reinforcement learning model gradually accumulates experience and the policy stabilizes, making the resource allocation model more inclined to utilize the currently learned optimal policy. The termination exploration probability is usually set to a small value (such as 0.01 or 0.05) to ensure that the resource allocation model relies mainly on the Q-network's decisions rather than random selection in the later stages of training. The exploration probability gradually decreases from the initial exploration probability to the termination exploration probability as the training cycle progresses, controlling the exploration behavior of the resource allocation model at different stages.

[0060] In practice, the exploration probability can be calculated using the following formula:

[0061] ε=ε e -(ε e -ε s ) / i;

[0062] Wherein, ε is the exploration probability; e The probability of terminating the exploration; the ε s is the initial exploration probability; i is the training period.

[0063] It's important to note that the changing trend of the exploration probability reflects a gradual transition from high exploration to high utilization. This ensures that the resource allocation model can fully explore the environment in the early stages of training, avoiding local optima, while simultaneously converging stably to the optimal strategy in the later stages. The initial high exploration probability causes the resource allocation model to tend to randomly select actions to acquire diverse empirical data. As training progresses, the exploration probability steadily decreases, and the resource allocation model gradually relies on the optimal strategy predicted by the Q-network, while still maintaining a certain degree of randomness to balance exploration and utilization. Near the end of training, the exploration probability approaches its termination value, and the resource allocation model primarily relies on learned strategies for decision-making, improving the stability and efficiency of resource allocation. This design facilitates a breadth-first search followed by depth-first optimization, enabling the resource allocation strategy to converge efficiently within a limited training period.

[0064] Optionally, the step of evaluating the prediction loss based on the target value and actual data corresponding to all experience bars and updating the weight parameters of the current resource allocation model includes: using the output value of the current resource allocation model and the target value, calculating the difference between the output value and the target value through a loss function; when the difference is less than a preset threshold, calculating the loss function value using a squared interpolation method; when the difference is greater than the preset threshold, calculating the loss function value using a linear interpolation method; calculating the gradient using a backpropagation algorithm based on the loss function value; and updating the weight parameters of the current resource allocation model based on the gradient.

[0065] In practice, the loss function can be calculated based on the following formula:

[0066]

[0067] Wherein, loss(x, y) is the loss function; n is the number of experience bars; x i The output value; the y i This is the target value.

[0068] The method provided in this embodiment matches different loss function calculation methods based on the relationship between error and a preset threshold. This can accelerate convergence speed while ensuring training stability and improving the robustness and prediction accuracy of the model. When the error is small, the squared difference calculation method is used, which is quadratic with respect to the error, making the optimization process more accurate and helping the model to converge stably to the optimal solution. When the error is large, the linear interpolation method is used, making the gradient change of the loss function smoother and avoiding training instability or gradient explosion caused by excessive gradients, thereby accelerating the model convergence speed and reducing training oscillations. In addition, this strategy can optimize the gradient update strategy, ensuring fine-grained fine-tuning when the error is small to avoid over-adjustment, and large-scale updates when the error is large to quickly correct model bias and shorten training time. The advantage of this method is that it balances convergence speed and training accuracy, enabling the model to learn more efficiently and maintain good stability and adaptability at different training stages.

[0069] Optionally, the action vector satisfies constraints, including a first constraint, a second constraint, a third constraint, and a fourth constraint. The first constraint indicates that the number of beams illuminated at the current time is less than the maximum number of satellite beams. The second constraint indicates that the sum of the power of all beams illuminated at the current time is less than the total available power of the satellite. The third constraint indicates that the power of any illuminated beam does not exceed the maximum power of a single beam. The fourth constraint indicates that the dwell time of any data packet does not exceed a delay threshold.

[0070] Specifically, the first constraint can be expressed as:

[0071]

[0072] Among them, the The departure time of the data packet; The arrival time of the data packet is T; where T is the time slot t within the range from the start time to the current time. j The set of; O is the set of ground wavefronts currently served by the satellite; pac is the set of ground wavefronts served by the satellite at t j A data packet that leaves the data packet buffer and proceeds to position i at time t. k The data packet arrives in the buffer at a specific time; This represents the number of wavelengths illuminated in the current time slot.

[0073] The second constraint can be expressed as:

[0074]

[0075] Among them, the This represents the current illumination status of the time slot beams; K is the maximum number of satellite beams; T is the time slot t within the range from the start time to the current time. j The set of O; where O is the set of ground wave positions currently served by the satellite.

[0076] The third constraint can be expressed as:

[0077]

[0078] Among them, the The power of a certain wave position in the current time slot; the P tot The total available power on the satellite; T is the time slot t from the start time to the current time. j A set of.

[0079] The fourth constraint can be expressed as:

[0080]

[0081] Among them, the The power of the current time slot wave position i; the P b The maximum power of a single beam; T is the time slot t within the range from the start time to the current time. j A set of.

[0082] The fifth constraint can be expressed as:

[0083]

[0084] Among them, the The departure time of the data packet; The arrival time of the data packet; T th This is the time delay threshold.

[0085] S104. Based on the updated resource allocation model, predict the expected return value, sort the predicted return value in descending order, determine the corresponding beam based on the action vector corresponding to the first preset number of expected return values, and light up the beam.

[0086] In a specific implementation, the step of arranging the expected reward values ​​in descending order, determining the corresponding beam based on the action vector corresponding to the first preset number of expected reward values, and lighting the beam includes: traversing the position of each beam and calculating the distance between every two beams; after lighting any beam, lighting the next beam whose distance is greater than a preset value based on the distance between the beam and other beams, until all beams are lit.

[0087] Specifically, the system obtains the position coordinates (e.g., polar or Cartesian coordinates) of all candidate beams and stores the position index of each candidate beam and its corresponding expected reward value. It calculates the distance between every two beams using Euclidean distance or other suitable metrics, constructs a distance matrix, and records the pairwise distances between all beams. Based on the calculated expected reward values, the beams are sorted in descending order, generating a sorted beam index list. The beam with the highest expected reward value is selected as the first beam to be lit and marked as "lit". The sorted beam list is traversed, checking the distance between each beam and the lit beams. If the distance between a beam and all lit beams is greater than a preset threshold, the beam is lit and marked as "lit". This process is repeated until the entire beam list has been traversed or the beam lighting condition is met.

[0088] The method provided in this embodiment optimizes beam-hopping resource allocation through deep reinforcement learning to adapt to the rapid channel changes caused by the high-speed movement of low-Earth orbit satellites, ensuring efficient resource scheduling, reducing resource waste, and lowering data transmission latency. First, this application obtains the state information of all arriving data packets based on the data packet buffer of the onboard equipment and constructs a delay matrix through state reconstruction. Since data packets arrive at different buffers at different times, directly using the raw data may lead to uneven distribution of input data, affecting the convergence speed and generalization ability of reinforcement learning. Therefore, this application employs a normalization operation to keep the data input within a stable numerical range, reducing the interference of time characteristics on the learning process and enabling the model to better learn the dynamic characteristics of data packet changes. The method of constructing the delay matrix effectively characterizes the traffic changes of data packets at different times, providing reasonable input for subsequent resource allocation.

[0089] Secondly, during the reinforcement learning training phase, this application utilizes an experience pool to store interaction data and dynamically adjusts the exploration probability based on the training cycle to balance exploration and utilization. In reinforcement learning tasks, if the exploration rate is too high, the model may frequently attempt random actions, making it difficult to converge to the optimal policy; if the exploration rate is too low, it may converge to a local optimum prematurely, causing the resource allocation scheme to be unable to adapt to changes in the environment. By dynamically adjusting the exploration probability, the model can extensively explore different resource allocation strategies in the early stages of training and gradually utilize the learned optimal policy in the later stages, improving the efficiency and stability of resource scheduling. Furthermore, the experience pool storage mechanism allows training data to be reused, and the random sampling mechanism avoids the problem of excessive data correlation, improving training stability and generalization ability.

[0090] During the resource allocation decision-making phase, the model predicts expected returns and sorts them in descending order to ensure that the resource allocation scheme with the highest return is prioritized. The Q-value (i.e., expected return value) under the reinforcement learning framework can dynamically evaluate the long-term benefits of different actions, thus making beam resource allocation more intelligent. Compared to traditional heuristic methods or scheduling strategies based on fixed rules, this application can adjust the allocation scheme in real time according to changes in the environment, thereby achieving better resource utilization efficiency under different business needs and network conditions.

[0091] Corresponding to the aforementioned embodiment of a beam hopping resource allocation method based on deep reinforcement learning, this application also provides an embodiment of a beam hopping resource allocation device based on deep reinforcement learning.

[0092] Figure 5 This is a schematic diagram of the beam-hopping resource allocation device based on deep reinforcement learning provided in Embodiment 2 of this application. Please refer to... Figure 5 The apparatus provided in this embodiment includes an acquisition module 210, a reconstruction module 220, a prediction module 230, and a determination module 240.

[0093] The acquisition module 210 is used to acquire the status information of all arriving data packets in each data packet buffer based on the data packet buffer provided by the spaceborne equipment.

[0094] The reconstruction module 220 is used to reconstruct the state of the data packet based on the state information to obtain a delay matrix corresponding to each data packet buffer; wherein, the data packets are normalized according to the arrival time of the data packets under different data packet buffers; the delay matrix represents the number of data packets arriving at the data packet buffer at different times;

[0095] The prediction module 230 is used to input the time delay matrix into the trained resource allocation model to predict different action vectors and their corresponding expected reward values. Specifically, it constructs a reinforcement learning experience pool, dynamically adjusts the exploration probability based on the training cycle, and determines the action vector in the current state based on the exploration probability. It executes the action vector, adjusts beam resource allocation, and updates the state of the data packet buffer based on the scheduling result. It calculates the reward value of the action vector in the current state and stores the current state, the action vector, the reward value, and the next state in the experience pool. When the experience pool is full, it selects training samples from the experience pool using a random sampling mechanism. It calculates the objective function value based on the training samples, constructs a loss function based on the objective function value, and updates the weight parameters of the resource allocation model based on the loss function.

[0096] The determining module 240 is used to predict the expected return value based on the updated resource allocation model, sort the predicted return value in descending order, determine the corresponding beam based on the action vector corresponding to the first preset number of expected return values, and light up the beam.

[0097] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0098] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0099] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0100] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A beam-hopping resource allocation method based on deep reinforcement learning, characterized in that, The method includes: Based on the data packet buffers equipped on the spaceborne equipment, obtain the status information of all arriving data packets in each data packet buffer; Based on the state information, the state of the data packets is reconstructed to obtain the delay matrix corresponding to each data packet buffer; wherein, the data packets are normalized according to the arrival time of the data packets under different data packet buffers; the delay matrix represents the number of data packets arriving at the data packet buffer at different times; The time delay matrix is ​​input into the trained resource allocation model to predict different action vectors and their corresponding expected reward values. Specifically, an experience pool for reinforcement learning is constructed, and the exploration probability is dynamically adjusted based on the training cycle. The action vector in the current state is determined based on the exploration probability. The action vector is executed, beam resource allocation is adjusted, and the state of the data packet buffer is updated based on the scheduling result. The reward value of the action vector in the current state is calculated, and the current state, the action vector, the reward value, and the next state are stored in the experience pool. When the experience pool is full, a random sampling mechanism is used to select training samples from the experience pool. A target Q-value is calculated based on the training samples, a loss function is constructed based on the target Q-value, and the weight parameters of the resource allocation model are updated based on the loss function. Based on the updated resource allocation model, the expected return value is predicted, the expected return value is sorted in descending order, and the corresponding beam is determined based on the action vector corresponding to the first preset number of expected return values, and the beam is lit up. The dynamic adjustment of the exploration probability based on the training cycle includes: Determine the exploration probability of the previous training cycle as the initial exploration probability; Calculate the current free proportion of the experience pool as the initial adjustment coefficient; Using the initial exploration probability as input, the repetition of the next training result with the existing stored information in the current experience pool is estimated based on the current training cycle, and the current rate of change is determined by the product of the repetition and the rate of change of the exploration probability in the previous training cycle. The candidate exploration probability for the current training cycle is determined by the product of the initial exploration probability and the current rate of change. The final exploration probability value is determined by multiplying the candidate exploration probability and the initial adjustment coefficient.

2. The method according to claim 1, characterized in that, The step of reconstructing the state of the data packets based on the state information to obtain the delay matrix corresponding to each data packet buffer includes: Obtain the status information for each data packet buffer; the status information includes the arrival time of different data packets arriving at the data packet buffer. The time window size is set based on different arrival times, and the window is slid based on the time window size to normalize data packets with different arrival times into a unified time window. Calculate the number of data packets in each data packet buffer within different time windows; A delay matrix is ​​constructed using the data packet buffer as the row index, the time window as the column index, and the matrix elements representing the number of data packets in the data packet buffer within the time window.

3. The method according to claim 1, characterized in that, The training process of the resource allocation model includes: Based on the time delay matrix, transmission distance matrix, and current beam assignment pattern, the current state is obtained; The resource allocation model interacts with the environment, the exploration probability is determined based on the training period, and the action vector in the current state is determined based on the exploration probability. The current state, the action vector, the reward value, and the next state are stored as an experience bar in the experience pool. When the experience pool is full, a preset number of experience bars are randomly drawn from the experience pool; For each experience bar, the target network is used to make predictions and calculate the target Q value of the resource allocation model; The predicted loss is evaluated based on the target Q-value and actual Q-value corresponding to all experience bars, and the weight parameters of the current resource allocation model are updated.

4. The method according to claim 3, characterized in that, The step of evaluating the prediction loss based on the target Q-value and actual Q-value corresponding to all experience bars, and updating the weight parameters of the current resource allocation model, includes: Using the output value of the current resource allocation model and the target Q value, the difference between the output value and the target Q value is calculated through a loss function; When the difference is less than a preset threshold, the loss function value is calculated using the squared difference calculation method; When the difference is greater than a preset threshold, the loss function value is calculated using a linear interpolation method; Based on the loss function value, the gradient is calculated using the backpropagation algorithm, and the weight parameters of the current resource allocation model are updated based on the gradient.

5. The method according to claim 1, characterized in that, The resource allocation model includes a Q-network, which is composed of a convolutional neural network and a deep neural network; the step of inputting the time delay matrix into the trained resource allocation model to predict different action vectors and corresponding expected reward values ​​includes: The convolutional neural network extracts spatial features from the time delay matrix based on convolutional layers to obtain the distribution of data packets at different times and beam positions; The spatial features are combined with the transmission distance matrix and the current beam allocation pattern to obtain the combined feature vector. The deep neural network processes the concatenated feature vector based on multiple hidden layers, fits the nonlinear relationship between the feature vector and the action vector, and determines the corresponding action vector and the corresponding expected reward value.

6. The method according to claim 1, characterized in that, The method, based on the data packet buffers equipped on the spaceborne equipment, obtains the status information of all arriving data packets in each data packet buffer, including: Predict the size of data packets for different links or service types; Under the current working environment of the spaceborne equipment, obtain a distribution map of the number of communication objects and determine the communication type of each communication object in the distribution map. A matching relationship is established in the distribution map of the number of communication objects, wherein the matching relationship is the two communication objects that are communicating; Based on the communication type and the current working environment, predict the data packet size of each communication object in the communication object quantity distribution diagram; The total data packet size of the communication object quantity distribution map is determined based on the matching relationship and the data packet size.

7. The method according to claim 2, characterized in that, The method of setting the time window size based on different arrival times includes: Determine the time distribution gap of the data packets received in the current buffer period, wherein the time distribution gap is the difference between the earliest arrival time and the latest arrival time; The time window adjustment coefficient is determined based on the ratio of the time distribution difference of data packets in the current cache period to the time distribution difference of data packets in the previous cache period. The endpoint values ​​of each time window in the previous cache period are multiplied by the time window adjustment coefficient to obtain the final time window of the current cache period.

8. A beam-hopping resource allocation device based on deep reinforcement learning, characterized in that, The device includes an acquisition module, a reconstruction module, a prediction module, and a determination module; The acquisition module is used to acquire the status information of all arriving data packets in each data packet buffer based on the data packet buffer provided by the spaceborne equipment. The reconstruction module is used to reconstruct the state of the data packets based on the state information to obtain a delay matrix corresponding to each data packet buffer; wherein, the data packets are normalized according to the arrival time of the data packets under different data packet buffers; the delay matrix represents the number of data packets arriving at the data packet buffer at different times; The prediction module is used to input the time delay matrix into the trained resource allocation model to predict different action vectors and their corresponding expected reward values. Specifically, it constructs a reinforcement learning experience pool, dynamically adjusts the exploration probability based on the training cycle, and determines the action vector in the current state based on the exploration probability. It executes the action vector, adjusts beam resource allocation, and updates the state of the data packet buffer based on the scheduling result. It calculates the reward value of the action vector in the current state and stores the current state, the action vector, the reward value, and the next state in the experience pool. When the experience pool is full, it selects training samples from the experience pool using a random sampling mechanism. It calculates a target Q-value based on the training samples, constructs a loss function based on the target Q-value, and updates the weight parameters of the resource allocation model based on the loss function. The determining module is used to predict the expected return value based on the updated resource allocation model, sort the expected return value in descending order, determine the corresponding beam based on the action vector corresponding to the first preset number of expected return values, and light up the beam. The dynamic adjustment of the exploration probability based on the training cycle includes: Determine the exploration probability of the previous training cycle as the initial exploration probability; Calculate the current free proportion of the experience pool as the initial adjustment coefficient; Using the initial exploration probability as input, the repetition of the next training result with the existing stored information in the current experience pool is estimated based on the current training cycle, and the current rate of change is determined by the product of the repetition and the rate of change of the exploration probability in the previous training cycle. The candidate exploration probability for the current training cycle is determined by the product of the initial exploration probability and the current rate of change. The final exploration probability value is determined by multiplying the candidate exploration probability and the initial adjustment coefficient.

Images