Satellite beam adjustment decision method and system based on reinforcement learning

Abstract

This invention provides a satellite beam adjustment decision-making method based on reinforcement learning, comprising: initializing system parameters; repeatedly performing the following operations until a preset total number of iterations is reached: calculating a first system score for the current scheme based on several system evaluation indicators; invoking a reinforcement learning module to generate a joint optimization scheme of frequency and power through a PPO agent; updating the agent state according to a reward function, wherein the reward function includes a capacity matching reward item and power over-limit penalty factors, bandwidth over-limit penalty factors, co-channel interference penalty factors, and resource reuse conflict penalty factors; calculating a second system score for the joint optimization scheme, and deciding whether to adopt the joint optimization scheme based on a dual threshold; updating the user position and entering the next loop. Therefore, this invention can solve the balance problem between on-board resource utilization efficiency and resource adjustment overhead, effectively improving system resource utilization and user demand satisfaction rate.

Description

Technical Field

[0001] This invention relates to the field of satellite communication technology, and in particular to a satellite beam adjustment decision-making method and system based on reinforcement learning. Background Technology

[0002] The next generation of high-throughput satellites will achieve throughputs of hundreds of gigabits per second or even terabits per second, with the number of beams increasing from dozens to hundreds. In practical applications, communication between satellites and various user terminals, such as those mounted on vehicles and ships, as well as fixed ground stations, presents challenges to the overall scheduling and optimization of different satellite resources. Traditional high-throughput satellites have issues such as regional limitations in meeting user needs and resource waste.

[0003] The application scenarios of frequency planning and scheduling are reflected in two aspects.

[0004] First, during the design phase of multi-beam satellites, the frequency planning and arrangement determine core performance indicators such as the satellite system's payload-to-interference ratio and effective capacity. Furthermore, it also affects the number of payload system channels, the types and quantities of active / passive units, and consequently, the system's weight, power consumption, and heat dissipation.

[0005] Secondly, during the on-orbit application phase of multi-beam satellites, due to the increasing diversification of user needs and the dynamic changes in time and space throughout the entire life cycle, a frequency planning and arrangement scheme that better matches user needs must be dynamically generated and uploaded to the on-board payload.

[0006] Existing methods for planning communication missions for flexible satellites have three significant drawbacks:

[0007] (1) Current task planning algorithms often suffer from the problem of transponder saturation in hot areas and idle transponders in non-hot areas due to uneven user distribution and fluctuating business demand. This results in uneven beam activity and the actual application capacity is less than or even far less than the design capacity.

[0008] (2) Current research on task planning algorithms has not taken into account the different needs of different user groups, such as the need for low latency for specific users and the communication guarantee for high-priority users.

[0009] (3) Current research on resource planning algorithms often focuses on achieving the optimal allocation of a certain type of resource, without fully considering the problem of joint optimization of multi-dimensional resources. Summary of the Invention

[0010] To address the problems existing in the prior art, the present invention aims to provide a satellite beam adjustment decision-making method and system based on reinforcement learning, so as to improve system resource utilization and user demand satisfaction.

[0011] To achieve the aforementioned technical effects, on the one hand, this invention provides a satellite beam adjustment decision-making method based on reinforcement learning, comprising the following steps:

[0012] Initialize system parameters, including user heatmap matrix, beam-user relationship matrix, user demand capacity satisfaction rate matrix, number of frequency band resource blocks, available capacity vector of each beam, power vector of each beam, total system power limit, available frequency bands, and gain set of each beam;

[0013] Repeat the following operations until the preset total number of times is reached:

[0014] a. Calculate the first system score of the current scheme based on several system evaluation indicators; the several system evaluation indicators include at least user capacity satisfaction rate, power saving, interruption loss and hardware loss, and the current scheme is the beam resource allocation scheme currently being implemented by the satellite system;

[0015] b. Call the reinforcement learning module to generate a joint optimization scheme for frequency and power through the PPO (Proximal Policy Optimization) agent;

[0016] c. Update the agent state according to the reward function, which includes a capacity matching reward item and a power over-limit penalty factor, a bandwidth over-limit penalty factor, a co-channel interference penalty factor, and a resource reuse conflict penalty factor;

[0017] d. Calculate the second system score of the joint optimization scheme, and decide whether to adopt the joint optimization scheme based on a dual threshold:

[0018] If the score of the first system is lower than the first threshold, then the joint optimization scheme is adopted;

[0019] If the score of the first system is higher than the first threshold and the score difference between the first system score and the second system score is greater than the second threshold, then the joint optimization scheme is adopted;

[0020] Otherwise, maintain the current solution;

[0021] e. Update the user's location and proceed to the next loop.

[0022] Optionally, the user heatmap matrix contains n u The column contains user information, and each column of user information includes the user's longitude. i latitude i The required communication capacity c for the user i , i∈{1,2,…,n u}

[0023] Optionally, the beam-user relationship matrix is ​​determined in the following way:

[0024] If the angle between the user's connection to the satellite and the center of the same beam is less than half the beamwidth angle, then the user is marked as belonging to that beam.

[0025] Optionally, the reward function is:

[0026] Where b is the beam number, n b r is the number of beams. CapacityMatch,b For the capacity matching rate of beam b, discount power,b Discount is the power constraint violation factor for beam b. bandwidth,b Discount is the bandwidth constraint violation factor for beam b. interference Discount is the system's co-channel interference violation factor. multiuse This refers to the system's resource reuse violation factor.

[0027] Optionally, the capacity matching rate of beam b is:

[0028]

[0029] Among them, c b =∑ u∈b c u C represents the sum of user demand capacity for beam b. b Let factor be the available capacity vector of beam b. capacity This is a preset fixed value;

[0030] The power constraint violation factor for beam b is:

[0031]

[0032] Among them, P b Let P be the power vector of beam b. max The factor represents the upper limit of the total power of the system. punish1 This is a preset fixed value;

[0033] The bandwidth constraint violation factor for beam b is:

[0034]

[0035] Where, n initial,b B is the starting frequency resource block number for beam b. b B is the bandwidth of beam b. singlr n represents the bandwidth of a single frequency resource block. blocks The number of frequency band resource blocks;

[0036] The co-channel interference violation factor of the system is:

[0037] discount interference =n 存在干扰的波束数 ·factor punish2 ;

[0038] Where, factor punish2 This is a preset fixed value; n 存在干扰的波束数 The number of beams that are subject to interference;

[0039] The resource reuse violation factor of the system is:

[0040] discount multiuse =n 波束间被重复使用的资源块数 ·factor punishi3 ;

[0041] Where, factor punishi3 n is a preset fixed value. 波束间被重复使用的资源块数 The number of resource blocks that are reused between beams.

[0042] Optionally, the first threshold is a combined tolerance threshold for cost increases and service decreases when maintaining the current scheme; the second threshold is a minimum net revenue threshold required to adopt the joint optimization scheme.

[0043] Optionally, the system score is calculated based on the following formula:

[0044]

[0045] Where, x j (j = 1, 2, ..., n) represents the system index, w j (j = 1, 2, ..., n) are the weights corresponding to the system indicators.

[0046] On the other hand, based on the same inventive structure, the present invention also provides a reinforcement learning-based satellite beam adjustment decision system for implementing the method described above.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) This invention proposes a reconstructing index system based on user heat map calculation system service satisfaction rate, beam power saving, interruption comprehensive loss, hardware loss, etc., and solves the balance between on-board resource utilization efficiency and resource adjustment overhead through a threshold-based reconstructing decision method.

[0049] (2) This invention proposes a fully flexible system frequency and power joint optimization resource management algorithm based on reinforcement learning, which effectively improves the system resource utilization rate and user demand satisfaction rate. Attached Figure Description

[0050] Figure 1 A flowchart illustrating the steps of a satellite beam adjustment decision-making method based on reinforcement learning, as provided in an embodiment of the present invention.

[0051] Figure 2 A flowchart illustrating the steps of a reinforcement learning-based satellite beam adjustment decision-making method in a cyclic execution operation, as provided in an embodiment of the present invention.

[0052] Figure 3 A flowchart illustrating the specific process of a satellite beam adjustment decision-making method based on reinforcement learning, as provided in an embodiment of the present invention.

[0053] Figure 4 A schematic diagram illustrating the change trend of agent reward over time in a satellite beam adjustment decision-making method based on reinforcement learning provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram illustrating the trend of average user satisfaction rate over time in a satellite beam adjustment decision method based on reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0056] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0057] Before describing the embodiments of this application in detail, the technical concept of this application is briefly described: The core concept of this invention lies in using reinforcement learning to achieve dynamic joint optimization of satellite beam resources, and balancing resource utilization and adjustment costs through a dual-threshold decision-making mechanism. Specifically, this includes: reflecting changes in user location and capacity demand in real time through a user heatmap matrix; using a PPO agent to synchronously optimize frequency allocation and power control, with the reward function taking into account both capacity matching rewards and four types of penalty items (power / bandwidth over-limit, co-channel interference, and resource reuse conflict); determining whether to activate a new scheme based on dual thresholds: forced switching when the comprehensive score of the original scheme is lower than this value (service quality cannot be guaranteed); or selective switching when the score difference between the old and new schemes is higher than this value (the benefits are sufficient to cover the adjustment costs). Ultimately, this improves satellite resource utilization and solves the three major defects of traditional methods: uneven workload, neglect of demand, and one-dimensional optimization.

[0058] This invention is mainly aimed at the communication environment between GEO (Geostationary Earth Orbit) satellites, mobile user terminals, and fixed ground stations, and aims to coordinate and schedule these resources to improve the utilization rate of flexible satellite resources.

[0059] The specific principles of the reinforcement learning-based satellite beam adjustment decision method of this application will be described below with reference to specific embodiments.

[0060] Figure 1 This invention illustrates a satellite beam adjustment decision-making method based on reinforcement learning, comprising the following steps:

[0061] S101: Initialize system parameters, including user heatmap matrix, beam-user relationship matrix, user demand capacity satisfaction rate matrix, number of frequency band resource blocks, available capacity vector of each beam, power vector of each beam, upper limit of total system power, available frequency bands, and gain set of each beam.

[0062] The user heatmap matrix in this embodiment contains n u The column contains user information, and each column of user information includes the user's longitude. i latitude i The required communication capacity c for the user i , i∈{1,2,…,n u}

[0063] The beam-user relationship matrix is ​​determined as follows: if the angle between the line connecting the user and the satellite and the beam center is less than half the beamwidth angle, then the user is marked as belonging to that beam. Specifically, in the beam-user relationship matrix M... relationshipIn this process, the beam to which a user belongs is determined based on the relationship between the angle between the connection between the user and the satellite and the connection between the beam center and the satellite, and the half-beamwidth angle. If the element is 1, it means that a user belongs to a certain beam; otherwise, it is 0.

[0064] User demand capacity fulfillment rate matrix M satisfied Initialize to 0;

[0065] The number of frequency resource blocks n initialized blocks ;

[0066] Initialize the number of beams n b n b It is a positive integer;

[0067] The capacity vectors C and C can be used to initialize each beam. i A positive number, in Mbps, i∈{1,2,…,n b};

[0068] Initialize the power vector P of each beam, with each element being the current power P of each beam. i (0≤P i ≤single_beam_power max ), i∈{1,2,…,n b};

[0069] Initialize the system's maximum total power P max ;

[0070] Initialize the available frequency band [d1, d2], where d1 and d2 are positive and in MHz, with d2 greater than d1, to obtain the bandwidth of a single frequency resource block.

[0071] Initialize the beam gain set G i , i∈{1,2,…,n b}

[0072] S102: Perform the following operations repeatedly until the preset total number of times is reached. In this embodiment, the total number of loop executions is N, and the counter t = 1:

[0073] The specific loop process is as follows: Figure 2 As shown, the process includes the following:

[0074] S111: Calculate the first system score of the current scheme based on several system evaluation indicators; the several system evaluation indicators include at least user capacity satisfaction rate, power saving amount, interruption loss and hardware loss, and the current scheme is the beam resource allocation scheme currently being implemented by the satellite system.

[0075] In practical implementation, an evaluation index system is adopted, including indicators such as system user capacity fulfillment rate, power saving, interruption loss, and hardware loss. This system is used to obtain the x values ​​for each indicator. j The overall weight w of (j=1,2,…,n) j After (j = 1, 2, ..., n), the current system score is calculated; the system score is calculated based on the following formula:

[0076]

[0077] S112: Invoke the reinforcement learning module to generate a joint optimization scheme for frequency and power through the PPO agent.

[0078] S113: Update the agent state according to the reward function, which includes a capacity matching reward item and a power over-limit penalty factor, a bandwidth over-limit penalty factor, a co-frequency interference penalty factor, and a resource reuse conflict penalty factor.

[0079] The reward function in this embodiment is:

[0080] Where b is the beam number, n b r is the number of beams. CapacityMatch,b For the capacity matching rate of beam b, discount power,b Discount is the power constraint violation factor for beam b. bandwidth,b Discount is the bandwidth constraint violation factor for beam b. interference Discount is the system's co-channel interference violation factor. multiuse This refers to the system's resource reuse violation factor.

[0081] That is, by calculating the reward function r jointly optimized by frequency and power, and updating the agent's state.

[0082] Furthermore, the capacity matching rate of beam b is:

[0083]

[0084] Among them, c b =∑ u∈b c u C represents the sum of user demand capacity for beam b. b Let factor be the available capacity vector of beam b. capacity This is a preset fixed value;

[0085] The power constraint violation factor for beam b is:

[0086]

[0087] Among them, P b Let P be the power vector of beam b. max The factor represents the upper limit of the total power of the system. punish1 This is a preset fixed value;

[0088] The bandwidth constraint violation factor for beam b is:

[0089]

[0090] Where, n initial,b B is the starting frequency resource block number for beam b. b B is the bandwidth of beam b. single n represents the bandwidth of a single frequency resource block. blocks The number of frequency band resource blocks;

[0091] The co-channel interference violation factor of the system is:

[0092] discount interference =n 存在干扰的波束数 ·factor punish2 ;

[0093] Where, factor punish2 This is a preset fixed value; n 存在干扰的波束数 The number of beams that are subject to interference;

[0094] The system's resource reuse violation factor is:

[0095] discount multiuse =n 波束间被重复使用的资源块数 ·factor punishi3 ;

[0096] Where, factor punishi3 n is a preset fixed value. 波束间被重复使用的资源块数 The number of resource blocks that are reused between beams.

[0097] S114: Calculate the second system score of the joint optimization scheme, and decide whether to adopt the joint optimization scheme based on the dual threshold:

[0098] If the score of the first system is lower than the first threshold, a joint optimization scheme is adopted;

[0099] If the score of the first system is higher than the first threshold and the score difference between the first system score and the second system score is greater than the second threshold, then the joint optimization scheme is adopted.

[0100] Otherwise, maintain the current solution;

[0101] S115: Update user location and proceed to the next loop.

[0102] See Figure 3 The specific process of the method described in this embodiment is as follows:

[0103] (1) Initialize the input, specifically the system parameters mentioned above.

[0104] (2) Input the total number of loop executions N, and the counter t = 1.

[0105] (3) An evaluation index system is adopted, including indicators such as system user capacity satisfaction rate, power saving, interruption loss, and hardware loss. This system is used to obtain the x values ​​for each indicator. j The overall weight w of (j=1,2,…,n) j After (j=1,2,…,n), calculate the current system comprehensive score:

[0106]

[0107] (4) Call the frequency-power joint optimization module based on reinforcement learning to initialize the machine learning environment, and record the step variable step=1.

[0108] (5) The PPO agent makes decisions.

[0109] (6) Calculate the frequency-power joint optimization reward function And update the agent's state.

[0110] (7) Determine whether the maximum number of steps has been reached. If yes, proceed to step (9); otherwise, proceed to step (8).

[0111] (8) Record the variable step = step + 1, and then repeat step (5).

[0112] (9) Calculate the system score of the joint optimized scheme, i.e. the final state of the agent.

[0113] (10) Determine whether the current system score is less than the threshold th1 (i.e., the first threshold), where the threshold th1 represents the overall tolerance level for the increase in cost and the decrease in service quality that may result from maintaining the current scheme unchanged. If the overall score of the original scheme under the new conditions is lower than th1, it means that maintaining the original scheme cannot well meet the communication service requirements under the new conditions, and the current beam resource allocation scheme should be changed. At this time, step (13) should be performed; otherwise, if it is higher than th1, step (11) should be performed.

[0114] (11) Calculate the difference between the score of the joint optimization scheme in step (9) and the score of the original scheme, and determine whether the difference is greater than the system comprehensive score threshold th2 (i.e., the second threshold). The threshold th2 refers to the overall acceptance level of the possible cost increase and service quality decrease brought about by implementing the new scheme if there is a new scheme with a higher comprehensive score, but maintaining the original scheme can still ensure that the comprehensive score is higher than the threshold th1. When the difference between the system scores of the new scheme and the original scheme is lower than th2, it means that this reconstruction is not worthwhile, and thus the reconstruction is rejected, and step (8) is performed; otherwise, it means that the reconstruction is acceptable, and step (12) is performed.

[0115] (12) Change the existing planning scheme into a new scheme of joint optimization, and then proceed to step (14).

[0116] (13) Keep the original plan unchanged, and then proceed to step (14).

[0117] (14) Determine whether the counter t is greater than the total number of loops N. If it is, proceed to step (16); otherwise, proceed to step (15).

[0118] (15) The counter t = t + 1, update the user position, and then re-execute step (3).

[0119] (16) End the loop.

[0120] In a specific application example, the scenario is set as a GEO satellite with a nadir of 155°E, 0°N, the number of loops N is 300, and the number of users n u Take 1000, set the loop to 300 times, with 200 time steps each time; according to the method provided in this embodiment, the agent reward for each round is calculated as follows: Figure 4 As shown, the average user satisfaction rate is as follows: Figure 5 As shown, the average user satisfaction rate significantly improved after 300 iterations, validating the synergistic optimization of resource utilization and service quality.

[0121] The present invention also provides a satellite beam adjustment decision system based on reinforcement learning. The system is used to implement the method described in the above embodiments. Specifically, the process steps described above can be achieved through the collaborative cooperation between multiple functional modules. For specific functions that can be implemented, please refer to the method described in the above embodiments.

[0122] In summary, this invention offers the following advantages: 1. Improved resource utilization. By jointly optimizing frequency and power allocation through reinforcement learning, it dynamically matches beam resources with user needs, solving the "uneven workload" problem (overload in hotspot areas / idleness in non-hotspot areas) in traditional methods, making the actual satellite application capacity approach the design capacity. 2. Guaranteed differentiated service quality. Based on real-time perception of demand distribution using user heatmaps, combined with a capacity matching reward function, it significantly improves the user demand satisfaction rate. 3. Multi-dimensional resource collaborative optimization. It pioneers a frequency-power joint optimization model, breaking through the limitations of traditional single-dimensional resource scheduling, and achieves efficient system-level resource collaboration through a penalty term. 4. Reduced system operation and maintenance costs. The dual-threshold decision mechanism avoids ineffective reconfiguration, reduces signaling overhead and hardware wear caused by frequent adjustments, balances service quality and adjustment costs, and extends satellite payload lifespan. 5. Enhanced system anti-interference capability. The co-channel interference penalty term forcibly optimizes spectral spatial isolation, reduces the risk of communication interruption, and improves system stability.

[0123] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0124] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A satellite beam adjustment decision-making method based on reinforcement learning, characterized in that, Including the following steps: Initialize system parameters, including user heatmap matrix, beam-user relationship matrix, user demand capacity satisfaction rate matrix, number of frequency band resource blocks, available capacity vector of each beam, power vector of each beam, total system power limit, available frequency bands, and gain set of each beam; Repeat the following operations until the preset total number of times is reached: a. Calculate the first system score of the current scheme based on several system evaluation indicators; the several system evaluation indicators include at least user capacity satisfaction rate, power saving, interruption loss and hardware loss, and the current scheme is the beam resource allocation scheme currently being implemented by the satellite system; b. Invoke the reinforcement learning module to generate a joint optimization scheme for frequency and power through the PPO agent; c. Update the agent state according to the reward function, which includes a capacity matching reward item and a power over-limit penalty factor, a bandwidth over-limit penalty factor, a co-channel interference penalty factor, and a resource reuse conflict penalty factor; The reward function is: in, b Number the beams. For the number of beams, For the capacity matching rate of beam b, For beam b Power constraint violation factor For beam b bandwidth; d. Calculate the second system score of the joint optimization scheme, and decide whether to adopt the joint optimization scheme based on a dual threshold: If the score of the first system is lower than the first threshold, then the joint optimization scheme is adopted; If the score of the first system is higher than the first threshold and the score difference between the first system score and the second system score is greater than the second threshold, then the joint optimization scheme is adopted; Otherwise, maintain the current solution; The first threshold is a combined tolerance threshold for cost increases and service decreases when maintaining the current solution; the second threshold is a minimum net revenue threshold required to adopt the joint optimization solution. e. Update the user's location and proceed to the next loop.

2. The satellite beam adjustment decision method based on reinforcement learning according to claim 1, characterized in that, The user heatmap matrix includes The column contains user information, and each column of user information includes the user's longitude. ,latitude Communication capacity required by the user .

3. The satellite beam adjustment decision method based on reinforcement learning according to claim 1, characterized in that, The beam-user relationship matrix is ​​determined as follows: if the angle between the beam center and the satellite connection line is less than half the beamwidth angle, then the user is marked as belonging to that beam.

4. The satellite beam adjustment decision method based on reinforcement learning according to claim 1, characterized in that, The beam b The capacity matching rate is: in, Represented as beam b The sum of user demand capacity, For beam b Available capacity vector, This is a preset fixed value; The beam b The power constraint violation factor is: in, For beam b The power vector, This represents the upper limit of the total power of the system. This is a preset fixed value; The beam b The bandwidth constraint violation factor is: ; in, For beam b The starting frequency resource block number. For beam b bandwidth, The bandwidth of a single frequency resource block. The number of frequency band resource blocks; The co-channel interference violation factor of the system is: in, This is a preset fixed value; The number of beams where interference exists; The resource reuse violation factor of the system is: in, It is a preset fixed value. This represents the number of resource blocks that are reused between beams.

5. The satellite beam adjustment decision method based on reinforcement learning according to claim 1, characterized in that, The system score is calculated based on the following formula: in, For system indicators, The weights are those corresponding to the system indicators.

6. A reinforcement learning-based satellite beam adjustment decision system for implementing the method as described in any one of claims 1 to 5.

Images